Headless Automation with Selenium + PhantomJS

Posted by Nilanjana on Apr 13, 2016 4:28:00 PM

Headless Automation testing is a way to run light-weight tests without opening a physical browser. Headless, as the word suggests, means without being seen. So basically, a headless browser is a web browser without a graphical user interface. Hence, with Headless Automation, users would not see any direct browser interaction and most of the functionalities would run from the back end, via command-line interface or network communication.

Quick fact: Google started using headless browsing from 2009 in order to index contents of websites using Ajax.

Now let's dive into Headless Automation with Selenium and PhantomJS

Initially, I used Casperjs to carry out automation for a few test cases.  Being new to JavaScript, it was a bit time consuming for me to learn the implementation of Casperjs along with the implementation of JavaScript. So now I use PhantomJS driver for headless automation with Casperjs as a wrapper over PhantomJS to extend its capability. It was quite easy to implement and interesting to see it run in a headless way.  That is when I started researching other ways of integrating PhantomJS with Selenium. After quite a bit of research I found out that the PhantomJS driver is itself readily available with the Selenium library.

Before we go any further, let’s understand what PhantomJS is. It is a webkit which runs headless with an inbuilt JavaScript API. It has fast and native support for various web standards such as DOM handling, CSS selector, JSON etc. It is quite fast as compared to running tests using Selenium web driver. PhantomJS uses the WebDriver Wire Protocol implementation which is achieved by GhostDriver running at the back end of PhantomJS.

Using PhantomJS driver with Selenium is quite easy, since one can use their own implementation technique to automate test suites using PhantomJS. For example, I used Java to implement the test scenarios with PhantomJS as a browser to run the desired test cases.

Now let’s get down to business.

Pre-requisites

  • You will need the binary executable file of PhantomJS. Make sure the PhantomJS is version 2 or above.

  • Java Development Kit installed on your machine.

  • Eclipse IDE (this is not mandatory). You can also run your test cases from your console.

  • Selenium jars.

  • Test-NG. I used Test-NG to run the test cases. One can also use the normal java implementation to run the test cases.

Steps for Automation

Step-1: Open Eclipse IDE and create a Java project.

The following screen will appear

headless_automation_phantomjs_selenium_1_-_srijan

headless_automation_phantomjs_selenium_2_-_srijan

Step-2: Create the required project. Now when you click on you project you will see the following folder structure.headless_automation_phantomjs_selenium_3_-_srijan

headless_automation_phantomjs_selenium_4_-_srijanStep-3: By default there would be a set of system jars present. You would have to import the required selenium jars in order to get started. Follow the required screenshots

headless_automation_phantomjs_selenium_5_-_srijan

Now comes the secret ingredient that would help you start running your automation using PhantomJS.

Capabilities caps = new DesiredCapabilities();

((DesiredCapabilities) caps).setJavascriptEnabled(true);

((DesiredCapabilities) caps).setCapability(“takesScreenshot”, true);

((DesiredCapabilities) caps).setCapability(

PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,

“//Users//soumyajit//Documents//Zippers//phantomjs//bin//phantomjs”

);

driver = new PhantomJSDriver(caps);

return driver;

 

As you can see, I have used Capabilities class and its required methods to configure the driver instance as in order begin execution on the browser side. In the above snippet, I have set the desired capability to get the path of the binary PhantomJS execution file. Also, I have set PhantomJS to take screenshots. 

You can also set the capability to ignore all SSL alerts appearing by using this line of code:

((DesiredCapabilities) caps).setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS,new String[] {“–web-security=no”, “–ignore-ssl-errors=yes”});

 

I have used absolute path to point to the PhantomJS binary execution file. But best practice is to use relative path as it would be more generic to a system.

And now we are set to run our automation test case.
headless_automation_phantomjs_selenium_6_-_srijanThis is the view of my console where the test cases are running.

A few points with respect to the console where you see the test cases running:

  • Your console or the CLI where the tests are being executed by PhantomJS starts acting as a browser console.
  • Users start seeing browser console related errors while the tests are running. This as an added advantage because if there is a fatal console error that may lead to someone cracking the system, these can be reported earlier to ensure the protection of the application.
  • There is a also the PhantomJSdriver log which separately guides the user with the browser behaviour, such as page settings for each session, what are the errors in case there is any failure in running the test cases etc.

Here's what the phantomjs driver log looks like:
headless_automation_phantomjs_selenium_7_-_srijanSo this is the long and short of headless automation using phantomjs. For further resources you can visit //github.com/SoumyajitBasu1988/Selenium-PhantomJSDriver

Please do provide your feedback and any doubts in the comments below. 

Topics: Framework and Libraries, QA and Testing

Why should you opt for Automated Testing?

Posted by Anil Chandna on Apr 1, 2016 4:59:00 PM

A manual QA, done with lots of practice and experience raises the bar for the quality improvement of products, which is a very desirable outcome. Nowadays, many of the products/projects require multiple deployments of new features in a single day (or within a few days), which might cause problems for some of the old features. In such cases, it is possible to do some sort of a manual impact analysis and verify the old features.

However, there are certain cases where manual testing will not suffice:

  • As the product size grows, the number of impacted areas per new feature also increases. So we have more things to test.

  • Individuals tend to lose interest by re-iterating this monotonous activity, creating higher chances of error.

  • Manual testing on multiple platforms is time consuming. Repeated communications between the dev and QA teams also takes time. This highly increases the "time to market", which is not a good news for business.

  • A human being can't work 24*7, but a machine and a code can.

  • There can be more/less problem depending on product's nature

 

An easy solution could be adding more bandwidth or giving more time to verify new features along with the old ones. But, "time to market" plays an important role. We are currently living in a "business driven development" world. A delay in the bringing new features to market can cause huge losses for business.

So, our solution to this problem is "verify quickly" which improves "time to market". This can be achieved effectively by using automation at the different levels of testing and in different ways.

Here are the benefits of doing automation:

  • Dev team gets to know about issues quickly

  • Issues can be quickly re-tested across platforms.

  • Executing repetitive tasks with automated software testing gives your team time to spend on more challenging and rewarding projects.

  • Team members improve their skill sets and confidence and, in turn, pass those gains on to their projects and organization

  • Test cases can be run on multiple platforms and browsers simultaneously to save time and to improve accuracy.

  • Code learning for QA people gives them a great insight into project code as well, which in turn leads to identifying some performance critical bugs which people can not observe through UIs.

  • Continuous Integration helps the teams to run the critical set of test case automatically whenever a new code is pushed.

  • Most of the automation tools come with neat and clean reports which can be sent automatically to the team.

Despite all this, human quality analysis still plays the most important role in maintaining the core quality of the product. The quality of automation scripts is also totally dependent on human beings who are developing these scripts. An automation script just does the work which is coded in scripts. It has no intelligence beyond those written checks. Human beings are the ones who are smart enough to go beyond these scripted boundaries and can contribute a lot to the quality of the product. So, their time is important. 

And to save their time automation is important.

Topics: QA and Testing

Behat + Mink + PhantomJS = Test all the things!

Posted by Nilanjana on Sep 25, 2015 3:44:00 PM

Michelle Sanver, co-president of PHP Women, a providing support for the PHP community, and an open source enthusiast passionate about coding placed her session at DrupalCon Barcelona on how can someone test their application to be sure of any known and unknown faults within the system. She covered how can we use BDD and TDD extensively to be sure of our product.

 

Here are the key insights from her session.

BDD or TDD – What is it and Why do we need it?

Behavioral or Test Driven Development is a guide to agile practices to reduce redundancy in the amount of bugs that we discover in our day to day quality analysis over a product. It also thrives in reducing the amount of time spent in manually testing an application. Furthermore, BDD can also reduce the pitfalls of an unknown fault in the application. 

Writing BDD scripts is simple because you can say it is a straight forward English written scripts depicting all the steps one by one, even before developing the application has started.

In her session, she did not focus much on Drupal but on web applications and how to test those applications. She mentioned how can one use Behat and PhantomJS and Mink to write down test cases.

So to start writing the test cases we have to use the gherkin format. Gherkin defines the behaviour which can then be executed by Behat.

Michelle defined Gherkin format by citing an example of a behaviour. She also mentioned that gherkin is only to define the behaviour and not run the test. For running the test you still need to depend upon the framework, like in her case, she used Behat.

So, next up she explained the use of Mink. Mink is an open source browser controller/emulator for web applications. The role of mink in running a test case is to simulate the interaction between the browser and the web application.

Next she spoke about PhantomJS, which can be used effectively to run your test scripts. PhantomJS is a headless kit. You need to explicitly make it behave like selenium driver by configuring the ghost driver. You can run functional tests with frameworks such as Jasmine, Mocha, Webdriver etc. Also, screenshots can be created for website preview. So definitely PhantomJS can be an effective tool to run all our test scripts.

Topics: Framework and Libraries, QA and Testing

SimpleTest to SmarTest

Posted by Nilanjana on Sep 25, 2015 3:29:00 PM

Ana Belén Sánchez, a computer engineer who works as a researcher and teacher at the University of Serville did a fantastic job with her presentation on SmarTest @ DrupalCon Barcelona. She target the exact pain area of test prioritization of automated tests and came with SmarTest solution. I liked the flow of the presentation, and thanks a lot to Ana for giving such an informative session. Here are key insights from her talk.

What is the real problem?

In today’s world, we have variability-intensive systems having billions of configurations and lots of dependencies. It’s a highly challenging job to test these systems because of following reasons:

  • Too Many Configurations in system
  • Time Consuming efforts

Before going to solution of above problems, let’s evaluate Drupal as a system with thousands of modules, millions of configurations and lots of dependencies.

Drupal Variability Model:

In presentation, Ana took a drupal system consisting of:

  • 48 modules (mandatory and optional)
  • Having 21 dependencies
  • 2000 millions of configurations
  • 352 test case with more than 24K assertions
  • 160 integration faults(integration of two or more modules) were found by some previous test runs

Statistical Studies done on Drupal Modules

Parameters

Growth

Faults Rate

Module Size

   

Module Changes

   

v7.22 >> v7.23

   

Module CC

   

Module with multiple contributor

   

Number of tests

   

Core Module

 

41% of modules

Optional Module

 

91% of modules

 

Proposed Solutions to above Problems:

Test case selection techniques

  • Reduce the test space by selecting a portion of configurations to be tested.

 

Test case prioritization

  • Schedule test cases to increase effectiveness by reviewing past results
  • Test Case Prioritization criterias:

     -Size-driven criteria

     -Fault-driven criteria

     -Change-driven criteria

     -CC-driven criteria

After applying the above proposed solutions:

  • Test case selection lead to increase the fault rate by 86%
  • Test case selection + prioritization lead to increase the fault rates by ~94%

 

Apply these proposal to SimpleTest of Drupal

SmarTest is a module for improving the testing process in Drupal. It extends the functionality of SimpleTest core module by integrating testing techniques and providing the Drupal developer with a dashboard with statistics about the Drupal system in real time. This information allows you to guide the testing in your system through faults propensity data in different parts of the code. This is based on a correlation study that relates number of commits made in a module, faults detected in last tests executions, cyclomatic complexity, etc with the faults propensity of the module. Also, SmarTest allows the testers to prioritize (i.e. order) the executions of the Drupal tests modules in order to detect faults as fast as possible basing the prioritization in an extensive correlation study.

Improvements with Respect to SimpleTest

  • Customizable Dashboard with run-time extracted data to guide the testing
  • Different Test prioritizations to detect faults faster
  • Automated testing with feedback in real time
  • Timeout option to automatically stop the test execution
  • Git configuration  for obtaining commits per module

 

Dashboards:

Prioritization criteria with SmarTest:

Topics: QA and Testing

A/B Test Content Experiments: Keeping users first

Posted by Nilanjana on Aug 27, 2014 2:55:00 PM

Sometime back we published results of an A/B test we carried out for conversion optimization. This time around we're sharing results of an A/B test carried out in order to test out content placement. 

Hypothesis: We were getting a lot more clicks for Outsourced Product Development (OPD) than any other service. So we wanted to test whether people are clicking on OPD more because its at the top spot or are they genuinely interested more in it. 

Test: We placed 2 versions of the Solutions page. One where "OPD" was listed as the first solution, another one with "Trustworthy solutions for B2B Marketers" at top spot.

Results: Below are the click results on various solutions in both the cases.

Case A: When OPD was at the 1st place: (113 page views)

  • 16 on OPD
  • 4 on B2B Marketer
  • 4 on Media Solutions
  • 7 on Drupal Solutions

Case B: When B2B Marketer was at 1st place: (103 page views)

  • 10 on B2B Marketer

  • 9 on OPD
  • 7 on Media Solutions
  • 7 on Drupal Solutions

Inferences: 

  • Given that OPD is clicked 4 times more than B2B when at 1st place and almost equal no of times when at 2nd, we can safely say that there is far more interest for OPD as compared to B2B Marketer.

  • Interestingly, if we combine the total number of clicks (for both versions), Drupal Solutions stands joints 2nd. This, despite being placed right at the bottom of the page.

 

Conclusion: A small test and we could figure out what our users want. We retained OPD at the top while moved the Drupal solutions up to the second spot. After all, users first :) 

Topics: QA and Testing

Check every piece of code you insert

Posted by Nilanjana on Sep 28, 2011 2:48:00 PM

Background

We had an issue where one of the customer sites was waiting on a particular page for over 3 seconds, sometimes more. To make it even more interesting, the site would default to xx.yy.com/undefined. The site generally hits peak loads rather often so we didn't realise for a long time that there was actually something fundamentally work in our code. We Programmers *always* write flawless code, right?

Details of the issue

Our SysAdmin team pointed out the issue many weeks before we began looking at it. They had to practically *prove* to us that the 3+ second wait is pretty consistent, though not always. Since it can never be a flaw in our program, it *must* be server issue; too many hits on the site and the server gets over-loaded. Programmers never write bad code, it is always SEP (Somebody Else' Problem). We finally got down to thinking, perhaps looking at the code wouldn't do much harm. Using firebug, we were able to isolate the problem fairly quickly -- fact that the site waited patiently for 3+ seconds meant that even our slow brain couldn't help but see where the site paused and twiddled its thumbs for a while before throwing the undefined error. "undefined" is Javascript's way of saying: "You didn't write anything in the variable, so stop asking me to give you any value". This was our first clue. We were accessing some part of the program where we were supposed to store some value, but had "forgotten". Given the clue above, stare at the code below for a bit and try to identify the problem -- If you aren't a programmer, or just aren't interested, jump to the last 2 paragraphs: 1    

{C}
1     <!-- XXX SMO tracking code start-- >
  2     <script type = "text/javascript" > var regexS = "[\\?&]smo=([^&#]*)";
  3 var regex = new RegExp(regexS);
  4 var results = regex.exec(window.location.href);
  5 var xxxSmo = new Image(1, 1);
  6 if (results != null) {
  7     var smoParam = results[1];
  8     var pinPath =
  9         'http://www.XXX.com/track/SMOTrack.php?projectID=720&smo=' +
 10         smoParam;
 11 }  
 12 xxxSmo.src = xxxPath;
 13 </script > <!--XXX SMO tracking code end here-- >

{C}

Looking at the code for a few minutes, it is obvious that not everything is kosher. If you haven't found the problem yet, look at lines 6 and 7 some more.

What you'll notice is that the code checks if results array is not null using regex magic - that i haven't deciphered yet. Since the results array is non-null, the program assumes that the next element of the array, which probably contained the magic params that it was looking for. Wrong assumption; results does exist, but not array[1].

The correct code would have been something like this: if (results != null  &&  results[1] != null)   # 'still not perfect, but this is blog, not a coding contest.

The code should have specifically checked for the variable it was accessing (results[1]), rather than *assume* that the entire array would exist as long as the array got created.

To be fair, the rest of the code is rather clean. Clean naming conventions, good flow and there was at least a basic check. The programmer simply assumed that his site would be live forever and never bothered to check what would happen if his site were down, or worse, if it just went. For keeps, which is the case here.

Since this is code that was written to be inserted into *all* sites that bought into the SMO (Social Media Optimization) tracking, what this code effectively did was to slow down every participating site by over 3 secs per page, every time *it* went down! Another issue is that a SMO code should try and self-annihilate - a customer may stop paying or is just a serious bother in some other way and there was no way for these folks to prevent being spammed from their own ex-customers. Like giving your not-so-SO your on-line debit card details ;-)

Coming back to the subject of the blog, the code should have checked *every* piece of input that was received, not just the first one. For a company like Srijan which creates web-sites for a living, the blog subject carries an even heavier message:

Check every piece of code you insert

Yes the *whole* code that you're putting in, not just the pieces that don't work. How many of us check the input parameters to functions that we wrote, or we borrowed from our colleagues or (worse?) from a third-party module? If a (seemingly) experienced programmer can make our sites slow down noticeably, imagine the chaos a ver 0.8 module that we are using without any verification can do?

Written by the Dinasaur aka Dumbledore at Srijan, I am his messenger ;)

Topics: Drupal, QA and Testing

Opening for a QA Engineer

Posted by Nilanjana on Mar 21, 2011 3:04:00 PM

Srijan has an opening for a QA Engineer, at its office in Gurgaon.

Responsibilities

  • Assist Project Manager/Team Leader in gathering project requirements from clients
  • Write Test Cases
  • Build/Maintain a Test Plan
  • Update/Maintain Functional Specs and Design documents
  • Black Box testing
  • Write Automated Test Cases using Selenium
  • GUI and Browser compatibility testing
  • Filing bugs in JIRA issue tracking system

Key Skills and Experience

  • Atleast 1 year QA/testing experience in Web Projects
  • Excellent English communication skills
  • Excellent inter-personal skills
  • Knowledge of continous integration systems like Hudson, CruiseControl, etc. is a big PLUS.

 

To apply for this vacancy please write to us at jobs [AT] srijan [DOT] in.

About Srijan

Srijan is an award winning Drupal development and consulting company that specialises in online media publishing. It has several successes in Drupal such as www.mnn.com, www.indiaenvironmentportal.org.in and www.openthemagazine.com (the web portal for the recently launched OPEN news and features magazine by the RPG Group). Srijan has a transparent and participative work culture that encourages innovation in work, dialogue in all facets of the company's functioning, opportunity to learn and diversify into new technology areas, and coaching and hand-holding in managing work-life better.

To know more about Srijan's Drupal competencies, please visit: http://drupal.org/node/1092958. Also, find us at Facebook and Twitter.

Topics: Life at Srijan, QA and Testing

Openings for "Testing Engineers"

Posted by Nilanjana on Oct 8, 2010 3:43:00 PM

Srijan is looking for Testing Engineers with the following experience and skills:

  • Can analyse User Stories and develop Test Cases for those
  • Must know Unit Testing, Integration Testing, Functional Testing, Regression Testing and System Testing(UI Testing)
  • Should be willing to learn Automated Testing using Selenium RC + PHP Unit
  • Mimimum experience required 1 year

To apply, write to us with a brief note at jobs@srijan.in

About Srijan

Srijan is an award winning Drupal development and consulting company that specialises in online media publishing. It has several successes in Drupal such as www.mnn.com, www.indiaenvironmentportal.org.in and www.openthemagazine.com (the web portal for the recently launched OPEN news and features magazine by the RPG Group). Srijan has a transparent and participative work culture that encourages innovation in work, dialogue in all facets of the company's functioning, opportunity to learn and diversify into new technology areas, and coaching and hand-holding in managing work-life better. A few highlights about the company:

  1. We're Acquia Silver Partner (as a Drupal development agency)
  2. We've just been confirmed as Acquia's Drupal Training partners in India, and will launching our first public training in November 2010
  3. Our leading Drupal deployments are: www.mnn.com, www.openthemagazine.com and www.indiaenvironmentportal.org.in; we also have intranet deployments (Drupal) for World Bank.
  4. We specialize in Search implementation for the content heavy sites we build. This is implemented using Apache Solr, one of the leading open source search technologies.
  5. We also specialize in scaling Drupal sites; MNN.com is known to manage 6m page-views in a single day and over 3000 concurrent users
  6. http://drupal.org/node/629664 - is a highly acclaimed Case Study on OPEN Magazine which ran on drupal.org for 5 weeks - a first by any company from this part of the world
  7. We make contributions to the Drupal community regularly with modules, case studies and sponsorships of community events. Some examples are:

Topics: Life at Srijan, QA and Testing

Discussion

Write to us

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms Of Service apply. By submitting this form, you agree to our Privacy Policy.

See how our uniquely collaborative work style, can help you redesign your business.

Contact us