You are viewing our old blog site. For latest posts, please visit us at the new space. Follow our publication there to stay updated with tech articles, tutorials, events & more.

Parallel Testing at Naukri

Wednesday, October 14, 2015 | By and
0.00 avg. rating (0% score) - 0 votes
What is parallel testing?

In simple terms, running your test scripts in parallel on same machine or on different machines is known as parallel testing.
 
Problem Statement:
Let’s consider one of the application of Naukri say “Job Search”, consists of 900 test cases which takes around 3 hours for complete execution on a standalone machine.
 
Whenever a build related to “Job Search” is about to go live, we need to run the complete automation suite to achieve quality. The issues found from the resulting report are logged and get fixed by the dev team. Once fixed the entire Job Search suite needs to be re executed to test the impact of the bug fixes. But what if we found a bug again? Suppose the suite is executed 3 times to verify the build which consumes 3+3+3 = 9 hours leading to slower delivery.
 
Solution: Here arises the need for parallel execution of test scripts to reduce the test execution time thus giving the faster testing results. In the above example, after implementing parallel execution, “Job Search” suite dramatically reduced the execution time from 3 hours to 25 mins only. So, the suite which was previously taking 9 hours to verify a particular build has now completed the task just in 75 minutes leading to effective automation testing and faster delivery.
 
Parallel execution can be done in two ways:
  1. Using single machine and running multiple instances of a browser.    
We observed this technique is useful in some cases and it’s not in some cases.
A. If the automation suite contains actions like mouse Hover, keyboard events etc. then if you run these test cases in parallel on same machine then chance of test failures may increase due to overlapping of actions.   
B. If the automation suite contains normal flows then this technique is useful, it’s the discretion of the tester to follow this approach or not.   
C. With increase in number of browser instances on a single machine, the chances of browser crashing increases due to less number of  available resources.
    2. Using multiple machines and running one instances of a browser on each machine:
We observed, this technique is more useful as each script runs in parallel on different machines. We have done this by using Selenium Grid.
Test Execution flow at Naukri before Grid implementation:
We have various applications at Naukri and each application has its own suite and these suites are scheduled to run daily through Jenkins job.
 
We have a Jenkins slave connected to the Automation Machine. Whenever a Jenkins job is triggered, the code from the Git is checked out to the Automation machine, and each test script runs sequentially in the same browser instance of that machine.

Test Execution flow at Naukri after Grid Implementation:

After grid implementation, the Automation machine on which Jenkins instance is running has now become a Hub and this hub is connected with various nodes.
 
If any Jenkins job is triggered on the hub then execution is as follows:
  1. Code will be checked out from the GIt and loaded on the HUB machine.  
  2. After building successfully, test execution starts and the selenium hub automatically distributes the tests across the nodes, this dramatically speeds up testing.
 
Changes done to achieve parallel execution through Selenium Grid:
We have implemented Grid in such a way that our automation suite does not get impacted and runs successfully in absence/presence of grid setup.
 
How? For this, we have created an environment variable “seleniumExecution” whose value is initialized as “grid” on HUB machine.
 
The following changes are done in the automation suite based on this environment variable:
  1. Instantiating     the WebDriver: The way we instantiate WebDriver for grid is  different from standalone execution.
         Start Driver function for Standalone execution: The method startDriver() which creates and returns the Webdriver instance looks like below for standalone:
         public WebDriver startDriver() {
               DesiredCapabilities capabilities = DesiredCapabilities.firefox();
                       capabilities.setCapability (FirefoxDriver.BINARY, “Path for firefox.exe”);
                       WebDriver driver = new FireFoxDriver(capabilities);
                       return driver;
          }

          Start Driver function for Grid/Parallel Execution:

          For Grid hub the environment variable “seleniumExecution” value is “grid” and for other machines as this variable is not configured its value would be null. Using this logic hub is differentiated from standalone machine.

          public WebDriver startDriver () {

          String seleniumExecution = System.getenv(“seleniumExecution”);
          if (seleniumExecution == null) {
              DesiredCapabilities capabilities = DesiredCapabilities.firefox();
              capabilities.setCapability(FirefoxDriver.BINARY, “Path for firefox.exe”);
              WebDriver driver = new FireFoxDriver(capabilities);
              return driver;   
          }
          else if (seleniumExecution.equalsIgnoreCase (“grid”)) {
      ThreadLocal<RemoteWebDriver> threadDriver = new ThreadLocal<RemoteWebDriver> ();
              DesiredCapabilities capabilities = DesiredCapabilities.firefox();
              capabilities.setCapability(FirefoxDriver.BINARY, “Path for firefox.exe”);
              WebDriver driver=new RemoteWebDriver(new URL(“http://localhost:4444/wd/hub”), capabilities);
              threadDriver.set((RemoteWebDriver) driver);
              return threadDriver.get();       
          }
          else{
          }
       }  
 
      2. TestNG.xml:       
      TestNG allows the tests to run in parallel and multi-threaded mode. This means that based on the test suite configuration, different threads are started simultaneously and the test methods are executed in them.
       At Naukri we have written code which automatically creates testing.xml based on test execution selection.
       Before Grid TestNG.xml: Before grid implementation, we used to run our test suite in a sequential manner and each <test > contains one class having multiple test methods to be executed. A sample testing.xml looked like:
<?xml version=”1.0″ encoding=”UTF-8″?>
<suite name=”Jobsearch”>
    <test name=”Search_1″ preserve-order=”true”>
        <classes>
            <class name=”test.SearchTest”>
                <methods>
                    <parameter name=”browserType” value=”firefox”/>
                        <include name=”TC1″></include>
                        <include name=”TC2″></include>
                </methods>
            </class>
        </classes>
    </test>
    <test name=”Search_1″ preserve-order=”true”>
        <classes>
            <class name=”test.ClustersTest”>
                <methods>
                    <parameter name=”browserType” value=”firefox”/>
                        <include name=”TC3″></include>
                        <include name=”TC4″></include>
                </methods>
            </class>
        </classes>
    </test>
</suite>
For Grid TestNG.xml: TestNG provides multiple ways to execute the tests in a multi-threaded condition, one of them is executing each test method in a single thread. This mode reduces the execution time significantly because more tests are executed in parallel, hence reducing the total execution time. So, we created testing.xml such that each <test> tag contains only one test method and we run the suite with attribute parallel = “tests”. The attribute thread-count can be set based on the number of nodes in the grid. At Naukri we have set up 8 nodes in Grid and on each node we want to run 2 browser instances, so we have set the thread-count=”16” (8*2).
A sample testing.xml after grid implementation looked like:
 
<?xml version=”1.0″ encoding=”UTF-8″?>
<suite name=”Jobsearch” parallel = “tests” thread-count=”16″ >
    <test name=”Search_1″ preserve-order=”true”>
        <classes>
            <class name=”test.SearchTest”>
                <methods>
                    <parameter name=”browserType” value=”firefox”/>
                        <include name=”TC1″></include>
                </methods>
            </class>
        </classes>
    </test>
    <test name=”Search_1″ preserve-order=”true”>
        <classes>
            <class name=”test.SearchTest”>
                <methods>
                    <parameter name=”browserType” value=”firefox”/>
                        <include name=”TC2″></include>
                </methods>
            </class>
        </classes>
    </test>
</suite>
Issues faced during execution of legacy code after grid implementation:
  1. Dependent Test cases: There were few test scripts which are dependent on some other test scripts, post grid implementation these test scripts were getting failed.   
  2. Multiple test scripts accessing /writing to same workbook: There were test scripts which were using the same workbook for reading/writing. Those test scripts when run in parallel failed as the same workbook could not be available to some test scripts.
  3. Larger test scripts lead to ineffectiveness of node machines: Some test scripts were very large in size taking hours to execute. Such test scripts hinders the purpose of reducing test execution time as they keep only one node busy for a larger period of time while other nodes keep resting.
 
Approaches to be followed to overcome the issues faced for Grid execution:
  1. Developing autonomous Tests: We should develop tests which could be run     independently. Test script should be able to sets up its own data and flow rather than relying on a previous test to do it. Avoid  using @dependsOnMethods annotation in the test script.   
  2. Avoid writing in test data workbooks: Multiple tests reading from same workbook at the same time does not cause any harm to the test scripts but writing to the same workbook at the same time causes test failures. So avoid writing to the workbooks if you are using same workbook for multiple tests.   
  3. Develop atomic Test scripts: Each test script should be written to test a single feature instead of multiple features at a time. In short, test needs to be concise for better quality and maintainability of your tests
 
Managing the Selenium Grid Hub and Nodes:
We are working on following for managing the Hub and Nodes for Selenium Grid:
  1. Automatic killing of browser instances before executing the suite.   
  2. Automatically setting host entries on all node machine(useful if we want to bind application to any IP)   
  3. Periodically restarting the Grid Nodes to clean up the JVM for any remaining memory leaks.   
  4. Automatic execution of selenium grid commands to launch hub and nodes after restarting the machine.
 
Conclusion: Parallel execution gives Naukri a lot of advantages over normal execution, mainly reduction in execution time and faster delivery. We just require below setup to make our automation suite feasible for parallel execution:
  1. Multiple machines: One machine acts as the Hub where automation suite is loaded and the other machines act as the Node where test execution will be performed.   
  2. Updating code of browser instantiation: Browser instantiation code should be handled in such a way that automation suite runs in sequential manner on standalone machine and runs parallel on a machine with grid setup.       
  3. Updating testing.xml: Testng.xml is updated to run each test method in parallel by configuring thread-count.
Posted in Automation, Automation Testing, Quality Assurance
Tags: , , , ,

6 thoughts on “Parallel Testing at Naukri”

  1. I have 3 nodes and one hub. For automating on three nodes do in need to create three
    remotewebdriver objects . On machines 147 148 and 149 as listed below

    driver = new RemoteWebDriver(new URL( “XXX://XXX.XX.XXX.147:5566/wd/hub”), capability);
    driver = new RemoteWebDriver(new URL( “XXX://XXX.XX.XXX.148:5566/wd/hub”), capability);
    driver = new RemoteWebDriver(new URL( “XXX://XXX.XX.XXX.149:5566/wd/hub”), capability);

    When i am doing this only the machine 149 the last of the above mentioned objects is executing . I want the code to be executed in all my machines. What is the solution for this.

    Do i need to create a separate method that return webdriver if so how to do it?

    Waiting for response. Thanks in Advance

    I believe the code base will be available only in hub and the node machines will not have any code base. Please correct me if i am wrong

  2. I got the solution. Grid automatically will listen to the nodes that were registerd so i have created only one driver object with localhost as hub is running in local host else not if hub is in particular server ip we need to provide that.

    WebDriver driver=new RemoteWebDriver(new URL(“http://localhost:4444/wd/hub”), capabilities);

    1. Yes, you got the solution right. We do not need to mention Ip’s of node as nodes are registerd to hub. So creating a driver object using hub IP address, hub automatically distributes the test scripts to different nodes based on their availability.

  3. Thank You Garima for responding. I have been going through almost all the articles that were published and i can say every article is at it’s best.

    I have a small query what is the use of ThreaLocal as part of the below code.

    ThreadLocal threadDriver = new ThreadLocal ();

  5. ThreadLocal helps in achieving thread safety for the objects which are not thread-safe by providing explicitly copy of Object to each thread.
    Here driver object is not thread-safe, so if multiple nodes access the same object at that same time, any one node will result in closing the driver object while the other will be executing the script at the same time which will lead to webdriver exception.
    So to resolve this issue, threadlocal is used in order to provide explicit copy of driver object to each node.
    Hope this answer helps!!

Comments are closed.