Continuous integration is the talk of the town. Every business is aiming to ship products quicker with no compromise on quality. With this evolution in the IT industry, there was a huge requirement to fine tune our test processes and revive the traditional test approaches. This blog details out some of the changes that happened at Naukri leading to more focused and grown-up QA practices.

Challenges faced during continuous improvement: The IT industry has evolved significantly with the changing time. This evolution is majorly driven by factors like advancement in information technology, country economy, takeovers and tie-ups, increasing customer demands and business needs, collaboration, numerous start-ups etc. This leads to various challenges for Naukri as a product company:

1. With increase in demanding customers and faster response time, multiple projects need to be handled simultaneously with the assurance of delivering a high quality product from QA perspective.
2. With changing business requirements, organization must have robust test processes in place and can quickly adapt to changing business requirements.
3. Due to progression in information technology, we must focus on faster delivery of systems at lower costs.
4. With numerous fresh launchers and increasing competition in market, it is very important to ensure that the product is validated and defect free
5. In today’s volatile economic conditions, organization also needs to focus on enhancing their business value with QA programs.

Grown-up QA practices: The outcome of a software project is based on the success of the testing phase. At Naukri, we are now more inclined towards mature QA testing practices by following a business value focused QA approach which includes:

• Team Composition: We have a fixed core team per scrum and a flexible & specialized variable team. Based on the upcoming project requirements with in a scrum, the variable team is moved from scrum to scrum. This also helps in maximum utilization of resources.

• Earlier involvement of QA: QA team is involved in every pre-iteration and iteration planning to have a proper understanding of core business requirements and their business values. They have the authority to ask questions from stakeholders about the changing business requirements and their impact on the overall business of the organization. This helps them in understanding the business value of the AUT (Application under Test).

• Test with a Business perspective: Once the QA team is having a thorough understanding of the requirement’s impact on the business values, they are able to detect critical business process defects in a quicker and efficient way. Testing from a business perspective helps in identifying requirement related defects at an earlier stage. This prevents us from building a wrong product.

• Defect prevention: Story grooming and Impact analysis is done at the time of iteration planning to discuss all the possible test cases of that requirement. QA team pens down the identified test cases and share it with development team at the start of the iteration itself. Dev team makes sure that all the test cases are verified on their servers before deploying the build to test. This improves build quality and helps in defect prevention at a very early stage thus reducing the testing costs.

• Progressive Automation: Automation now happens in parallel to the development phase. Front End Development team shares the HTML of the new page under development with the QA. QA team creates the page object framework and automation test scripts using that HTML in parallel to the development team. This progressive automation helps in following ways:

1. When build comes to testing phase, QA run the automated test scripts in the first cycle of testing itself. Thus reducing the repeated manual effort and identifying the defects quicker.
2. A developer provides the build of AUT multiple times during the testing phase. Testing a large chunk of test cases manually every time a build comes to test is very difficult and sometimes tester misses the impacted scenarios. However, the automated test scripts can be executed repeatedly and helps in identifying the impacted areas.

• Minimizing Flaky Tests: With progressive automation, the maintenance of our automated test scripts has become more crucial. There is nothing worse than a test that passes and fails randomly without any new bugs being introduced. It actually makes us numb to failures and we miss an actual failure condition. So it is very important to prevent creating flaky tests and maintain a healthy and reliable automated testing system. At Naukri, we are planning towards Continuous integration system where our automated tests will run automatically to verify each build. In order to achieve this, it’s very important that we and our developers trust our tests. Our QA team is continuously working towards optimization and refactoring of their automation code to reduce flaky tests on immediate basis. We first identify all the flaky tests by writing meaningful error messages in our automation code and then fix them periodically.

• Build Verification Tests: Build verification testing is somewhat similar to smoke testing. Module wise Jenkins jobs are created having automated sanity test scripts. These test scripts are core functionality test cases that ensure application is stable and can be tested thoroughly. Dev team executes the respective AUT job on their servers to verify the build before deploying to QA. BVT is successful if the newly developed feature is failed in the test suite while the non-impacted areas are passed. If BVT fails, that build is again fixed by the developer. This prevents “Dead on Arrival” situations and saves a lot of testing time. Test team can now focus on finding more complex functional bugs rather than just logging the basic requirement bugs.

• Reuse of regression test scripts: Although running regression suites is an old QA practice but it’s now carried out in the earlier phases of testing. Productivity of tests has increased significantly when test scripts from previous releases are executed in the first cycle of testing thus unknown impacted areas are found in the early stages of testing lifecycle.

• New Skills and Toolsets: Based on the Business strategies, management keeps introducing the new skills and toolsets. These are implemented by the QA team for continuous improvement in testing techniques. Recently we have

  implemented:

1. Selenium Grid – Selenium Grid is used to run automation test scripts in parallel on different machines and browsers. Through its implementation at Naukri, we are able to run our 1000s of automated test scripts in parallel. This gives us a lot of advantages over normal execution, mainly reduction in execution time and faster delivery.
2. Test Link – At Naukri, Test link act as a centralized repository for all test cases and their results. It is used for executing and linking automation test scripts with manual test cases. It helps in tracking story wise test case execution results, thus making multiple projects management easier.
3. Jenkins – Jenkins helped us in putting first step towards continuous integration. We have scheduled our automation test suites through Jenkins jobs to execute them periodically. This relieves us of manual intervention of executing the test suites whenever a build deployed to production.

• Key Stakeholders involvement: All key stakeholders (like pr

  oduct manager, business head, Dev Manager, QA manager etc.) are involved in the QA processes and their timely feedback on the “Application under Test” has a positive impact on the business goals. Eg: QA team calls for PAT (Product Acceptance Testing) just after first cycle of testing completes.

• Maintaining Quality reports and trends: Creating reports to track the software quality in its current state, as well as to compare the improvement with previous versions. This will help increase the value and maturity of the testing

  process. At Naukri, we maintain a number of reports categorized as:

1. Scrum Delivery Report: Scrum report is created at the end of every iteration to track the improvement status of each scrum. These include trends like:

a. Build Quality Trend: It measures the quality of build deployed to test with each iteration and the trend helps team improve the build quality rate over time. Less the number of bugs found with each build, better the build quality.

b. Velocity report: Calculating velocity of a scrum team is the powerful method for accurately measuring the rate at which scrum teams consistently deliver business value. It is the sum of the estimates of delivered (i.e., accepted) features per iteration. This directly helps in estimating how long it will take a team to complete a software development project. Velocity trends are maintained over iterations so that if velocity fluctuates widely for more than one or two iterations, the Scrum team may need to re-estimate and/or renegotiate the release plan.

c. Functional Automation Coverage Trend: Functional automation coverage trend tracks the number of stories automated out of the total number of stories done per iteration. Our target is to automate all the functional test cases of an automatable story through progressive automation.

2. Quality Report: Quality report contains coverage trends tracked over every month. These include:

a. Test Case Coverage: By calculating the test case coverage per module, we can measure the amount of testing performed by a set of test cases. It also helps in finding the areas not exercised by a set of test cases and in result makes us to create additional test cases to increase coverage.

b. Automation Coverage: The purpose of maintaining Automation coverage trend quarterly is to track the Automation progress and to improve the % automated. First we identify the automatable test cases out of the total number of test cases written, and then we calculate the number of test cases automated among those automatable cases. This helps in measuring the amount of testing performed through automation and finding the areas not automated.

c. False Positives: False Positives refer to those flaky test scripts which fail randomly without even code change. There is no use of doing automation testing if it does not give us true results. At Naukri, we aim at maintaining our suites to reduce the false positives as much as possible and the same is measured through False Positives trend.

d. Execution time: Just automating and not doing execution of the same because the entire suite takes a large amount of time is not a good testing practice. At Naukri, we are continuously working towards reduction of test execution time to get the faster testing results. Amount of execution time taken per automation project is tracked quarterly through Execution time trends.

Conclusion: Today, all organizations want high level quality products which increase their business value. A good indicator of successful testing is based solely on how well a company’s product performs in the production environment. While we cannot predict all possible defects in a product, a business value focused QA approach minimizes the risk of developing a defective product and helps in realizing overall business goals.

Happy Testing!!!

Related Posts

• Tip-toe through Naukri, India’s No.1 job site #OneDayTale
• Selenium Grid Using Docker
• The story behind how Naukri won DSTC 2016 Automation contest
• Understanding Disk I/O – when should you be worried?
• Testing on our private mobile device farm using STF