Problem Statement:
We at Naukri define and follow various test processes and the aim of these test processes is to deliver quality products to our customers.
One of these processes is to define and upgrade compatibility matrix once in a quarter and we test our web products against this matrix.
We can perform manual testing against the defined matrix, when it comes to automation testing following questions arise?
Can we execute our automation suites against the defined matrix?
If so which selenium version is compatible for latest Firefox version?
In case of Chrome, which combination of selenium jar plus Chrome driver is compatible with the latest chrome browser?
Solution:
We identified a process around these to cater above need and the process is:
“For each quarter we do a POC to identify best compatible combination (Selenium jar and/or Chrome driver) for above matrix”
Steps for this POC is as follows
- Identifying set of scenarios for doing POC
- Defining the POC matrix
- Execution of identified scenarios against the defined matrix
- Consolidation of above results to form a logical conclusion on compatibility.
Step1: How we identified the scenarios:
We identified scenarios for doing POC on the basis of:
- Frequently used web features
- Complex web features in context of automation(taken from the past experience)
And the scenarios are as follows:
- Switching Frame
- Switching Tabs
- Uploading Files
- Mouser hover actions
- Handling custom dropdowns
- Clicking on non-viewport items
- Actions on lightbox
- Regular Test Case
Step2: Finalizing the Matrix:
We are currently using selenium 2.44 version and we want to test the compatibility of all upper versions against available browser versions and driver versions.
And final matrix looks like this:
Selenium version | Chrome driver version | Browser version |
2.44 | 2.15 | Chrome41 |
2.45 | 2.15 | Chrome41 |
2.44 | 2.16 | Chrome41 |
2.45 | 2.16 | Chrome41 |
2.46 | 2.16 | Chrome41 |
2.47 | 2.16 | Chrome41 |
2.44 | 2.17 | Chrome41 |
2.45 | 2.17 | Chrome41 |
2.46 | 2.17 | Chrome41 |
2.47 | 2.17 | Chrome41 |
2.44 | 2.18 | Chrome41 |
2.45 | 2.18 | Chrome41 |
2.46 | 2.18 | Chrome41 |
2.47 | 2.18 | Chrome41 |
2.44 | 2.19 | Chrome41 |
2.45 | 2.19 | Chrome41 |
2.46 | 2.19 | Chrome41 |
2.47 | 2.19 | Chrome41 |
2.46 | 2.16 | Chrome45 |
2.47 | 2.16 | Chrome45 |
2.44 | 2.17 | Chrome45 |
2.45 | 2.17 | Chrome45 |
2.46 | 2.17 | Chrome45 |
2.47 | 2.17 | Chrome45 |
2.44 | 2.18 | Chrome45 |
2.45 | 2.18 | Chrome45 |
2.46 | 2.18 | Chrome45 |
2.47 | 2.18 | Chrome45 |
2.44 | 2.19 | Chrome45 |
2.45 | 2.19 | Chrome45 |
2.46 | 2.19 | Chrome45 |
2.47 | 2.19 | Chrome45 |
2.44 | FF39 | |
2.45 | FF39 | |
2.46 | FF39 | |
2.47 | FF39 | |
2.44 | FF40 | |
2.45 | FF40 | |
2.46 | FF40 | |
2.47 | FF40 |
Step3: Results:
Results are as follows after executing the identified scenarios against the defined matrix:
Selenium version | Chrome driver version | Browser version | Result |
2.44 | 2.15 | Chrome41 | Not compatible |
2.45 | 2.15 | Chrome41 | Not compatible |
2.44 | 2.16 | Chrome41 | Not compatible |
2.45 | 2.16 | Chrome41 | Not compatible |
2.46 | 2.16 | Chrome41 | Not compatible |
2.47 | 2.16 | Chrome41 | Not compatible |
2.44 | 2.17 | Chrome41 | COMPATIBLE |
2.45 | 2.17 | Chrome41 | COMPATIBLE |
2.46 | 2.17 | Chrome41 | COMPATIBLE |
2.47 | 2.17 | Chrome41 | COMPATIBLE |
2.44 | 2.18 | Chrome41 | FAIL on Mouse Hover |
2.45 | 2.18 | Chrome41 | COMPATIBLE |
2.46 | 2.18 | Chrome41 | COMPATIBLE |
2.47 | 2.18 | Chrome41 | COMPATIBLE |
2.44 | 2.19 | Chrome41 | FAIL on Mouse Hover |
2.45 | 2.19 | Chrome41 | COMPATIBLE |
2.46 | 2.19 | Chrome41 | COMPATIBLE |
2.47 | 2.19 | Chrome41 | COMPATIBLE |
2.46 | 2.16 | Chrome45 | FAIL on Upload Functionality |
2.47 | 2.16 | Chrome45 | FAIL on Upload Functionality |
2.44 | 2.17 | Chrome45 | COMPATIBLE |
2.45 | 2.17 | Chrome45 | FAIL on Upload Functionality |
2.46 | 2.17 | Chrome45 | FAIL on Upload Functionality |
2.47 | 2.17 | Chrome45 | FAIL on Upload Functionality |
2.44 | 2.18 | Chrome45 | COMPATIBLE |
2.45 | 2.18 | Chrome45 | COMPATIBLE |
2.46 | 2.18 | Chrome45 | COMPATIBLE |
2.47 | 2.18 | Chrome45 | COMPATIBLE |
2.44 | 2.19 | Chrome45 | COMPATIBLE |
2.45 | 2.19 | Chrome45 | COMPATIBLE |
2.46 | 2.19 | Chrome45 | COMPATIBLE |
2.47 | 2.19 | Chrome45 | COMPATIBLE |
2.44 | FF39 | Not compatible | |
2.45 | FF39 | COMPATIBLE | |
2.46 | FF39 | COMPATIBLE | |
2.47 | FF39 | COMPATIBLE | |
2.44 | FF40 | Not compatible | |
2.45 | FF40 | COMPATIBLE | |
2.46 | FF40 | COMPATIBLE | |
2.47 | FF40 | COMPATIBLE |
Courtesy: Himani Mittal, Pradeep Kumar, Pratyush Shukla, Rahul Kumar and Anjali Vaswani (Naukri Quality)
Step4: Conclusion:
After analysing the results, Selenium jar 2.47.1 was found suitable with Firefox 40 and Chrome 45(with chrome driver 2.19)
Really informative blog…!!