It’s time to choose smart : New age test automation tools
Selenium 🆚Cypress 🆚 Puppeteer 🆚 Playwright
The process of test automation has seen an unprecedented change over the last few years in terms of testing tools and techniques. The open source test automation community has seen a massive surge in contribution from various product companies. Microsoft, which was at one point in time, infamous for lack of contribution towards the open source community, has also now joined the bandwagon after its take over of GitHub in 2018.
There were times where browser automation only meant “Selenium” . Now, when you look around, you have options galore. During late 2017 — early 2018, Cypress became a massive hit amongst test engineers. Somewhere in mid 2018, Google released Puppeteer 1.0. And then towards late 2019—early 2020, when the whole world was witnessing a massive pandemic (COVID-19), the test automation world was seeing the rise of Playwright.
In this article we will see how these 4 open source automation tools : Selenium, Cypress, Puppeteer and Playwright fare, against each other and then let’s analyze to see which one we should pick.
To be able to come up with an unbiased comparative analysis, I first needed to level the playing field for all. The following things were done :
- All automated tests were authored against the same React e-commerce application.
The scenarios were :
- Create an account
- Search a product and add it to the cart
- Purchase the product - For all 4 tools, the same framework was chosen — Cucumber
Let the games begin!
Round 1 — Ease of authoring
Playwright, Puppeteer and Cypress are relatively very simple to start up with you can create your very first test case in practically no time, unlike Selenium, which requires a comparatively heavier set up. However what stands out is the Playwright CLI with which you can record your user journey and interactions with the application, thereby creating your playwright scripts for your framework.
Round 2 — Support for Multiple tabs and frames
Playwright and Puppeteer intuitively support switch between frames and multiple browser tabs, a feature that Cypress does not have. Selenium solutions are rather complex implementations.
Round 3— Support for Shadow DOM (Polymer applications)
Playwright’s support for shadow root elements are very fluid and natural. No additional syntax is required. For Cypress however .shadow()needs to be chained in order to reach deep rooted elements. For Selenium and Puppeteer, the support is natively not present. There are tedious work arounds like recursively finding shadow elements using CSS selectors or using JS Paths.
Round 4— Locators and auto waiting
Playwright’s auto wait capability is a gamechanger. It eases the test authoring experience big time! For the rest of the tools, there needs to be specific waiting mechanisms that need to be configured, which means additional lines of code.
Playwright prefers to use front facing element identifiers like text="Submit" which is very user friendly. When we record using Playwright CLI, these front facing identifiers are picked up naturally, unlike the Cypress Spy which will pick up only CSS.
Round 5— Speed of Execution
To measure this, I created 4 pipelines in Azure DevOps :
For each of them, the following variables were standardized :
1.All tests run in headful mode
2.All run on chrome
3.All use the same framework
4.All run on ‘Ubuntu Latest’ agent on Azure
5.For each run, a fresh sanitized Azure agent is spun up every time
6.The automation scripts are same for all
The result was:
Round 6— Parallelism and Remote Execution
Selenium has a great support for parallel execution as well as support for remote execution in Grids. It is hands down a winner in this category. Cypress also supports parallel execution but is only available in its *paid version. Playwright and Puppeteer can be configured to execute on Selenoid and Moon only.
Round 7— Community and Support
Selenium is the oldest and by far the most popular of these tools. Naturally the community is huge and the support documents and references are available in abundance. Cypress also has a very nice documentation and loads of reference support. Puppeteer has been around for more time than Playwright so its tutorial count is higher. At the time of writing this article, Playwright team is still updating the tutorials. So the support is not abundant yet.
So there you have it! You can now make an informed decision on which automation tool to choose for your project. This may not give you a definitive answer to all your queries, but will surely get you thinking !
One thing to remember is, test automation is not merely a matter of which tool you choose. Its a combination of multiple factors, like skillset of the automation engineers, application landscape and complexity, end consumers of automation (Technical folks or Business Analysts), etc.
Important links :
Important point to note :
The above is just my point of view. I arrived at these inferences after analyzing the results of the tests which were run 10 times. I chose the above parameters as I felt they are super critical to analyze before choosing a tool. But you may or may not have the above criteria applicable.
