Speed comparison — Cypress, WDIO, and Selenium WebDriver
In this new article, you will find a speed comparison for some of the most used frameworks for e2e testing: Cypress, WDIO, and Selenium
Reason for this test
The reason behind this test is that I was responsible for choosing a framework for e2e testing in my company, and after a huge spike, we decided to move forward with Cypress, WDIO, or maybe Selenium.
So, I decided to do some tests regarding performance, because it was a very important requirement for us. If you want to go directly to the experiment that I ran, move to the My Speed Comparison session.
Background research
Before moving on to our speed comparison, I decided to do some research with some other comparisons available over the internet.
The first comparison shown in this first article (it is a scientific one), shows us that Cypress is quite faster if compared with Selenium WebDriver in JAVA. In this one, which contains only 10 test cases, we can see that Cypress was faster than Selenium.
On the other hand, when it comes to WebDriverIO (WDIO), which uses the WebDriverJS, we can see in this second article that many times WebDriverIO is faster than Cypress. It happens in general because even if Cypress runs directly in the browser, WebDriverJS is surprisingly optimized. Moreover, Cypress exhibits a longer startup time compared to WDIO. It weighs down short execution scenarios, while it shows much less in longer ones. Therefore, for bigger test suites, WDIO and Cypress may have the same overall performance in slow live websites (since waiting times will be the same for both).
After analyzing these online experiments, there are some good points to have in mind:
- Cypress exhibits a longer startup time compared to WDIO. Compared with WDIO, this weighs down short execution scenarios, while it shows much less in longer ones. Therefore, is not likely to make a real difference in the context of classic E2E testing builds.
- The most important point: local testing in headless mode is what will enable to use Cypress to its fullest speed potential. So, while debugging in a live slow website can look quite slower if compared with WDIO, when the tests are running in pipelines, where the server is locally configured to run across services, containers, etc., this is where Cypress will be probably much faster than WDIO. Therefore, Cypress will still perform very well in live slow websites, but in local ones (which is basically how the majority of CI/CD pipelines work) is where it runs faster with its full potential.
My speed comparison
Test setup:
- A small suite of 10, 100, and 200 test executions.
- Headless mode.
- That’s the chosen website due to its high loading velocity: https://the-internet.herokuapp.com/. The test case selected was the Form Authentication, which ran in loop mode.
- All tests were passed.
- Chrome 88, Firefox 84, and Edge 88.
- The tests were the same, running in loop mode for all frameworks, so we had the same scenario for each one of them.
- Cypress version 6.5.0.
- WDIO version 5.12.5.
- Selenium WebDriver in a Java/maven project, and with the latest drivers.
- The environment was the same.
See the results below for Chrome:
See the results below for Firefox:
See the results below for Edge:
See a summary comparing all the browsers:
Conclusions about this test
We saw that in many articles the results were quite different from the current ones we can see in my speed comparison. Cypress proved itself to be much faster than WDIO and Selenium in our tests. It probably happened because we are testing against a very responsive live website, so Cypress took advantage of this.
Therefore, according to my tests, we can assure you that Cypress is the fastest one for an e2e test, and it is the fastest by far. As bigger the test suites increase, the bigger the difference of time we get while running with Cypress.
Acknowledgments
Beyond me, this article was revised by Larissa Ferreira.
https://www.linkedin.com/in/laertedemesquita/
https://www.linkedin.com/in/larissacamilaferreira/
