Speed comparison — Cypress, WDIO and Selenium WebDriver

In this new article, you will find a speed comparation for some of the most used frameworks for e2e testing: Cypress, WDIO and Selenium

Photo by Luke Chesser on Unsplash

Reason for this test

The reason behind this test is because I was responsible for choosing a framework for e2e testing in my company, and after a huge spike, we were deciding to move forward with Cypress, WDIO or maybe Selenium.

So, I decided to do some tests regarding performance, because it was a very important requirement for us. If you want to go directly to the experiment that I ran, move to the My own speed Comparison session.

Background research

Before moving on in our own speed comparison, I decided to do some research with some other comparisons available over the internet.

The first comparison shown in this first article (it is a scientific one), show us that Cypress is quite faster if compared with Selenium WebDriver in JAVA. In this one, which contains only 10 tests cases, we can see that Cypress was faster than Selenium.

Data from the first article

Data from the first article

In another hand, when it comes to WebDriverIO (WDIO), which uses the WebDriverJS, we can see in this second article that in many times WebDriverIO is faster than Cypress. It happens in general because even Cypress runs direct in the browser, the WebDriverJS is surprising optimized. Moreover, Cypress exhibits a longer startup time compared to WDIO. It weighs down short execution scenarios, while it shows much less in longer ones. Therefore, for bigger test suites, WDIO and Cypress may have basically the same overall performance in slow live websites (since waiting times will be basically the same for both).

Data from the this second article

After analyzing these online experiments, there are some good points to have in mind:

• Cypress exhibits a longer startup time compared to WDIO. Compared with WDIO, this weighs down short execution scenarios, while it shows much less in longer ones. Therefore, is not likely to make a real difference in the context of classic E2E testing builds.
• The most important point: local testing in headless mode is what will enable to use Cypress to its fullest speed potential. So, while debugging in a live slow website can look quite slower if compared with WDIO, when the tests are running in pipelines, where the server is locally configured to run across services, containers, and etc., this is where Cypress will be probably much faster than WDIO. Therefore, Cypress will still perform very well in live slow websites, but in local ones (which it is basically how the majority of CI/CD pipelines works) is where it runs faster with its full potential.

My own speed comparison

Test setup:

• A small suite of 10, 100 and 200 test executions.
• Headless mode.
• Website chosen due to its high loading velocity: https://the-internet.herokuapp.com/. The test case selected was the Form Authentication, which ran in loop mode.
• All tests were passing.
• Chrome 88, Firefox 84, and Edge 88.
• The tests were the same, running in loop mode for all frameworks, so we had exactly the same scenario for each one of them.
• Cypress version 6.5.0.
• WDIO version 5.12.5.
• Selenium WebDriver in a Java/maven project, and with the latest drivers.
• The environment was the same.

See the results bellow for Chrome:

See the results bellow for Firefox:

See the results bellow for Edge:

See a summary comparing all the browsers:

Conclusions about this test

We saw that in many articles the results were quite different from the current ones we can see in my own speed comparison. Cypress proved itself to be much faster than WDIO and Selenium in our tests. It probably happened because we are testing against a very responsive live website, so Cypress took advantage on this.

Therefore, according with my tests, we could assure that Cypress is the fastest one for an e2e test, and it is actually the fastest by far. As bigger as the test suites increase, bigger is the difference of time we get while running with Cypress.

Acknowledgments

Beyond me, this article was revised by Larissa Ferreira.

https://www.linkedin.com/in/laertedemesquita/
https://www.linkedin.com/in/larissacamilaferreira/