Cypress parallelization tools and approaches from a high perspective

Introduction:

Upon choosing Cypress as our main framework for e2e testing, we also strived to choose the best strategies regarding test parallelization and test execution tracking. Therefore, I investigated 4 approaches/tools to do so. In this page, we are going to talk about these approaches, which are Cypress Dashboard, Sorry Cypress, Cypress-Parallel, and a manual parallelization approach without load balance within your CI provider.

Cypress Dashboard

Cypress Dashboard is the official tool offered and supported by the Cypress company itself. It brings a complete solution to setup parallelization with automatic load balance and it also brings a very interesting dashboard to follow test executions.

Cypress dashboard main page: Test and debug faster with the Cypress Dashboard

Steps to implement Cypress Dashboard in any CI provider: Parallelization | Cypress Documentation

Price and features: Pricing

With Cypress dashboard, we have:

• Parallelization with Load Balancing controlled by AI
• Debugging and analytics tools
• Video recording and screenshots of everything as much as you want
• Flake detection system
• Jira Integration
• Premium support on Cypress bugs or new features

It is used by big companies like Disney and PayPal. More features are being added to the Cypress Dashboard currently, so if you go over there you will see some new features with a tag “coming soon”.

Sorry Cypress / Currents

Sorry Cypress is probably the best-known workaround solution/tool for Cypress Dashboard. It brings the parallelization and flaky detection system with a way more simple dashboard to follow test executions. Currents is the Sorry Cypress solution in the cloud that provides you with the tool without any setup.

Let’s resume some points by comparing it with Cypress Dashboard:

Pros:

• It is free and unlimited by using your own CI/CD infrastructure and it is supported in many cloud platforms, such as AWS, Azure, and Google Cloud.
• It is not hard to set it up since they provide us a good documentation, including a docker-compose file that you can build in your server and have everything up and working, including a database to keep the data persistent and track test execution history.
• We keep all the data on our side without sending any data to a third-party server. It may be an advantage regarding security.
• The dashboard is simple but it offers the most used tools for test executions such as screenshots, video recording, error stack trace, and flaky detection.
• Currents is the Sorry Cypress cloud-based solution, which is way cheaper than the official Cypress Dashboard solution, and also does not bring any kind of server/cloud support and setup for your part. This solution is in the beta stage (Update: It came out of beta on November 08th, 2021), but looked pretty stable and well-functional during my tests.
• Currents declare to offer an Optimal Sorting feature to smartly distribute tests to reduce overall run time. I sent an email asking for more details about it and the answer was: “We track the duration for each file and when orchestrating a new run, we sort the files to be tested by their expected duration and serve the longest files first. The average duration is the moving average of the last 2 weeks' history.“
• Good integration with some CI providers, like GitHub, GitLab, and Bitbucket. I tested it with Bitbucket and the integration and stability were very good.
• Take a look here to see a demo of Sorry Cypress or test the Currents through the 14-day free trial.

Cons:

• You are required to set it up in your infrastructure in case you want to use it without any cost, which means you may need to have a server or something like that separated for the Sorry Cypress server, and we may need to maintain it as well.
• During some tests in my project, was observed an incompatibility within our Cypress Snapshot Comparison plugin (cypress-plugin-snapshots, version 1.4.4). So, we would need to remove it or look for another snapshot comparison solution/plugin if we decide to move forward with Sorry Cypress.
• It was also detected an incompatibility with the cypress-mochawesome-reporter, in which our Bitbucket pipeline raised an error in the plugins file (cypress/plugins/index.js) in an “after run” hook.
• Cypress version upgrade can take up to 2 weeks on Currents.
• The information regarding the minimum/recommended hardware requirement to run it in our cloud solution was not found.
• The documentation for Currents is not as good if we compare it with Cypress Dashboard.
• Analytics for Currents is good (including for example flaky detection), but is not as good if we compare it with Cypress Dashboard, which brings us much more data in detail.
• The Dashboard UI is very simple yet, so it is not as good as the one we see in Cypress Dashboard.

Cypress-Parallel plugin

The tool called cypress-parallel declares itself to reduce up to 40% of your Cypress suite execution time parallelizing the test run on the same machine. It does not offer any dashboard or management solution, and it only runs tests in parallel by using threads.

Pros:

• Simple to set up and run tests in parallel.

Cons:

• Pretty new solution, basically without any support besides the community (which is very small yet). However, the developer seems to interact a lot with the community on the GitHub page.
• It runs tests in parallel by using the same job instance, which makes it challenging to follow test execution or debug in the logs.
• It was observed an issue while running it in a Bitbucket pipeline, which failed the job even though the test was successful. However, it may be a problem with my cloud infrastructure, so further investigation would be necessary to find an answer to that.
• It also observed an incompatibility within our Cypress Snapshot Comparison plugin (cypress-plugin-snapshots, version 1.4.4).
• At the moment, it just runs tests in parallel, without any tool or dashboard to support executions.
• There is no AI feature to load balance, so the tests are chosen in the order that your spec files are disposed of in your project folders.

Manual CI Approach

This approach is basically to use your own CI provider infrastructure to run your tests in parallel and use the mockawesome reporter as the dashboard or another dashboard provider like Grafana. In this approach, we would split the tests by spec files on “n” machines by using the “spec” flag.

Pros:

• You can use your infrastructure without any plugins, side-server, or tools.
• No need to maintain any side server.
• It is a pretty simple solution, without needing any setup but only your usual pipeline setup.
• The mockawesome reporter can be easily used as a simple dashboard for each execution.
• You don’t exchange your data with any other service, so your data is secured by you (Be aware it may be a pro or a con :)).

Cons:

• You will need to manually split and maintain the specs files scattered/organized over the jobs/machines.
• The load balance is manual by the splitting approach mentioned before.
• The test execution can be followed basically by logs. Grafana or some other dashboard can also be used, but you would be required to set them up entirely.
• In the case of choosing a static reporter, the mockawesome reporter, for example, is generated only at the end of the execution, so there is no way to follow the execution while it is still active.
• As this is a manual approach, you would not be able to guarantee that it would be the fastest execution in parallel (probably it will not). However, depending on the CI provider, it can still be really fast and be worth it.
• While using your infrastructure be aware you will need to keep your artifacts, and depending on your setup it can be expensive. Otherwise, you may not be able to have many runs saved for future comparison and debugging.

Conclusions:

Cypress Dashboard is still by far the most complete solution. However, it is expensive if compared with Sorry Cypress (free), which also brings some of the most relevant features that Cypress Dashboard has to guarantee a good parallelization approach and test management. The cypress-parallel plugin is a pretty new solution, so at this point, we would not consider it.

Besides the manual effort, the Manual CI Approach can also be something you can consider as a backup solution (without any cost) if you decide to move on with Cypress Dashboard or Sorry Cypress/Currents. In this case, you may not even need to use it any time soon, so it may be a good free backup solution in case your main tool/solution is unavailable for some reason.

Currents, from Sorry Cypress, can be something you could also keep in mind, because it brings Sorry Cypress into the cloud, so you will not need to be worried about maintenance and updates. It was very fast and stable during my tests, which makes me very confident about it.

Therefore, if you want the official and most complete solution with total support by Cypress (very good when you find bugs on Cypress), and that it is already used by big companies, and money is not the problem, go on with Cypress Dashboard. However, if you are looking for something that is also very good, and without lots of analytics, but that does the job and offers a pretty decent dashboard as well, go with Sorry Cypress (using your infrastructure) or Currents (for a cloud solution without any setup). Finally, there is always the Manual CI Approach without any financial cost (other than your CI/Infra costs), which may be also interesting to use as a backup solution that maybe will not even be used daily.

Thanks for reading!