Post summary: In the current post I will share some lessons I’ve learned using Cypress for quite a long time. Along this journey, I created a framework which solves some of the pain points that Cypress has.
Introduction
More than a year ago I made a bold presentation about Cypress. Back then I had been using Cypress on a small and very nice React application, and I was fascinated by the tool. You can read the presentation content in Cypress vs. Selenium, is this the end of an era? post. Now more than a year later and 10K lines of test code I am still fascinated by Cypress and also I have discovered several things that were causing me pain during my work. In the current post, I will try to write for some of them, some of them I truly had forgotten. In the course of using Cypress, I had decided to change things I do not like and make them in the way I really enjoy it. The result of this is a framework, maybe this is too overrated, more likely a set of helper files which you can pick and directly use in your project. The code is located in cypress-testing-framework GitHub repository.
Post in the series
This is the first of series of posts dedicated to testing with Cypress and making your tests easier to write. All posts from the series are:
- Testing with Cypress – Build a React application with Node.js backend
- Testing with Cypress – Basic API overview
- Testing with Cypress – Custom logging of errors and JUnit results
- Testing with Cypress – Code coverage with Istanbul
Application under test
In order to demonstrate some of the features, I have built a very simple React application. It has a backend that manipulates the data and the React application is consuming the backend APIs. More about the application itself can be found in Testing with Cypress – Build a React application with Node.js backend post.
Cypress API
Cypress has a rich API, offering lots of functionality. So far many of us are very used to Selenium, and it is a little surprise when you first deal with Cypress. There is some ramp-up time needed. Once you get acknowledged, things start to happen pretty fast and easy. Read more about the API along with some examples Testing with Cypress – Basic API overview post.
Page Object Model
Cypress does not recommend using POM but prefers using Cypress custom commands instead. See а very good and justified post on the topic, named Stop using Page Objects and Start using App Actions. Although the justification seems very logical, I do not agree with that approach. I still use custom commands, but not as a replacement of page objects, I am not giving up the Page Object Model. It gives me more focus, while with custom commands you can easily start duplicate functionality. Check an example of a Page Object Model I am comfortable with in Testing with Cypress – Basic API overview post.
Test Runner going out of memory
This is my biggest pain. I have tried a lot to overcome this but I could not find any solution. It happens in case of a long test suite with lots of actions in it. Cypress keeps a before/after version of the page on every action, memory drains pretty fast and the browser crashes with Aw snap error.
The most recommended option is to use numTestsKeptInMemory to reduce the memory footprint, but then you need it to be at least one, so you can debug and inspect data into the console.
I also tried to pass –max-old-space-size to Node process. If you pass it to Cypress directly it crashes, so what I did was to rename the node executable to node_exec, and then create a new file named node in which I put node_exec –max-old-space-size $@ to forward all arguments to node executable. This did not help either.
Finally, I settled with the option to have custom commands to locate elements, which suppress more of the logging with {log: false}. Before/after version of locating the element is not needed, a snapshot is needed after a click or other significant action. Note that this log: false gave me a hard time when using cy.get because it was resetting the default timeout, so I had to pass the timeout as an option as well.
Cypress.Commands.add('getSilent', locator =>
cy.get(locator, {
log: false,
timeout: Cypress.config('defaultCommandTimeout'),
}),
);
This workaround did not solve the out of memory issue either, just allowed me to have a longer scenario before the Test Runner crashes.
On the other hand, this limitation is kind of a motivation for you to plan better, make more focused and short test suites.
Cypress error logging and JUnit results
Cypress does not provide very good logging, the stack trace is practically useless, as your code is wrapped into Cypress’ code. In order to work around this, I use some custom code which collects Cypress commands and then when a test fails it dumps the commands to a custom log file and a screenshot. The same code also creates custom JUnit test result files and it inserts the errors collected. Custom files are saved into cypress/logs folder of the project. You can read more about this custom logging in Testing with Cypress – Custom logging of errors and JUnit results post.
Rerun failed tests
Although Cypress is very stable, it still happens that some tests fail from time to time. I have added a task to rerun failed tests. This is done with yarn cypress:retry. This task iterates all custom created JUnit XMLs described in the previous section and makes a list of all tests that had failed. This list is saved into a file named retry-output.txt in cypress/logs folder. Those files are run again. The internal command that is called by retry code is yarn cypress:run –spec=’cypress/tests/TestSuite.js’. The same command you can use manually to run a single test suite or more using an asterisk as a wildcard.
Code coverage
Code coverage is not mandatory, more likely a nice to have a metric, we try to monitor and improve on. Read more about code coverage in What about code coverage post. For capturing code coverage Istanbul is used. Code coverage is described in more details in Testing with Cypress – Code coverage with Istanbul post.
Generate reports
An HTML report is generated in the end, it is invoked with yarn cypress:report command. This command relies on custom JUnit XMLs generated during the test run. You can read more details in Testing with Cypress – Custom logging of errors and JUnit results post.
Running tests in parallel
Cypress supports running tests in parallel. This is done with –parallel option when you run your tests. In order to do so, you need a subscription to Cypress Dashboard. There are various subscription plans, which are quite affordable. The idea is that Cypress records all your test runs and based on the timing and the available machines, it distributes evenly the tests across your machines. You can read more in Cypress Parallelization article. I have not tried that and also I do not know how it is going to work with current customizations I am doing in the current post.
Another option is to do the parallelism on your own. For this purpose, xargs Linux command can be used. The command that you run under Linux is:
find ./cypress/tests -name "*_spec.js" | xargs -n1 -P4 bash -c 'yarn cypress:run --spec="$@"' --
Where the P4 is the number you threads you want to have. The command finds all files ending with _spec.js with each if it, it invokes Cypress with a given number of simultaneous threads. Note that this parallelization is not very stable in case of Docker container. Randomly there are issues with Xvfb frame buffer.
What worked for me is to have a docker-compose-yml file with several Cypress services. Each one of them is running a group of the tests, which I manually split. All services share the same volume so results are kept in one place. After those services finish, then another service is run which retries failed tests and aggregates the results and the code coverage. This service is sharing the same volume so it has access to all the test results.
End to end process
To put the bits together. The process suggested in the current post consists of the following steps:
- yarn cypress:run – run the tests. During the run JUnit XML files are generated. In order to speed up tests can be run in parallel as well. Set TEST_CODE_COVERAGE=true is code coverage is needed.
- yarn cypress:retry – retry failed tests, based on the JUnit XMLs generated from the previous step. You can retry twice if you need to.
- yarn cypress:report – generate code coverage report, HTML report with results and also semaphore file that indicates if the tests passed or not.
Conclusion
Cypress is a great tool, I strongly recommend it. It is very stable and reliable. With the improvements, you can find with this series of posts, you can make automation with Cypress even more effective, reliable and enjoyable. Very good article with useful Cypress tips is Bahmutov’s Cypress tips and tricks, I suggest you read it as well.
