Monthly Archives: April 2018

Cypress vs. Selenium, is this the end of an era?

Last Updated on by

Post summary: Blog post about a Cypress talk I did recently on a local conference. The presentation compares Cypress with well knows Selenium.

Background

This weekend I did a small talk about Cypress, named “Cypress vs. Selenium, the end of an era?” on QA Challenge Accepted, a local testing conference. This is my second talk on this conference. In 2016 I spoke about Gatling. I haven’t blogged about my Galing talks because my blog covers the tool very extensively. In Performance testing with Gatling post, there is complete Gatling tutorial. In the current post, I will show most of the slides of my presentation and will describe what I have spoken about. The full Cypress presentation can be found on SlideShare: QA Challenge Accepted 4.0 – Cypress vs. Selenium.

Presentation

Selenium Overview

Selenium is a very well known tool, so I will not get into details about it. I will emphasize on its architecture which will be important for the rest of the presentation. Selenium consists of two components. One is so-called bindings, libraries for different programming languages that we use to write our tests with. The other component is the WebDriver. WebDriver is a program that can manage and fully control a specific browser, for which it is designated. The important bit here is that those two components communicate over HTTP by exchanging JSON payload. This is well defined by WebDriver Protocol, which is W3C Candidate Recommendation. Every command used in tests results to a JSON sent through the network. This network communication happens even if tests are run locally. In this case, requests are sent to localhost behind which there is loopback network interface. Even on localhost request travels to Layer 3 of the OSI Model. Request travels through 5 layers, only layers 1 (physical) and 2 (data link) are skipped.

After the conference, I spoke to Anton Angelov, the founder of Automate The Planet and he pointed out that on some WebDrivers on .NET Core, the resolution from localhost in the request to 127.0.0.1, which is the IPv4 address of loopback interface, can take up to a second. This is, of course, .NET Core specific thing. Anyway, in general, resolving localhost to 127.0.0.1 also needs some time which is added to total execution time.

The bottom line of the slide: By architecture Selenium works through the network and this brings delay, which can sometimes be significant.

Cypress Overview

Cypress is used for UI testing but is not based on Selenium. There are many tools out there which bring a lot of abstractions over the WebDriver, but they are limited to the WebDriver as a technology for browser manipulation. All those tools inherit WebDriver limitations. Cypress has its own mechanism for manipulation DOM in the browser. Cypress runs directly in the browser, no network communication involved. By running directly in the browser Cypress has access to everything in the browser, including your application under test. I do not know a valid reason for this, but in my observations, developers strongly do not like Selenium. Cypress is designed with developers in mind, so it is very developer-friendly. Debugging tests with Cypress is easy, there is so-called travel back in time. I will speak for it later.

The bottom line of the slide: Cypress is made from scratch with its own unique DOM manipulation technology and is made with developers in mind.

How to do it

The bottom line of the slide: It is easy to install Cypress. It is easy to write tests with it. It is very easy to debug tests. It is easy to include it in continuous integration or continuous delivery pipelines.

Debug tests in Cypress Test Runner

Cypress Test Runner is a browser instance in which you see all your tests’ steps on the left-hand side. You can click on any step and in the right-hand side window, the application under test is visualized. Cypress makes DOM snapshot before each test steps, so you can easily inspect them.

The bottom line of the slide: Cypress provides DOM snapshots at each test step for easy test debugging.

Library or Framework

Comparison between both tools now begins. Selenium is a library. If you want to make real UI automation you have to combine it with a unit testing framework or make your own runner; you may want to add assertions library or reporting one. This is handy and gives you great flexibility because if you know what you do you can make miracles. You become a creator! If you don’t know what you do you can very easily shoot yourself in the foot. I think this is mostly because developers hate Selenium. Its usage is not straightforward. In order to start writing tests, you have to do a lot of preparational work. This is something developers do not want to invest in, they invest enough in learning all the frameworks related to their work. They do not want to spend time on several more. Cypress, on the other hand, is a complete framework. You install it and start writing tests. It includes Mocha, very famous JavaScript unit testing framework; Chai is assertions library; Chai-jQuery adds jQuery chainer methods to Chai; Sinon is famous JavaScript mocking library that provides mocks, stubs, and spies; Sinon-Chai brings Chai assertions on stubs and spies.

The bottom line of the slide: Selenium is a library allowing you great flexibility. It requires a lot of preparational work before you can start writing tests. Cypress is a complete framework. You install it and start writing tests.

Test Pyramid

Test pyramid illustrates how your test portfolio should look like. Currently is show a test pyramid for a modern web application. In the bottom are the unit tests. They are fast and test most of the functions in your code. By integration tests is meant the following thing. Defacto the standard now for web applications is to use some JavaScript framework and work with data from APIs. Modern web applications process data from APIs. With integration testing, we want to be in control of this data. We want for e.g. to test how web application behaves when API returns an error. If we have stable and well-tested API this scenario we won’t be able to test in reality. By stubbing the data web application works with it is possible to fully test the web application. UI tests are the one executed against deployed, configured and working web application. They are slow and flaky, thus they are limited in number. Selenium works in the UI part of the pyramid. Cypress is there as well, but Cypress is also very good in Integration tests. The dotted line is mostly a wishful thinking – we have unit testing framework included, why not create some unit tests.

The bottom line of the slide: Selenium works only in the UI part of the test pyramid, while Cypress is involved in UI tests and most important in the integration tests.

Programming languages

There are bindings in almost any programming language existing nowadays. If there is not such, by following WebDriver protocol you can create your own binding. Cypress on the other hand only uses JavaScript and will continue to only use JavaScript. There are two reasons for this. FIrst one is not significant, but it is good your tests code to be as you application under test’s code. The most important reason though is that as I said modern web applications are written in JavaScript frameworks. Developers do know JavaScript, so they can very easily write their own Cypress tests.

The bottom line of the slide: Selenium is available in all programming languages. Developers of modern web application know JavaScript. With Cypress, they can write their own tests.

Selectors

Selenium supports 8 different locators. CSS and XPath are the most powerful ones. Cypress supports jQuery selectors. What you can use as a CSS selector in Selenium, you can directly use as a selector in Cypress. The benefit is that jQuery provides more selectors on hand.

The bottom line of the slide: jQuery selectors give more capabilities than CSS selectors.

Supported Browsers

Selenium supports all significant browsers. You can even create your own browser, make WebDriver for it following WebDriver protocol and your current tests will work exactly the same on this new browser. Cypress at this point supports only Chrome. This is maybe the biggest weakness of the tool. Good thing though is that more than 60% of the web uses Chome. Another good thing is that Firefox support is on its way. IE 11 and Edge support is also on the roadmap but with no clear dates.

The bottom line of the slide: Cypress is weak at cross-browser testing. Cypress team is working through to get better in this area.

Cypress vs. Selenium (1)

Comparison of different characteristics:

  • Speed – Selenium tests are generally slow. WebDriver starting is slow, WebDriver is working slowly. Network operating nature of WebDriver also brings some delay. Cypress is super fast. There is no noticeable delay because of the tool itself.
  • Wait for element – in order to do some effective automation with Selenium, waiting for an element is an important part of your framework. You have to have good error catching mechanism as well as retry logic. It takes significant effort to make tests non-flaky. With Cypress, you do not wait. Cypress runs in the browser and knows what is happening behind the scenes, whether the application under test is still busy. In Cypress, you request the element and you get it when the element is ready, no extra code or logic needed.
  • Remote execution – this is what Selenium is made for. You can use Selenium Grid with different browsers and different browser versions. Cypress does not support remote execution.
  • Parallel execution – Selenium is a library that can manipulate the browser. If you make your code thread safe and use a unit testing framework that supports parallel runs then you will have parallel execution. Selenium does not really care about this. Cypress currently does not support parallel execution. It is possible to do it on your own with Docker images, but this involves additional effort. Currently, Cypress team is working on developing parallel execution, so this will happen soon.
  • Headless – both tools support headless Chrome.

Cypress vs. Selenium (2)

Comparison of different characteristics:

  • Screenshot – both perform equally bad because both make screenshot only of the visible part of the page. In order to get the full page, you need to use external JavaScript libraries to capture page and save it as a screenshot. Selenium is a little bit better on screenshot though, because it gives you screenshot object in your tests and you can save it wherever you want. Cypress makes an automatic screenshot with a fixed name. In order to make second screenshot for one test, you need to do some file manipulations for renaming the previous file.
  • Video – Selenium does not record video. Cypress records video by default when tests are run from command line.
  • Documentation – Selenium documentation for me is ugly and not complete. If one has to get acknowledged with Selenium by reading its documentation only that would be very difficult. Cypress team had invested a lot in the documentation. They have their API well described, they have examples and FAQ page.
  • Community – Selenium is an institution. Everybody is using Selenium. For every problem, you may encounter there are already tens of solutions. Cypress does not have such a community yet, not many people are using it. They have chat though in which Cypress developers answer your queries. This chat gets flooded with information, so you can easily get lost.

Cypress vs. Selenium (3)

Comparison of different characteristics:

  • Execute JS – Selenium allows JavaScript execution and this is fast. I’ve seen frameworks where Selenium is used only for JavaScript execution. Cypress is designed to work with JavaScript. It has full access to everything in the browser, including application under test.
  • Switch tabs – Selenium can switch between two tabs of the same browser, Cypress cannot.
  • Several browsers – Selenium can work with several browser windows, even from different browsers. Cypress can work with only one browser instance.
  • Load extensions – both tools allow you to run your tests with some Chrome extension.
  • Manage cookies – both tools manage cookies equally well.

Test Mushroom

This is some funny, ironic but mostly tragic term I’ve made up. It is made up with an analogy to the test pyramid. This mushroom represents a very common test portfolio where the quality of releases totally depends on UI tests. The mushroom leg represents Unit and Integration tests. It is shown with a dotted line as such test are missing. UI tests are slow and flaky, every release sign off takes a lot of time for debugging failed tests. Every release has a risk of failure.

The bottom line of the slide: There are many real-life scenarios where web application quality depends only on UI tests, which are brittle.Cypress gives you instrumentation to act on integrations test, thus reducing the number of UI ones.

The end of an era?

Now is the time to give an answer to the most interesting part of presentation name. Is this an end of an era? My presentation is not really about the competition between those two tools. Everyone has its strengths and weaknesses. They work perfectly combined together. My main point is and I truly believe it is the end of Developers don’t test era. Selenium may not be their favorite tool, but Cypress is made from developers for developers. It is easy to work with and provides features to speed up test writing. If you try the tool and do not like it, at least try to introduce it to developers in your company.

The bottom line of the slide: Cypress is a tool created by developers for developers. Try to introduce it to developers in your organization.

Cypress Sugar (1)

Those are features that make Cypress interesting tool and that make integration testing easy. Cypress runs directly in the browser and has access to everything in the browser, including application under test. Sometimes Selenium tests go through several pages just to bring the application in some desired state. With Cypress, you can programmatically bring the application to this desired state. Cypress provides spies, stubs, and clocks. With spies, you can verify if given JavaScript function has been called, with what arguments or how many times. Stubs allow you to change the default behavior of JavaScript functions and feed to the application under test the data that you need. For e.g. window.fetch is the new way of getting data from API. This can be very easily stubbed. Cypress provides control over the clock in the browser. If you have some animation, instead of waiting for it, you can move the clock forcing animation to show. Cypress allows you to have full control over the network traffic within the browser. You can assert on XMLHttpRequest to an API, verifying that API is called with proper arguments. You can intercept, change, delay, or block response from the API. This allows you to cover various integration scenarios. With Cypress, you can develop even though there is no backend ready yet. You can do TDD (test driven development), create tests and stub the data from missing API in them, develop the UI and then run the tests until they get green.

The bottom line of the slide: Cypress provides great functionalities for stubbing JavaScript functions and control on network traffic within the browser. Those can be used for creating various integration tests.

Cypress Sugar (2)

The essential functionality of most applications is hidden behind a login. Login through the UI slows up the tests. Cypress allows you to send a login request to the backend and it extracts cookies from the response, injects them in the browser and from now on the user is logged in tests. This request takes browser user agent and existing cookies but skips some security limitations, such as CORS (cross-origin resource sharing). Same can be done with Selenium. You can use some HTTP client, send login request, get the response and inject the cookies in the browser. With Selenium, this requires additional effort though, with Cypress it comes out of the box. Last but not least, with Cypress you can test Electron applications. Electron is a framework which enables you to write desktop applications in HTML, CSS, and JavaScript. Those applications are run within Electron browser, which is based on Chromium and Node.js. A good example of an Electron application is Postman (check Introduction to Postman with examples post). Cypress Test Runner is also an Electron application. And Cypress uses Cypress to test Cypress (Test Runner). Testing of Electron applications is not really a straight-forward task. It requires some amount of functions stubbing.

Conclusion

Cypress is a really great tool. It provides very good features to enable you to create integration tests. I have used Selenium way too much in order to dislike it. Tests with it are slow and flaky. I really hope there is something better out there. On the other hand, I have used Cypress way too little to like it very much and think this is the tool. In any way do try Cypress. If you do not like it, then definitely introduce it to your developers.

Related Posts

Read more...