How to gather code coverage with Istanbul and Selenium and pitfalls to avoid

Last Updated on by

Post summary: Istanbul does not seem to be recoding code coverage correctly, it turned out that the tests do navigation by changing the URL, which resets the code coverage.

How to use Istanbul for code coverage of Cypress automated tests was explained in detail in Testing with Cypress – Custom logging of errors and JUnit results post.

Code coverage with Istanbul and Selenium

Recently I had to do it again, this time with Selenium. There are several approaches, which can be taken to measure code coverage with Selenium. Whichever approach is taken, the first step is to instrument the frontend. How to do it with React and create-react-app is described in Testing with Cypress – Code coverage with Istanbul post. Coverage is present in __coverage__ JS frontend variable.

Once the frontend is instrumented, it is important to collect the code coverage after the tests are run. This is where approaches differ. One option is to use istanbul-middleware. In this case, a Node.js backend has to be created and the tests should post the coverage results, taken from __coverage__ to the backend. I find this approach not convenient, so I took the easier one.

Once the test is finished, the code coverage data is collected and saved as a JSON file in a test results folder, then all the results are used to generate the report. I use C# and the code to do so is as simple as:

public void CollectCodeCoverage()
		var data = ((IJavaScriptExecutor)_webDriver)
			.ExecuteScript("return window.__coverage__");
		if (data != null)
			var jsonString = JsonConvert.SerializeObject(data);
			var fileName = $"{_testResultsFolder}/coverage_{DateTime.Now.Ticks}.json";
			File.WriteAllText(fileName, jsonString);

Generating code coverage report

The report is generated with the nyc cli tool. Once all the JSON files are copied into a folder with the name .nyc_output, the command to run the report is nyc report –reporter=html. Nyc can be installed as a global NPM package or can be added to the frontend project inside package.json.

The issues measuring the code coverage

The setup described above is clear and easy to achieve. Although when tests were run, they did not record coverage, which was supposed to be there. I have spent several days trying to figure out what the issue was. And finally, I was able to understand. In my tests, I use _webDriver.Navigate().GoToUrl(). This actually visits a new URL, basically invalidating all the coverage results gathered so far. Once the problem was identified, the solution was pretty simple – save the cove coverage every time before a new URL is about to get opened.


Istanbul is a very good tool to measure the code coverage for web automation tests. In the current post, I have described a pitfall, which should be avoided when using it.

Related Posts


Dockerize React application with a Docker multi-staged build

Last Updated on by

Post summary: How to build React application inside a Docker container, with a multi-staged build and then run it with NGINX or Caddy.

In the current post, I am not going to compare NGINX vs. Caddy. I will show how to build a React application and package it into a Docker container with both of them. Examples code is located in cypress-testing-framework GitHub repository.


NGINX is open-source software for web serving, reverse proxying, caching, load balancing, media streaming, and more. It started out as a web server designed for maximum performance and stability. In addition to its HTTP server capabilities, NGINX can also function as a proxy server for email (IMAP, POP3, and SMTP) and a reverse proxy and load balancer for HTTP, TCP, and UDP servers.


Caddy is an open-source, HTTP/2-enabled web server written in Go. It uses the Go standard library for its HTTP functionality. One of Caddy’s most notable features is enabling HTTPS by default.


Docker multi-staged building is going to be used in the current post. I have slightly touched the topic in the Optimize the size of Docker images post. The main idea is to optimize the Docker images, so they become smaller. In the current post, I will show two flavors of builds. One is with the standard NPM package manager and is described in Build and run with NGINX section.

The other is with Yarn package manager and is described in Build and run with Caddy section. Current examples are configured to use Yarn. I personally prefer Yarn as for local development it has very effective caching and also it has a reliable dependency locking mechanism.

Build and run with NGINX

Following Dockerfile is describing the building of the React application with NPM package manager and packaging it into NGINX image.

# ========= BUILD =========
FROM node:8.16.0-alpine as builder


COPY package.json .
COPY package-lock.json .
RUN npm install --production

COPY . .

RUN npm run build

# ========= RUN =========
FROM nginx:1.17

COPY conf/nginx.conf /etc/nginx/nginx.conf
COPY --from=builder /app/build /usr/share/nginx/html

The keyword as builder is used to put the name to the image. Both package.json and package-lock.json are copied to the already configured work directory /app. Installation of the packages is done with npm install –production, where the –production switch is used to skip the devDependencies. In the current example, Cypress takes a lot of time to install, and it is not needed for a production build. Afterward, all project files are copied to the image. The files configured in .dockerignore are skipped. All source code files are intentionally copied to the image only after the NPM packages installation. Packages installation takes time, and they need to be installed only if the package.json file has been changed. In case of code changes only, Docker cache is used for the packages layer, this speeds up the build. The build is initiated with npm run build and takes quite a time. Now there the build artifacts are ready. Next stage is to copy the artifacts to nginx:1.17 image into /usr/share/nginx/html folder from builder image’s /app/build folder. Also, NGINX configuration file is copied.

worker_processes auto;
worker_rlimit_nofile 8192;

events {
  worker_connections 1024;

http {
  include /etc/nginx/mime.types;
  sendfile on;
  tcp_nopush on;

  gzip on;
  gzip_static on;
  gzip_proxied any;
  gzip_vary on;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;

  server {
    listen 3000;
    server_name localhost;
    root /usr/share/nginx/html;
    auth_basic off;

    location / {
      try_files $uri $uri/ /index.html;

    # 404 if a file is requested (so the main app isn't served)
    location ~ ^.+\..+$ {
      try_files $uri =404;

I will not go into NGINX configuration details, the configuration can be checked in details in NGINX documentation. Important in the configuration above is that gzip compression is enabled and NGINX listens to port 3000. Then with try_files unknown routes are redirected to index.html, so React can bootstrap the routes.

Build and run with Caddy

Following Dockerfile is describing the building of the React application with Yarn package manager and packaging it into Caddy image.

# ========= BUILD =========
FROM node:8.16.0-alpine as builder


RUN npm install yarn -g

COPY package.json .
COPY yarn.lock .
RUN yarn install --production=true

COPY . .

RUN yarn build

# ========= RUN =========
FROM abiosoft/caddy:1.0.3

COPY conf/Caddyfile /etc/Caddyfile
COPY --from=builder /app/build /usr/share/caddy/html

Absolutely the same logic applies here as above. Yarn is installed as an additional Linux package, then package.json and yarn.lock files are copied. It is very important to copy the yarn.lock, otherwise every run lates dependencies will be fetched, and there might be inconsistent behavior. Only production dependencies are installed with yarn install –production=true. After the application is built with yarn build it is being copied to abiosoft/caddy:1.0.3 image in /usr/share/caddy/html folder from builder image. Caddyfile is copied as well to configure Caddy. {
	log / stdout "{method} {path} {status}"
	root /usr/share/caddy/html
	rewrite {
		regexp .*
		to {path} /

Caddy is configured to listen to port 3000, gzip compression is enabled and there is rewrite rule which redirects unknown paths to the main path, so React can bootstrap the router.


In the current post, I have shown how to build React application inside a Docker image with both NPM and Yarn and then pack the build artifacts to NGINX or Caddy Docker image, which later can be run as a container. This process optimizes the Docker image size and also it does not put extra requirements to the build machine to have Node JS installed, as Node JS is inside the builder image.

Related Posts


Testing with Cypress – Build a React application with Node.js backend

Last Updated on by

Post summary: Short introduction to the application under test that is created for and used in all Cypress examples. It is React frontend created with Create React App package. Backend is a Node.js application running on Express.

This post is part of a Cypress series, you can see all post from the series in Testing with Cypress – lessons learned in a complete framework. Examples code is located in cypress-testing-framework GitHub repository.


The backend is a simple Node.js application build with Express web server. It supports several APIs that can save a person, get a person by id, get all persons or delete the last person in the collection. You can read the full description in Build a REST API with Express on Node.js and run it on Docker post.


Current post is mainly devoted to the frontend. It described how the React application is built. In order to make this part easy, Create React App is used. The best thing about it is that you do not need to handle lots of configurations and you just focus on your application. In order to create an application, Create React App has to be installed as a global NPM package with npm install -g create-react-app. The application itself is created with create-react-app my-application-name. Once this is done you can start building your application. See more details on application creation in How to Create a React App with create-react-app. I have added Bootstrap for better styles and Toastr for nicer notifications. I also use Axios for API calls. I am not going into details about how to work with React as this is a pretty huge topic and I am not really expert at it. You can inspect the GitHub repository given above of how controllers are structured.

Instrumented for code coverage

After having the application ready I wanted to add support for code coverage. The tool used to measure code coverage is Istanbul. Because of Create React App, adding the configuration is not straight-forward as practically there is no webpack.config.js file, it is hidden.

One option is to eject the application. Maybe for a big project where you need full control over the configurations, this is OK, but for this small application, I would not want to deal with it.

Another option is to use a package that builds on top of Create React App. One such plugin is react-app-rewired. It is installed along with istanbul-instrumenter-loader, the actual code coverage plugin. Once those two are installed the actual configuration is pretty simple. A file named config-overrides.js is created with the following content:

const path = require('path');
const fs = require('fs');

module.exports = function override(config, env) {
  // do stuff with the webpack config...
    test: /\.js$|\.jsx$/,
    enforce: 'post',
    use: {
      loader: 'istanbul-instrumenter-loader',
      options: {
        esModules: true
    include: path.resolve(fs.realpathSync(process.cwd()), 'src')
  return config;

Also, package.json has to be changed. The default react-scripts start/build/test is changed to react-app-rewired start/build/test. In order to verify that code coverage is enabled, go to Dev Tools (hit keyboard F12), then go to Console and search for __coverage__ variable.


In order to make it easy to run a Dockerfile has been added. It installs Yarn as a package manager, then copies package.json. Important is to copy yarn.lock as well since the actual dependencies are in it. If this is not copied, every time an install is run it will pick the latest dependencies, which may lead to instability. Then the installation of dependencies is done with command yarn, short for yarn install. Finally, all local files are copied. This is done in the end so installation is not triggered on every file change, but only on package.json or yarn.lock change.

FROM node:8.16.0-alpine

ENV APP /app

RUN npm install yarn -g

COPY package.json $APP
COPY yarn.lock $APP
RUN yarn

COPY . .

The docker-compose.yml file is also very simple. It has two services. The first is the backend which is exposed to 9000 port of the host. This is needed because Cypress tests directly access the APIs. It uses the image uploaded to the Docker hub repository: image: llatinov/nodejs-rest-stub. The second service is the frontend. It uses local Dockerfile: build: .. When frontend container is started yarn start command is executed and is exposed to port 3030 of the host machine. One more thing, that is added as configuration, is the backend API URL that can be controlled by setting API_URL environment variable, which then is set to REACT_APP_API_URL, used by the frontend. If no API_URL is provided then the default of http://localhost:9000 is taken.

version: '3'

    image: llatinov/nodejs-rest-stub
      - '9000:3000'
    build: .
    command: yarn start
      - REACT_APP_API_URL=${API_URL:-http://localhost:9000}
      - '3030:3000'

Run the application

There are several ways to run the application under test in order to try Cypress examples. One way is to download both repositories of the backend and the frontend and run them separately.

Second is to run the backend with Docker command docker run -p 9000:3000 llatinov/nodejs-rest-stub. The command maps the 3000 port of the container to 9000 port of the host, this is where the APIs are available. I have uploaded the backend image to the public Docker hub repository. After backend is running, the frontend is run with yarn start command. In this case, frontend is running on port 3000, so you have to adjust the proper URL in the Cypress configurations.

The third option is to run with docker-compose with docker-compose up command. This runs the backend on port 9000 and the frontend on port 3030.


The application is very simple, it has few pages where user can add a person or see already existing persons in the backend. On each successful action, there is a notification, in case of a network error, a message is shown.

Persons list

Add person

Version page


In order to demonstrate the Cypress examples, a separate React application with a backend is created with Create React App package. It is also configured to support code coverage with Istanbul.

Related Posts