Running Cucumber tests in parallel

Last Updated on by

Post summary: Details with code samples how to run Cucumber JVM tests in parallel.

Cucumber code examples can be found in selenium-samples-java/cucumber-parallel GitHub repository.

More details what is and how to use Cucumber JVM can be found in Introduction to Cucumber and BDD with examples post.

Why parallel execution?

In general, an automation project starts with a smoke test scenario as a proof of concept. It seems good enough and team start to add tests for different features and modules. Eventually, a team can end up with hundreds of tests taking hours to execute. This becomes a problem as tests can be run only overnight. It is not easy to run tests for an emergency quick fix for example. This is why running tests in parallel is important and it is better to always start an automation project with this in mind.

Running Feature files

In order to run one or several .feature files, an empty class is created. It is annotated with @RunWith(Cucumber.class). If the class name starts or ends with “test” then JUnit automatically runs this class, which then calls Cucumber which is picking feature files to be executed. It is a good idea to put @CucumberOptions annotation also to specify different setting when running the feature files. One such setting is features which allow you to make a runner for each and every feature file.

Running JUnit test in parallel

Maven Surefire plugin is designed for running unit tests. Maven Failsafe plugin is designed for running functional tests and it gracefully handles failures. A good thing is both plugins support running JUnit tests in parallel. In the current example, Maven Surefire plugin is used. Details about its usage are be given below.

Separate runner for each feature file

Knowing that Surefire can run JUnit tests in parallel and feature files are run by an empty JUnit class then best strategy to enable efficiently parallelism is to have many small feature files and to have runner class for each and every feature file. With this approach, there is granularity which can allow many tests to run independently in parallel.

Cucumber and multi-threading

Cucumber reporters are not thread-safe. This means if several parallel runners want to write in one and the same Cucumber report file for sure file will get scrambled. It is mandatory to avoid such cases. This is another requirement to have each runner reporting to separate file.

Automatic runners generation

Having a runner for each feature file in general case implies a lot of copy/pasting. The project will end up with hundreds of empty classes which purpose is just link a feature file. Also, copy/paste is always an error-prone process. The best solution is to generate runners automatically. This can be done with Cucumber JVM parallel plugin. This plugin internally uses Apache Velocity to generate classes based on an internal template file. So it is an option to have the jar locally and modify velocity template inside. Not really recommended. The plugin is included in Maven POM file with following XML fragment:

<plugin>
	<groupId>com.github.temyers</groupId>
	<artifactId>cucumber-jvm-parallel-plugin</artifactId>
	<version>1.0.1</version>
	<executions>
		<execution>
			<id>generateRunners</id>
			<phase>validate</phase>
			<goals>
				<goal>generateRunners</goal>
			</goals>
			<configuration>
				<glue>com.automationrhapsody.cucumber.parallel.tests</glue>
				<featuresDirectory>src/test/resources/com</featuresDirectory>
				<cucumberOutputDir>target/cucumber-parallel</cucumberOutputDir>
				<format>json,html</format>
				<tags>"~@ignored"</tags>
			</configuration>
		</execution>
	</executions>
</plugin>

General option for this plugin can be seen on its homepage. In current example following options are used:

  • glue – this is a comma-separated list of packages where the classes with step definitions can be found. Cucumber looks for all child packages, so if you have step definitions in many different packages a higher level package can be defined, e.g. com.automationrhapsody or defined different packages with a comma.
  • featuresDirectory – directory where feature files can be found. Note that this is very tricky to define. Plugin behaves very weirdly. Given that feature files are in folder src\test\resources\com\automationrhapsody\cucumber\parallel\tests\wikipedia then only src/test/resources/com works for current example. Neither src/test/resources is working, nor src/test/resources/com/automationrhapsody. This is because plugin replaces part of the path with “classpath:”. So this will be a real bummer to set it up correctly.
  • cucumberOutputDir – where to output Cucumber reports.
  • format – Cucumber reports format.
  • tags – features or scenarios with what tags to be run only. The other issue with the plugin is that this setting cannot be empty. If it is empty then it defaults to both “@complete”, “@accepted”. In order to run all features, you can use negation “~@ignored” – run all features without tag @ignored.

This plugin is invoked at Maven lifecycle validate phase. This is the first build phase and it guarantees that Java class files are being generated so they can get compiled later.

Automatic runner class

After the plugin is configured and build is started it produces a class for each feature file with a link to the feature file.

import org.junit.runner.RunWith;

import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;

@RunWith(Cucumber.class)
@CucumberOptions(strict = true,
	features = {"classpath:com/automationrhapsody/cucumber/parallel/tests/wikipedia/ignored.feature"},
	format = {"json:target/cucumber-parallel/1.json", "html:target/cucumber-parallel/1.html", "pretty"},
	monochrome = false,
	tags = {"~@ignored"},
	glue = { "com.automationrhapsody.cucumber.parallel.tests" })
public class Parallel01IT {
}

Configure Maven Surefire plugin

Once plugin that generates Cucumber runners is setup it is time to configure Maven Surefire plugin to run those tests in parallel. This is done with following XML fragment:

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-surefire-plugin</artifactId>
	<version>2.19</version>
	<configuration>
		<forkCount>10</forkCount>
		<reuseForks>true</reuseForks>
		<includes>
			<include>**/Parallel*IT.class</include>
		</includes>
	</configuration>
</plugin>

forkCount can be set up as POM property and changed during runtime. Forking is a special feature that creates separate JVM process. In current example 10 JVM processes will be created. reuseForks actually doesn’t matter if true or false. They are always reused.

includes part is very important. By default, plugin runs only classes that either start or end with test word. Automatically generated classes are with not such name, so they have to be explicitly included in the run with <include>**/Parallel*IT.class</include>.

Fork vs Parallel

Forking creates separate JVM process which is more thread safe as it isolates resources. Still, in some cases, this can cause troubles to the machine running the tests as it requires more resources. Surefire supports another type of parallelism where tests are run in one JVM process but different threads. More details are available at Fork Options and Parallel Test Execution page.

In order to use many threads instead of many JVMs then use <threadCount>10</threadCount> <parallel>classes</parallel> instead of <forkCount>10</forkCount> <reuseForks>true</reuseForks>. Very important is if you use parallel you MUST change <cucumberOutputDir>target/cucumber-parallel</cucumberOutputDir> to <cucumberOutputDir>target</cucumberOutputDir>. This a must because all Cucumber threads try to create cucumber-parallel directory and since Cucumber is not thread safe most of them fail.

Table below shows how parallel and fork manage resources, where following Java code is added to @Before method and results, are observed. Parallel has one process with many threads. The fork has several processes with one thread. As seen in table reuseForks does not have any effect.

@Before
public void before() {
	long threadId = Thread.currentThread().getId();
	String processName = ManagementFactory.getRuntimeMXBean().getName();
	System.out.println("Started in thread: " + threadId + ", in JVM: " + processName);
}

Result is:

forkCount = 10, reuseForks = true
Started in thread: 1, in JVM: 9884@WKS-SOF-L011
Started in thread: 1, in JVM: 11160@WKS-SOF-L011
Started in thread: 1, in JVM: 10892@WKS-SOF-L011
Started in thread: 1, in JVM: 11160@WKS-SOF-L011
Started in thread: 1, in JVM: 10892@WKS-SOF-L011

forkCount = 10, reuseForks = false
Started in thread: 1, in JVM: 8792@WKS-SOF-L011
Started in thread: 1, in JVM: 7884@WKS-SOF-L011
Started in thread: 1, in JVM: 9332@WKS-SOF-L011
Started in thread: 1, in JVM: 8792@WKS-SOF-L011
Started in thread: 1, in JVM: 9332@WKS-SOF-L011

parallel = classes, threadCount = 10
Started in thread: 15, in JVM: 7352@WKS-SOF-L011
Started in thread: 14, in JVM: 7352@WKS-SOF-L011
Started in thread: 15, in JVM: 7352@WKS-SOF-L011
Started in thread: 14, in JVM: 7352@WKS-SOF-L011
Started in thread: 13, in JVM: 7352@WKS-SOF-L011

Tests are run with mvn clean test.

Conclusion

Running Cucumber JVM tests in parallel is essential for successful test automation. Better to start automation project with this thought in mind rather get disappointed at a later stage where tests get impossible to run in a feasible time span. Automatic Cucumber runner classes generation is a good approach as this keeps the project clean and tidy. Using Maven Surefire or Failsafe plugins is the proper way to run already automatically generated runner classes.

Related Posts

Category: Java | Tags: