Running Cucumber tests in parallel

Last Updated on by

Post summary: Details with code samples how to run Cucumber JVM tests in parallel.

Cucumber code examples can be found in selenium-samples-java/cucumber-parallel GitHub repository.

More details what is and how to use Cucumber JVM can be found in Introduction to Cucumber and BDD with examples post.

Why parallel execution?

In general an automation project starts with a smoke test scenario as a proof of concept. It seems good enough and team start to add tests for different features and modules. Eventually team can end up with hundreds of tests taking hours to execute. This becomes a problem as tests can be run only overnight. It is not easy to run tests for emergency quick fix for example. This is why running tests in parallel is important and it is better to always start an automation project with this in mind.

Running Feature files

In order to run one or several .feature files an empty class is created. It is annotated with @RunWith(Cucumber.class). If the class name starts or ends with “test” then JUnit automatically runs this class, which then calls Cucumber which is picking feature files to be executed. It is good idea to put @CucumberOptions annotation also to specify different setting when running the feature files. One such setting is features which allows you to make runner for each and every feature file.

Running JUnit test in parallel

Maven Surefire plugin is designed for running unit tests. Maven Failsafe plugin is designed for running functional tests and it gracefully handles failures. Good thing is both plugins support running JUnit tests in parallel. In current examples Surefire plugin is used. Details about its usage are be given bellow.

Separate runner for each feature file

Knowing that Surefire can run JUnit tests in parallel and knowing that feature files are run by an empty JUnit class then best strategy to enable efficiently parallelism is to have many small feature files and to have runner class for each and every feature file. With this approach there is granularity which can allow many tests to run independently in parallel.

Cucumber and multi-threading

Cucumber reporters are not thread safe. This means if several parallel runners want to write in one and the same Cucumber report file for sure file will get scrambled. It is mandatory to avoid such cases. This is another requirement to have each runner reporting to separate file.

Automatic runners generation

Having a runner for each feature file in general case implies a lot of copy/pasting. Project will end up with hundreds of empty classes which purpose is just just link a feature file. Also copy/paste is always error prone process. Best solution is to generate runners automatically. This can be done with Cucumber JVM parallel plugin. This plugin internally uses Apache Velocity to generate classes based on internal template file. So it is an option to have the jar locally and modify velocity template inside. Not really recommended. Plugin is included in Maven POM file with following XML fragment:

<plugin>
	<groupId>com.github.temyers</groupId>
	<artifactId>cucumber-jvm-parallel-plugin</artifactId>
	<version>1.0.1</version>
	<executions>
		<execution>
			<id>generateRunners</id>
			<phase>validate</phase>
			<goals>
				<goal>generateRunners</goal>
			</goals>
			<configuration>
				<glue>com.automationrhapsody.cucumber.parallel.tests</glue>
				<featuresDirectory>src/test/resources/com</featuresDirectory>
				<cucumberOutputDir>target/cucumber-parallel</cucumberOutputDir>
				<format>json,html</format>
				<tags>"~@ignored"</tags>
			</configuration>
		</execution>
	</executions>
</plugin>

General option for this plugin can be seen in its home page. In current example following options are used:

  • glue – this is comma separated list of packages where the classes with step definitions can be found. Cucumber looks for all child packages, so if you have step definitions in many different packages a higher level package can be defined, e.g. com.automationrhapsody or defined different packages with comma.
  • featuresDirectory – directory where feature files can be found. Note that this is very tricky to define. Plugin behaves very weird. Given that feature files are in folder src\test\resources\com\automationrhapsody\cucumber\parallel\tests\wikipedia then only src/test/resources/com works for current example. Neither src/test/resources is working, nor src/test/resources/com/automationrhapsody. This is because plugin replaces part of the path with “classpath:”. So this will be a real bummer to set it up correctly.
  • cucumberOutputDir – where to output Cucumber reports.
  • format – Cucumber reports format.
  • tags – features or scenarios with what tags to be run only. The other issue with plugin is that this setting cannot be empty. If it is empty then it defaults to both “@complete”, “@accepted”. In order to run all features you can use negation “~@ignored” – run all features without tag @ignored.

This plugin is invoked at Maven lifecycle validate phase. This is first build phase and it guarantees that Java class files are being generated so they can get compiled later.

Automatic runner class

After plugin is configured and build is started it produces a class for each feature file with link to the feature file.

import org.junit.runner.RunWith;

import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;

@RunWith(Cucumber.class)
@CucumberOptions(strict = true,
	features = {"classpath:com/automationrhapsody/cucumber/parallel/tests/wikipedia/ignored.feature"},
	format = {"json:target/cucumber-parallel/1.json", "html:target/cucumber-parallel/1.html", "pretty"},
	monochrome = false,
	tags = {"~@ignored"},
	glue = { "com.automationrhapsody.cucumber.parallel.tests" })
public class Parallel01IT {
}

Configure Maven Surefire plugin

Once plugin that generates Cucumber runners is setup it is time to configure Maven Surefire plugin to run those tests in parallel. This is done with following XML fragment:

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-surefire-plugin</artifactId>
	<version>2.19</version>
	<configuration>
		<forkCount>10</forkCount>
		<reuseForks>true</reuseForks>
		<includes>
			<include>**/Parallel*IT.class</include>
		</includes>
	</configuration>
</plugin>

forkCount can be setup as POM property and changed during runtime. Forking is special feature that creates separate JVM process. In current example 10 JVM will be created. reuseForks actually doesn’t matter if true or false. They are always reused.

includes part is very important. By default plugin runs only classes that either start or end with “test” word. Automatically generated classes are with not such name, so they have to be explicitly included in the run with <include>**/Parallel*IT.class</include>.

Fork vs Parallel

Forking creates separate JVM process which is more thread safe as it isolates resources. Still in some cases this can cause troubles to machine running the tests as it requires more resources. Surefire supports other type of parallelism where tests are run in one JVM process but different threads. More details are available at Fork Options and Parallel Test Execution page.

In order to use many threads instead of many JVMs then <forkCount>10</forkCount> <reuseForks>true</reuseForks> should be changed to <threadCount>10</threadCount> <parallel>classes</parallel>. Very important is if you use parrallel you MUST change <cucumberOutputDir>target/cucumber-parallel</cucumberOutputDir> to <cucumberOutputDir>target</cucumberOutputDir>. This a must because all Cucumber threads try to create cucumber-parallel directory and since Cucumber is not thread safe most of them fail.

Table bellow shows how parallel and fork manage resources where following Java code is added to @Before method and results are observed. Parallel has one process with many threads. Fork has several processes with one thread. As seen in table reuseForks does not have any effect.

long threadId = Thread.currentThread().getId();
String processName = ManagementFactory.getRuntimeMXBean().getName();
System.out.println("Started in thread: " + threadId + ", in JVM: " + processName);

Result is:

forkCount = 10, reuseForks = true
Started in thread: 1, in JVM: 9884@WKS-SOF-L011
Started in thread: 1, in JVM: 11160@WKS-SOF-L011
Started in thread: 1, in JVM: 10892@WKS-SOF-L011
Started in thread: 1, in JVM: 11160@WKS-SOF-L011
Started in thread: 1, in JVM: 10892@WKS-SOF-L011

forkCount = 10, reuseForks = false
Started in thread: 1, in JVM: 8792@WKS-SOF-L011
Started in thread: 1, in JVM: 7884@WKS-SOF-L011
Started in thread: 1, in JVM: 9332@WKS-SOF-L011
Started in thread: 1, in JVM: 8792@WKS-SOF-L011
Started in thread: 1, in JVM: 9332@WKS-SOF-L011

parallel = classes, threadCount = 10
Started in thread: 15, in JVM: 7352@WKS-SOF-L011
Started in thread: 14, in JVM: 7352@WKS-SOF-L011
Started in thread: 15, in JVM: 7352@WKS-SOF-L011
Started in thread: 14, in JVM: 7352@WKS-SOF-L011
Started in thread: 13, in JVM: 7352@WKS-SOF-L011

Conclusion

Running Cucumber JVM tests in parallel is essential for successful test automation. Better to start automation project with this thought in mind rather to get disappointed at later stage where tests get impossible to run in a feasible time span. Automatic Cucumber runner classes generation is a good approach as this keeps the project clean and tidy. Using Maven Surefire or Failsafe plugins is the proper way to run already automatically generated runner classes.

If you find this post useful, please share it to reach more people. Sharing is caring!
Share on FacebookShare on LinkedInTweet about this on TwitterShare on Google+Email this to someone
Category: Java | Tags:
  • Pankaj

    Hi,

    I have implemented the same plugin ‘cucumber-jvm-parallel-plugin’ in my pom.xml. Also the build got succeeded but its not creating the TestRunner classes for each feature file. Could you please help

    Thanks in Advance

    • Lyudmil Latinov

      Try:
      com.???
      src/test/resources/com
      target/cucumber-parallel

      Put some package for the glue.

  • Rajiv

    same issue I am facing. Not able to run features files from the pom.xml using temyers/cucumber-jvm-parallel-plugin: My pom.xml file is -: No single file being picked.

    com.github.temyers
    cucumber-jvm-parallel-plugin
    2.2.0

    generateRunners
    validate

    generateRunners

    src.test.java.StepDef.stepdef_desktop.HomePageStepDef
    src/test/resources/pageTests/
    target/cucumber
    html
    true
    true
    false

    • Rajiv

      I checked in target folder that feature file and java class files, both are exit in ‘pageTests’ folder.

    • Lyudmil Latinov

      Glue should be package. Like: StepDef.stepdef_desktop.HomePageStepDef or StepDef.stepdef_desktop

  • Moose

    Greetings!

    I am trying to implement the automatic runner plugin in my project but I am having a lot of difficulties with it. I have added everything in the pom.xml file, and am running the features through a runner class I have created. it doesn’t seem like it is even accounting for the things on the pom file, and whenever I run it, it just opens one browser and runs all the features one by one. Isn’t it supposed to open the browser multiple times and run the different features at the same time? Because currently it is going through them one by one.

    Cucumber options in the runner class:
    @CucumberOptions(strict = true,
    features = {“src/test/java/portal/propertyadmin/”},
    format = {“pretty”, “html:cucumber-html-reports”,
    “json: cucumber-html-reports/cucumber.json”},
    monochrome = false,
    glue = {“binders”})

    The pom configuration within the

    com.github.temyers
    cucumber-jvm-parallel-plugin
    2.1.0

    generateRunners
    validate

    generateRunners

    binders.???
    src/test/java/portal/propertyadmin/
    target/cucumber-parallel
    json,html
    “~@ignore”

    org.apache.maven.plugins
    maven-surefire-plugin
    2.19

    10
    true

    **/PortalPropertyAdminTestRunner.class

  • Anand JOis

    Please help!!!
    Do you the same project done in gradle?

    • Lyudmil Latinov

      Unfortunately I don’t and also I think it is not possible to have it in Gradle as https://github.com/temyers/cucumber-jvm-parallel-plugin does not have plugin for Gradle, just for Maven. I suggest to contact the maintainer of the plugin to understand future plans.

  • Bindhu Madhavi

    Some help please?
    multiple feature files are generated in target but looks like its not being picked while running and I could only see the tests running in sequence

    org.apache.maven.plugins
    maven-compiler-plugin

    1.7
    1.7

    com.github.temyers
    cucumber-jvm-parallel-plugin
    4.0.0

    generateRunners
    generate-test-sources

    generateRunners

    stepdefinitions
    ${project.build.directory}/generated-test-sources/cucumber
    src/test/java/features
    ${project.build.directory}/cucumber-parallel

    json

    com.example.CustomHtmlFormatter
    html

    true
    true

    @wip

    false
    SCENARIO

    org.apache.maven.plugins
    maven-surefire-plugin
    2.19.1

    test

    all
    true
    true
    2C
    false

    **/*Runner.class

    Could someone please help me in understanding where am going wrong. FYI am using cucumber-spring dependency injection in my project

    info.cukes
    cucumber-spring
    1.2.5
    test

  • vikram

    Currently I’m using maven-failsafe-plugin to run multiple feature files in parallel with Selenium Grid + nodes ( all running in docker containers )

    I’m basic questions as below

    when to use cucumber-jvm-parallel-plugin ?
    what benefits / disadvantages of cucumber-jvm-parallel-plugin over maven-failsafe-plugin parallel execution ?
    Thanks in advance.

    • Lyudmil Latinov

      In examples above maven-surefire-plugin is used. It is also the same as maven-failsafe-plugin and it gives the ability to run the tests in parallel. Despite its name cucumber-jvm-parallel-plugin does not run any tests, it just automatically generates the Cucumber runners for you. This saves you time from making empty runner files and also just by changing the config of the cucumber-jvm-parallel-plugin you can have brand new set of runner files.

  • vikram