Create Cucumber JVM custom formatter

Last Updated on by

Post summary: How to implement custom formatter for Cucumber JVM.

Cucumber code examples can be found in selenium-samples-java/cucumber-parallel GitHub repository.

Cucumber has different types of formatters available. See Cucumber formatters for more details. Still those might not fulfil your needs. If this is the case Cucumber provides a way to create own custom formatter.

Why to create Cucumber JVM custom formatter

There could be different reasons for this. Standard formatters are know not to be thread safety, so one reason could be to create own thread safety formatter. Other reason could be if you want you results to be stored directly into database or sent to a REST service. There could be many reasons, reading this post means you already have one.

How to create Cucumber JVM custom formatter

Custom formatter can be created by implementing two interfaces: gherkin.formatter.Reporter and gherkin.formatter.Formatter. This is it. You have to know though when each method from interfaces is called and this is pretty tricky. One simple implementation of custom formatter can be found in GitHub CustomFormatter.java file. This is implementation from scratch. Other option is to extend some already existing formatter and modify its behaviour.

Using custom formatter in Cucumber runner

Custom formatter is invoked by fully qualified class name in @CucumberOptions annotation. In this example it is: plugin = {“com.automationrhapsody.cucumber.formatter.CustomFormatter”}.

import org.junit.runner.RunWith;

import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;

@RunWith(Cucumber.class)
@CucumberOptions(
		plugin = {
				"com.automationrhapsody.cucumber.formatter.CustomFormatter:custom-formatter-output.txt"
		}
)
public class RunWikipediaTestsWithCustomFormatter {
}

Passing argument to custom formatter

In code above along with fully qualified class name of CustomFormatter there is also file name (and location) with colon – :custom-formatter-output.txt.  This is the way to pass argument to CustomFormatter. In order to receive the argument CustomFormatter constructor should have one of the following parameters:

  • null – empty constructor, this is just to instantiate some variable, nothing can be received from outside.
  • interface java.lang.Appendable – object to which values should be appended. It should have implementation which can be custom or gherkin.formatter.NiceAppendable can be used. See example bellow.
  • class java.net.URI – URI that can be used in case of database connection strings or URL. Also applicable for passing a plain String.
  • class java.net.URL – only for web URL.
  • class java.io.File – file path can be passed, could be absolute or relative.

Example:

public CustomFormatter(Appendable appendable) {
	output = new NiceAppendable(appendable);
	output.println("CustomFormatter()");
	System.out.println("CustomFormatter(): " + output.toString());
}

Conclusion

Cucumber JVM gives lot of flexibility by providing a way to implement custom formatter based on your current needs. It is pretty straightforward to do it. It is possible also to pass arguments to a parametrised constructor in custom formatter.

Read more...

Running Cucumber tests in parallel

Last Updated on by

Post summary: Details with code samples how to run Cucumber JVM tests in parallel.

Cucumber code examples can be found in selenium-samples-java/cucumber-parallel GitHub repository.

More details what is and how to use Cucumber JVM can be found in Introduction to Cucumber and BDD with examples post.

Why parallel execution?

In general an automation project starts with a smoke test scenario as a proof of concept. It seems good enough and team start to add tests for different features and modules. Eventually team can end up with hundreds of tests taking hours to execute. This becomes a problem as tests can be run only overnight. It is not easy to run tests for emergency quick fix for example. This is why running tests in parallel is important and it is better to always start an automation project with this in mind.

Running Feature files

In order to run one or several .feature files an empty class is created. It is annotated with @RunWith(Cucumber.class). If the class name starts or ends with “test” then JUnit automatically runs this class, which then calls Cucumber which is picking feature files to be executed. It is good idea to put @CucumberOptions annotation also to specify different setting when running the feature files. One such setting is features which allows you to make runner for each and every feature file.

Running JUnit test in parallel

Maven Surefire plugin is designed for running unit tests. Maven Failsafe plugin is designed for running functional tests and it gracefully handles failures. Good thing is both plugins support running JUnit tests in parallel. In current examples Surefire plugin is used. Details about its usage are be given bellow.

Separate runner for each feature file

Knowing that Surefire can run JUnit tests in parallel and knowing that feature files are run by an empty JUnit class then best strategy to enable efficiently parallelism is to have many small feature files and to have runner class for each and every feature file. With this approach there is granularity which can allow many tests to run independently in parallel.

Cucumber and multi-threading

Cucumber reporters are not thread safe. This means if several parallel runners want to write in one and the same Cucumber report file for sure file will get scrambled. It is mandatory to avoid such cases. This is another requirement to have each runner reporting to separate file.

Automatic runners generation

Having a runner for each feature file in general case implies a lot of copy/pasting. Project will end up with hundreds of empty classes which purpose is just just link a feature file. Also copy/paste is always error prone process. Best solution is to generate runners automatically. This can be done with Cucumber JVM parallel plugin. This plugin internally uses Apache Velocity to generate classes based on internal template file. So it is an option to have the jar locally and modify velocity template inside. Not really recommended. Plugin is included in Maven POM file with following XML fragment:

<plugin>
	<groupId>com.github.temyers</groupId>
	<artifactId>cucumber-jvm-parallel-plugin</artifactId>
	<version>1.0.1</version>
	<executions>
		<execution>
			<id>generateRunners</id>
			<phase>validate</phase>
			<goals>
				<goal>generateRunners</goal>
			</goals>
			<configuration>
				<glue>com.automationrhapsody.cucumber.parallel.tests</glue>
				<featuresDirectory>src/test/resources/com</featuresDirectory>
				<cucumberOutputDir>target/cucumber-parallel</cucumberOutputDir>
				<format>json,html</format>
				<tags>"~@ignored"</tags>
			</configuration>
		</execution>
	</executions>
</plugin>

General option for this plugin can be seen in its home page. In current example following options are used:

  • glue – this is comma separated list of packages where the classes with step definitions can be found. Cucumber looks for all child packages, so if you have step definitions in many different packages a higher level package can be defined, e.g. com.automationrhapsody or defined different packages with comma.
  • featuresDirectory – directory where feature files can be found. Note that this is very tricky to define. Plugin behaves very weird. Given that feature files are in folder src\test\resources\com\automationrhapsody\cucumber\parallel\tests\wikipedia then only src/test/resources/com works for current example. Neither src/test/resources is working, nor src/test/resources/com/automationrhapsody. This is because plugin replaces part of the path with “classpath:”. So this will be a real bummer to set it up correctly.
  • cucumberOutputDir – where to output Cucumber reports.
  • format – Cucumber reports format.
  • tags – features or scenarios with what tags to be run only. The other issue with plugin is that this setting cannot be empty. If it is empty then it defaults to both “@complete”, “@accepted”. In order to run all features you can use negation “~@ignored” – run all features without tag @ignored.

This plugin is invoked at Maven lifecycle validate phase. This is first build phase and it guarantees that Java class files are being generated so they can get compiled later.

Automatic runner class

After plugin is configured and build is started it produces a class for each feature file with link to the feature file.

import org.junit.runner.RunWith;

import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;

@RunWith(Cucumber.class)
@CucumberOptions(strict = true,
	features = {"classpath:com/automationrhapsody/cucumber/parallel/tests/wikipedia/ignored.feature"},
	format = {"json:target/cucumber-parallel/1.json", "html:target/cucumber-parallel/1.html", "pretty"},
	monochrome = false,
	tags = {"~@ignored"},
	glue = { "com.automationrhapsody.cucumber.parallel.tests" })
public class Parallel01IT {
}

Configure Maven Surefire plugin

Once plugin that generates Cucumber runners is setup it is time to configure Maven Surefire plugin to run those tests in parallel. This is done with following XML fragment:

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-surefire-plugin</artifactId>
	<version>2.19</version>
	<configuration>
		<forkCount>10</forkCount>
		<reuseForks>true</reuseForks>
		<includes>
			<include>**/Parallel*IT.class</include>
		</includes>
	</configuration>
</plugin>

forkCount can be setup as POM property and changed during runtime. Forking is special feature that creates separate JVM process. In current example 10 JVM will be created. reuseForks actually doesn’t matter if true or false. They are always reused.

includes part is very important. By default plugin runs only classes that either start or end with “test” word. Automatically generated classes are with not such name, so they have to be explicitly included in the run with <include>**/Parallel*IT.class</include>.

Fork vs Parallel

Forking creates separate JVM process which is more thread safe as it isolates resources. Still in some cases this can cause troubles to machine running the tests as it requires more resources. Surefire supports other type of parallelism where tests are run in one JVM process but different threads. More details are available at Fork Options and Parallel Test Execution page.

In order to use many threads instead of many JVMs then <forkCount>10</forkCount> <reuseForks>true</reuseForks> should be changed to <threadCount>10</threadCount> <parallel>classes</parallel>. Very important is if you use parrallel you MUST change <cucumberOutputDir>target/cucumber-parallel</cucumberOutputDir> to <cucumberOutputDir>target</cucumberOutputDir>. This a must because all Cucumber threads try to create cucumber-parallel directory and since Cucumber is not thread safe most of them fail.

Table bellow shows how parallel and fork manage resources where following Java code is added to @Before method and results are observed. Parallel has one process with many threads. Fork has several processes with one thread. As seen in table reuseForks does not have any effect.

long threadId = Thread.currentThread().getId();
String processName = ManagementFactory.getRuntimeMXBean().getName();
System.out.println("Started in thread: " + threadId + ", in JVM: " + processName);

Result is:

forkCount = 10, reuseForks = true
Started in thread: 1, in JVM: 9884@WKS-SOF-L011
Started in thread: 1, in JVM: 11160@WKS-SOF-L011
Started in thread: 1, in JVM: 10892@WKS-SOF-L011
Started in thread: 1, in JVM: 11160@WKS-SOF-L011
Started in thread: 1, in JVM: 10892@WKS-SOF-L011

forkCount = 10, reuseForks = false
Started in thread: 1, in JVM: 8792@WKS-SOF-L011
Started in thread: 1, in JVM: 7884@WKS-SOF-L011
Started in thread: 1, in JVM: 9332@WKS-SOF-L011
Started in thread: 1, in JVM: 8792@WKS-SOF-L011
Started in thread: 1, in JVM: 9332@WKS-SOF-L011

parallel = classes, threadCount = 10
Started in thread: 15, in JVM: 7352@WKS-SOF-L011
Started in thread: 14, in JVM: 7352@WKS-SOF-L011
Started in thread: 15, in JVM: 7352@WKS-SOF-L011
Started in thread: 14, in JVM: 7352@WKS-SOF-L011
Started in thread: 13, in JVM: 7352@WKS-SOF-L011

Conclusion

Running Cucumber JVM tests in parallel is essential for successful test automation. Better to start automation project with this thought in mind rather to get disappointed at later stage where tests get impossible to run in a feasible time span. Automatic Cucumber runner classes generation is a good approach as this keeps the project clean and tidy. Using Maven Surefire or Failsafe plugins is the proper way to run already automatically generated runner classes.

Read more...

Introduction to Cucumber and BDD with examples

Last Updated on by

Post summary: Code examples and introduction to Cucumber, a framework that runs automated tests written in behaviour driven development (BDD) style.

Cucumber examples can be found in selenium-samples-java/cucumber-parallel GitHub repository.

Before going into details how to create tests with Cucumber first is good to do some context introduction why this framework is created and getting more popular.

Test driven development (TDD)

TDD (test driven development) is software development process in which developers first write the unit tests for feature or module based on requirements and then implement the feature or module itself. In short TDD cycle is “red > green > refactor”:

  • Red – phase where tests are implemented according to requirements, but they still fail
  • Green – phase where module or feature is implemented and tests pass
  • Refactor – phase where working code is made more readable and well structured

Behaviour driven development (BDD)

BDD emerged from and extends TDD. Instead of writing unit tests from specification why not make the specification a test itself. The main idea is that business analysts, project managers, users or anyone without technical, but with sufficient business knowledge can define tests.

Gherkin

BDD ideas sound very nice but actually are not so easy to put in practice. In order to make documentation sufficient for testing it should follow some rules. This is where Gherkin comes in place. It is a Business Readable, Domain Specific Language created especially to describe behaviour without defining how to implement it. Gherkin serves two purposes: it is your project’s documentation and automated tests.

Cucumber

Cucumber is not a testing tool it is a BDD tool for collaboration between all members of the team. So if you are using Cucumber just for automated testing you can do better. Testwise Cucumber is framework that understands Gherkin and runs the automated tests. It sounds like a fairy tale, you get your documentation described in Gherkin and tests just run. Actually it is not that simple, each step from documentation should have underlying test code that manipulates the application and should have test conditions. All process will be described in details with code samples bellow.

Setup Maven project

Proper way to do things is by using some build automation tool. Most used ones are Gradle and Maven. Maven is more simple to use and works fine for normal workflows. Gradle can do anything you want to do, but it comes at a price of complexity since you have to write Groovy code for custom stuff. In order to use Cucumber-JVM (Cucumber implementation for for the most popular JVM languages: Java, Groovy, Scala, etc.) in Java 8 POM looks like this:

<properties>
	<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
	<cucumber.version>1.2.4</cucumber.version>
	<selenium.version>2.48.2</selenium.version>
</properties>
<dependencies>
	<dependency>
		<groupId>info.cukes</groupId>
		<artifactId>cucumber-java8</artifactId>
		<version>${cucumber.version}</version>
		<scope>test</scope>
	</dependency>
	<dependency>
		<groupId>info.cukes</groupId>
		<artifactId>cucumber-junit</artifactId>
		<version>${cucumber.version}</version>
		<scope>test</scope>
	</dependency>
	<dependency>
		<groupId>org.seleniumhq.selenium</groupId>
		<artifactId>selenium-server</artifactId>
		<version>${selenium.version}</version>
		<scope>test</scope>
	</dependency>
</dependencies>

In order to build project in Java 8 this should be specified explicitly in POM by using maven-compiler-plugin:

<build>
	<plugins>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-compiler-plugin</artifactId>
			<version>3.3</version>
			<configuration>
				<source>1.8</source>
				<target>1.8</target>
			</configuration>
		</plugin>
	</plugins>
</build>

Once project is setup then real Cucumber usage can start.

Feature files

Everything start with a plain text file with .feature extension written in Gherkin language that describes only one product feature. It may contain from one to many scenarios. Bellow is a file with simple scenario for searching in Wikipedia.

Feature: search Wikipedia

  Scenario: direct search article
    Given Enter search term 'Cucumber'
    When Do search
    Then Single result is shown for 'Cucumber'

In Gherkin, each line that isn’t blank has to start with a Gherkin keyword, followed by any text you like. The main keywords are:

  • Feature
  • Scenario
  • Given, When, Then, And, But (Steps)
  • Background
  • Scenario Outline
  • Examples

More information can be found in Cucumber reference page.

Don’t repeat yourself

In some features there might be one and the same Given steps before each scenario. In order to avoid copy/paste it is better to define those steps as feature prerequisite with Background keyword.

Feature: search Wikipedia

  Background:
    Given Open http://en.wikipedia.org
    And Do login

  Scenario: direct search article
    Given Enter search term 'Cucumber'
    When Do search
    Then Single result is shown for 'Cucumber'

Data driven testing

Cucumber makes it very easy to handle cases of different business scenarios with different input data and different results based on that input data. Scenario is defined with Scenario Outline and . Then data is fed to this scenario with Examples table where variables are defined with concrete values. Example bellow shows scenario where search is done for two keywords and expected results for each are defined. It is very easy just to add more keywords and expected result which is actually treated by Cucumber reporting as different scenario.

Feature:

  Scenario Outline:
    Given Enter search term '<searchTerm>'
    When Do search
    Then Multiple results are shown for '<result>'

    Examples:
      | searchTerm | result                |
      | mercury    | Mercury may refer to: |
      | max        | Max may refer to:     |

Organise features and scenarios

Tags can be used in order to group feature files. Tag can be applied as annotation above Feature or Scenario keyword in .feature file. Having correctly tagged different scenarios or features runners can be created per each tag and run when needed. See more for tags in Cucumber tags page.

Runners

Next step in the process is to add runners. Cucumber supports running tests with JUnit and TestNG. In current post JUnit will be used. It has been imported in POM project file with cucumber-junit. In order to run a test with JUnit a special runner class should be created. The very basic form of the file is an empty class with @RunWith(Cucumber.class) annotation.

import org.junit.runner.RunWith;

import cucumber.api.junit.Cucumber;

@RunWith(Cucumber.class)
public class RunWikipediaTest {
}

This will run the tests and JUnit results file will be generated. Although it works it is much better to provide some Cucumber options with @CucumberOptions annotation.

import org.junit.runner.RunWith;

import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;

@RunWith(Cucumber.class)
@CucumberOptions(
		format = {
				"json:target/cucumber/wikipedia.json",
				"html:target/cucumber/wikipedia.html",
				"pretty"
		},
		tags = {"~@ignored"}
)
public class RunWikipediaTest {
}

Runner above runs all feature files which does not have tag @ignored and outputs the results in three different formats: JSON in target/cucumber/wikipedia.json file, HTML in target/cucumber/wikipedia.html and Pretty – a console output with colours. All formatting options are available in Cucumber options page.

Runners strategy

Different runners can be created for different purposes. There could be runners based on specific tags and those to be invoked in specific cases like regression and smoke testing, or full or partial regression testing. Depends on the needs. It is possible to create runners for each feature and run tests in parallel. There is a way to automatically generate runners and run tests in parallel. This will speed up execution and will keep the project clean from unessential runners. More details can be found in Running Cucumber tests in parallel post.

Runners feature file

If runner is not linked to any feature file (no features section defined in @CucumberOptions, like the one shown above) then runner automatically searches for all feature files in current and all child folders.

Java package structure is represented as a folder structure. Feature files can be place in some package structure in resources folder of the project, e.g. com.automationrhapsody.cucumber.parallel.tests.wikipedia. When compiled those are copied to same folder structure in the output. So in order to run all feature files from package above runner class need to be placed in same package in java folder of the project. It is possible to place the runner in parent package and it will still work as it searches all child folders e.g com.automationrhapsody.cucumber.parallel.tests will work. Placing in different package will not work though, e.g. com.automationrhapsody.cucumber.parallel.tests.other will not work. There will be message:

None of the features at [classpath:com/automationrhapsody/cucumber/parallel/tests/other] matched the filters: [~@ignored]

This means either there are no feature files into com.automationrhapsody.cucumber.parallel.tests.other resources package or those files have tag @ignored.

Easiest way is to put just one runner into default package and it will run all feature files available. This gives more limitations than benefits in the long run though.

Running feature files without step definitions

So far feature file and runner has been created. If now tests are started build will successes but tests will be skipped. This is because there is no implementation available for Given, When, Then commands in feature file. Those should be implemented and Cucumber gives hints in build output how to do it.

You can implement missing steps with the snippets below:

Given("^Enter search term 'Cucumber'$", () -> {
	// Write code here that turns the phrase above into concrete actions
	throw new PendingException();
});

Step definitions code / glue

So far feature file has been defined with runner for it. In terms of BDD this is OK, but in terms of testing a step definitions should be created so tests can actually be executed. As shown in hint above a method with annotation @Given is needed. Annotation text is actually a regular expression this is why it is good to start with ^ and end with $ which means whole line from the feature file to be matched. In order to make the step reusable some regular expression can be added in order to be able to pass different search terms, not only ‘Cucumber’. If there is regular expression defined then method can take arguments which will be actually what regex will match in feature file text. Step definition for Given command is:

@Given("^Enter search term '(.*?)'$")
public void searchFor(String searchTerm) {
	WebElement searchField = driver.findElement(By.id("searchInput"));
	searchField.sendKeys(searchTerm);
}

(.*?) is very basic regex, but it can do the work. It matches all zero or more characters surrounded by quotes in the end of the line after “Enter search term ” text. Once matched this is passed as argument to searchFor() method invocation.

Full step definitions for initial feature file are shown in class bellow:

package com.automationrhapsody.cucumber.parallel.tests.wikipedia;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;

import cucumber.api.java.After;
import cucumber.api.java.Before;
import cucumber.api.java.en.Given;
import cucumber.api.java.en.Then;
import cucumber.api.java.en.When;

import static junit.framework.Assert.assertTrue;
import static junit.framework.TestCase.assertFalse;
import static org.junit.Assert.assertEquals;

public class WikipediaSteps {

	private WebDriver driver;

	@Before
	public void before() {
		driver = new FirefoxDriver();
		driver.navigate().to("http://en.wikipedia.org");
	}

	@After
	public void after() {
		driver.quit();
	}

	@Given("^Enter search term '(.*?)'$")
	public void searchFor(String searchTerm) {
		WebElement searchField = driver.findElement(By.id("searchInput"));
		searchField.sendKeys(searchTerm);
	}

	@When("^Do search$")
	public void clickSearchButton() {
		WebElement searchButton = driver.findElement(By.id("searchButton"));
		searchButton.click();
	}

	@Then("^Single result is shown for '(.*?)'$")
	public void assertSingleResult(String searchResult) {
		WebElement results = driver
			.findElement(By.cssSelector("div#mw-content-text.mw-content-ltr p"));
		assertFalse(results.getText().contains(searchResult + " may refer to:"));
		assertTrue(results.getText().startsWith(searchResult));
	}
}

Before and After

As noticeable above there is method annotated with @Before. This is executed before each scenario being run. It is used to allocate resources needed for particular scenario to run. It is a good idea to instantiate WebDriver into this method. @After method is run in the end of the scenario so this is where resources are released, e.g. quitting WebDriver.

Page objects

Steps definitions code above are NOT well written. It is not good idea to locate elements directly in method where they are used. Never do this in real automation! Always use Page Object pattern. More details on it can be found in Page objects design pattern post.

Conclusion

BDD as an idea is good as it is according to Agile methodologies and stories definitions. Cucumber is very good BDD tool giving lots of flexibility and features with big community supporting it. Cucumber-JVM is very easy to start working with. Be careful though, when steps definitions grow in count the main challenge will be to make those reusable. Challenge will be to avoid writing new steps code if such is already written for semantically the same commands but written in different words.

Read more...