NTestsRunner for functional automated tests

Last Updated on by

Post summary: NTestsRunner implementation details and features.

In previous post I’ve described unit testing frameworks and why they are not suitable for running functional automated tests. I introduced NTestsRunner – very simple runner that can be used for running your automation tests. This topic is dedicated for implementation details of the NTestsRunner.

Verifications

It is important in functional testing to be able to place several verification points in one test. For this purpose abstract class Verification is implemented. It has two properties to store more details about verification and time it was taken. Constructor receives comma separated string values. In case of zero strings are passed then result is empty string. If one string is passed then this is the result. If more than one string is added then first string is taken as formatting string and others are used to build up the result. Logic is similar to string.Format(String, Object[]) method.

public abstract class Verification
{
	public string Result { get; private set; }
	public DateTime ExecutedAt { get; private set; }

	public Verification(params object[] args)
	{
		...
	}
}

Passed or Failed

In automation test may have two conditions – passed or failed. This is why two concrete classes are extending Verification: VerificationPassed and VerificationFailed. They do not add any other functionality. Those classes use parent’s class constructor. This is example how to instantiate object from those classes:

string value = "number";
int number = 1;
Verification result =
	new VerificationFailed("This is formatting string {0} {1}. ",
		value,
		number);

Test case result

Test case is generally a set of conditions to verify whether given scenario works are per user requirements. In automation world test case is test method with several verification points inside. In NTestsRunner TestCaseResult is class representing the idea of a test case. It has properties for name, time to run and list of all verifications with count of passed and failed.

public class TestCaseResult
{
	public List<Verification> Verifications = new List<Verification>();
	public string Name { get; set; }
	public int VerificationsFailed { get; set; }
	public int VerificationsPassed { get; set; }
	public TimeSpan Time { get; set; }
}

Test plan result

TestPlanResult in NTestsRunner has nothing to do with test plan term from QA world. Here this is a representation of a test class with test methods inside. It has properties for name and time to run. Also there is list with all TestCaseResults, i.e. test methods in that class. There are counters for passed and failed test cases and also counters for all passed and failed verifications inside all TestCaseResults.

public class TestPlanResult
{
	public List<TestCaseResult> TestCases = new List<TestCaseResult>();
	public string Name { get; set; }
	public int TestCasesPassed { get; private set; }
	public int TestCasesFailed { get; private set; }
	public int VerificationsPassed { get; private set; }
	public int VerificationsFailed { get; private set; }
	public TimeSpan Time { get; private set; }

	public void Count()
	{
		...
	}
}

Class and method attributes

In order to make one class a test class it should have with [TestClass] attributes. To convert method to a test one it should have [TestMethod] attribute. Just the attribute is not enough though. Method should have special method signature. This is required by NTestsRunner.

Test method signature

In order to run without exception test method needs to conform to two rules:

  1. To have attribute [TestMethod]
  2. Method to receive parameter List verifications in its signature, i.e.
    [TestMethod]
    public void TestMethod1(List<Verification> verifications)
    

Configurations

Configurations can be found on NTestRunner home page.

Execution

Once object from NTestsRunner is instantiated and configured tests with Execute() method. In side this method all classes from calling assembly (the one that holds the tests) are taken. If TestsToExecute is configured then only those with name matching given values are taken. If no TestsToExecute is provided then all classes with attribute [TestClass] are taken. Methods from each class are taken by default in order of appearance in the class. If method has [TestClass] attribute then method is executed by passing List object to it. Inside the method Verifications are collected as list into TestCaseResult object. After method is run TestCaseResult is added to its parent TestPlanResult which is added to list with all results. In the end results are saved as XML and HTML.

Results in jUnit XML

In order to integrate with CI tools such as Jenkins or Bamboo results are exported to XML file after execution has finished. File is named Results.xml and is located in test results folder. XML format is implemented according to junit-4.xsd.

Results in HTML

Tests result are saved as HTML report for better readability. File is named Results.html and is located in test results folder.

Usage

In order to use NTestsRunner a console application project is needed. This project will hold test classes. As the one bellow. Take into consideration that this is very simplified usage pattern. In reality Page objects design pattern will be used. Page objects will make the verifications and return them.

[TestClass]
public class TestClass1
{
	[TestMethod]
	public void TestMethod1(List<Verification> verifications)
	{
		// Do some actions
		verifications.Add(new VerificationFailed("There is error"));
		// Do some actions
		verifications.Add(new VerificationPassed("Everythign is OK"));
	}
}

In its main method new instance of NTestsRunner is created. Configurations are done and test executions is started. It is that simple to use it.

class Program
{
	static void Main(string[] args)
	{
		NTestsRunnerSettings settings = new NTestsRunnerSettings();
		settings.TestResultsDir = @"C:\temp";
		settings.MaxTestCaseRuntimeMinutes = 2;
		settings.TestsToExecute.Add("TestClass1");
		settings.PreventScreenLock = true;

		NTestsRunner runner = new NTestsRunner(settings);
		runner.Execute();
	}
}

Pros and cons

NTestsRunner has its pros and cons.
Pros are:

  • Pretty easy to use
  • Open source and can be customised to you specific needs
  • Gives you ability to make several verifications in one test and in case of fail it doesn’t break current test method
  • Tests are stored into console application that can be easily run
  • Results are saved in jUnit XML for CI integration
  • Results are saved in HTML

Cons are:

  • Test methods should have specific signature
  • It is not easy to migrate existing tests to new format

Conclusions

This is pretty good tool for running functional automated tests. It is very easy to use and is made especially for running functional automated tests. You can definitely give it a try.

If you find this post useful, please share it to reach more people. Sharing is caring!
Share on FacebookShare on LinkedInTweet about this on TwitterShare on Google+Email this to someone