Post summary: NTestsRunner implementation details and features.
In the previous post I’ve described unit testing frameworks and why they are not suitable for running functional automated tests. I introduced NTestsRunner – very simple runner that can be used for running your automation tests. This topic is dedicated to implementation details of the NTestsRunner.
Verifications
It is important in functional testing to be able to place several verification points in one test. For this purpose, abstract class Verification is implemented. It has two properties to store more details about verification and time it was taken. Constructor receives comma separated string values. In case of zero strings are passed then the result is an empty string. If one string is passed then this is the result. If more than one string is added then first string is taken as formatting string and others are used to build up the result. Logic is similar to string.Format(String, Object[]) method.
public abstract class Verification
{
public string Result { get; private set; }
public DateTime ExecutedAt { get; private set; }
public Verification(params object[] args)
{
...
}
}
Passed or Failed
In automation test may have two conditions – passed or failed. This is why two concrete classes are extending Verification: VerificationPassed and VerificationFailed. They do not add any other functionality. Those classes use parent’s class constructor. This is an example how to instantiate an object from those classes:
string value = "number";
int number = 1;
Verification result =
new VerificationFailed("This is formatting string {0} {1}. ",
value,
number);
Test case result
Test case is generally a set of conditions to verify whether given scenario works are per user requirements. In automation, world test case is test method with several verification points inside. In NTestsRunner TestCaseResult is class representing the idea of a test case. It has properties for name, time to run and list of all verifications with a count of passed and failed.
public class TestCaseResult
{
public List<Verification> Verifications = new List<Verification>();
public string Name { get; set; }
public int VerificationsFailed { get; set; }
public int VerificationsPassed { get; set; }
public TimeSpan Time { get; set; }
}
Test plan result
TestPlanResult in NTestsRunner has nothing to do with test plan term from QA world. Here this is a representation of a test class with test methods inside. It has properties for name and time to run. Also, there is a list of all TestCaseResults, i.e. test methods in that class. There are counters for passed and failed test cases and also counters for all passed and failed verifications inside all TestCaseResults.
public class TestPlanResult
{
public List<TestCaseResult> TestCases = new List<TestCaseResult>();
public string Name { get; set; }
public int TestCasesPassed { get; private set; }
public int TestCasesFailed { get; private set; }
public int VerificationsPassed { get; private set; }
public int VerificationsFailed { get; private set; }
public TimeSpan Time { get; private set; }
public void Count()
{
...
}
}
Class and method attributes
In order to make one class a test class, it should have with [TestClass] attributes. To convert method to a test one it should have [TestMethod] attribute. Just the attribute is not enough though. The method should have special method signature. This is required by NTestsRunner.
Test method signature
In order to run without exception test method needs to conform to two rules:
- To have attribute [TestMethod]
- Method to receive parameter List verifications in its signature, i.e.
[TestMethod] public void TestMethod1(List<Verification> verifications)
Configurations
Configurations can be found on NTestRunner home page.
Execution
Once object from NTestsRunner is instantiated and configured tests with Execute() method. Inside this method, all classes from calling assembly (the one that holds the tests) are taken. If TestsToExecute is configured then only those with name matching is given values are taken. If no TestsToExecute is provided then all classes with the attribute [TestClass] are taken. Methods from each class are taken by default in order of appearance in the class. If the method has [TestClass] attribute then the method is executed by passing List object to it. Inside the method, Verifications are collected as a list into a TestCaseResult object. After the method is run TestCaseResult is added to its parent TestPlanResult which is added to list with all results. In the end, results are saved as XML and HTML.
Results in JUnit XML
In order to integrate with CI tools such as Jenkins or Bamboo results are exported to XML file after execution has finished. The file is named Results.xml and is located in test results folder. XML format is implemented according to junit-4.xsd.
Results in HTML
Tests result are saved as HTML report for better readability. The file is named Results.html and is located in test results folder.
Usage
In order to use NTestsRunner a console application project is needed. This project will hold test classes. As the one below. Take into consideration that this is very simplified usage pattern. In reality, Page objects design pattern will be used. Page objects will make the verifications and return them.
[TestClass]
public class TestClass1
{
[TestMethod]
public void TestMethod1(List<Verification> verifications)
{
// Do some actions
verifications.Add(new VerificationFailed("There is error"));
// Do some actions
verifications.Add(new VerificationPassed("Everythign is OK"));
}
}
In its main method, a new instance of NTestsRunner is created. Configurations are done and test executions are started. It is that simple to use it.
class Program
{
static void Main(string[] args)
{
NTestsRunnerSettings settings = new NTestsRunnerSettings();
settings.TestResultsDir = @"C:\temp";
settings.MaxTestCaseRuntimeMinutes = 2;
settings.TestsToExecute.Add("TestClass1");
settings.PreventScreenLock = true;
NTestsRunner runner = new NTestsRunner(settings);
runner.Execute();
}
}
Pros and cons
NTestsRunner has its pros and cons.
Pros are:
- Pretty easy to use
- Open source and can be customized to your specific needs
- Gives you ability to make several verifications in one test and in case of failure it doesn’t break current test method
- Tests are stored in console application that can be easily run
- Results are saved in JUnit XML for CI integration
- Results are saved in HTML
Cons are:
- Test methods should have a specific signature
- It is not easy to migrate existing tests to a new format
Conclusions
This is a pretty good tool for running functional automated tests. It is very easy to use and is made especially for running functional automated tests. You can definitely give it a try.
