Soft assertions for C# unit testing frameworks (MSTest, NUnit, xUnit.net)

Last Updated on by

Post summary: Code example of very easy and useful custom implementation of soft assertions in C# unit testing frameworks such as MSTest, NUnit or xUnit.net.

Code shown in examples bellow is available in GitHub DotNetSamples/SoftAssertions repository.

Unit vs Functional testing

Unit testing paradigm states that each test exercises particular code behaviour. So in a perfect world one unit test would have one assertion which defines unit test result – either passed or failed. This is why unit testing frameworks provide only asserts which stop further execution of current test method. In functional testing usually one test verifies several conditions. Not debating if this is good or bad. Assume you are doing GUI testing, once you have opened particular page you’d better do as much verification as possible to reduce the risk of bugs. Having this page opened over and over for each single check is not the most efficient way of testing. This is why when you run functional tests you need some kind of assert that indicates whether passed or failed but to let the test continue in no critical issue is present. Those are generally called “soft” asserts.

Soft assertions code

Following code is an implementations of soft assertions:

public class SoftAssertions
{
	private readonly List<SingleAssert> 
		_verifications = new List<SingleAssert>();

	public void Add(string message, string expected, string actual)
	{
		_verifications.Add(new SingleAssert(message, expected, actual));
	}

	public void Add(string message, bool expected, bool actual)
	{
		Add(message, expected.ToString(), actual.ToString());
	}

	public void Add(string message, int expected, int actual)
	{
		Add(message, expected.ToString(), actual.ToString());
	}

	public void AddTrue(string message, bool actual)
	{
		_verifications
			.Add(new SingleAssert(message, true.ToString(), actual.ToString()));
	}

	public void AssertAll()
	{
		var failed = _verifications.Where(v => v.Failed).ToList();
		failed.Should().BeEmpty();
	}

	private class SingleAssert
	{
		private readonly string _message;
		private readonly string _expected;
		private readonly string _actual;

		public bool Failed => _expected != _actual;

		public SingleAssert(string message, string expected, string actual)
		{
			_message = message;
			_expected = expected;
			_actual = actual;
		}

		public override string ToString()
		{
			return $"'{_message}' assert was expected to be '{_expected}' " +
				$"but was '{_actual}'";
		}
	}
}

Soft assertions details

Actual assertion is handled by SingleAssert class. It contains a message to be displayed to user in case of fail as well as expected and actual values. They are stored as strings. All asserts during testing are stored in a List<SingleAssert>. There are several methods that add assert. There are such that accept bool, string and int. You can extend and add as many as you want. It is mandatory to call AssertAll() method so asserts can be evaluated. Evaluation consists of filtering out passed asserts leaving only failed: var failed = _verifications.Where(v => v.Failed).ToList(). Then list with failed is checked for empty failed.Should().BeEmpty(). In this case FluentAssertions framework is used, but code can be changed to such that suits your particular needs.

Soft assertions usage

Usage is pretty straight forward. SoftAssertions object should be created before each test and asserted after each test:

[TestClass]
public class UnitTest
{
	private SoftAssertions _softAssertions;

	[TestInitialize]
	public void SetUp()
	{
		_softAssertions = new SoftAssertions();
	}

	[TestCleanup]
	public void TearDown()
	{
		_softAssertions.AssertAll();
	}

	[TestMethod]
	public void TestMixedSoftAssertions()
	{
		_softAssertions.Add("Passing bool Add assertion", true, true);
		_softAssertions.Add("Failing bool Add assertion", true, false);
		_softAssertions
			.Add("Passing string Add assertion", "SameString", "SameString");
		_softAssertions
			.Add("Failing string Add assertion", "SameString", "OtherString");
		_softAssertions.Add("Passing int Add assertion", 1, 1);
		_softAssertions.Add("Failing int Add assertion", 1, 2);
		_softAssertions.AddTrue("Passing AddTrue assertion", true);
		_softAssertions.AddTrue("Failing AddTrue assertion", false);
	}
}

Soft assertions result

Result of test shown above is: Result Message: Expected collection to be empty, but found {‘Failing bool Add assertion’ assert was expected to be ‘True’ but was ‘False’, ‘Failing string Add assertion’ assert was expected to be ‘SameString’ but was ‘DifferentString’, ‘Failing int Add assertion’ assert was expected to be ‘1’ but was ‘2’, ‘Failing AddTrue assertion’ assert was expected to be ‘True’ but was ‘False’}.

This comes out of the box because FluentAssertions is used. Otherwise you have to do some other output and assertions.

Other soft assertions

Some custom implementation of soft assertions is as well available in NTestRunner framework, but it is more complex and demanding special approach for writing tests.

Conclusion

Soft assertions are very useful in functional testing. With this simple class you can directly have them in your functional tests.

Read more...

Soft assertions that do not fail JUnit test

Last Updated on by

Post summary: Code examples how to use assertions that do not fail the unit test immediately.

Code shown in examples bellow is available in GitHub java-samples/jersey1 repository.

Unit vs Functional testing

Unit testing paradigm states that each test exercises particular code behaviour. So in a perfect world one unit test would have one assertion which defines unit test result – either passed or failed. This is why unit testing frameworks provide only asserts which stop further execution of current test method. In functional testing usually one test verifies several conditions. Not debating if this is good or bad. Assume you are doing GUI testing, once you have opened particular page you’d better do as much verification as possible to reduce the risk of bugs. Having this page opened over and over for each single check is not the most efficient way of testing. This is why when you run functional tests you need some kind of assert that indicates whether passed or failed but to let the test continue in no critical issue is present. Those are generally called “soft” asserts.

Soft assertions and JUnit

TestNG provides org.testng.asserts.SoftAssert class for soft asserts as it is more oriented towards functional testing. JUnit is unit testing framework, so it does not provide any soft assertions. In order to create such behaviour additional libraries are needed.

AssertJ

AssertJ is library providing fluent assertions. It is very similar to Hamcrest which comes by default with JUnit. Along with all the asserts AssertJ provides soft assertions with its SoftAssertions class inside org.assertj.core.api package.

Usage

Bellow is a functional test run against Dropwizard stub described in Build a RESTful stub server with Dropwizard post. Important is to instantiate new SoftAssertions object before the test verifications and to call assertAll() method in the end to collect results. Best way to do this is to use JUnit’s @Before and @After annotated methods.

package com.automationrhapsody.jersey1;

import com.automationrhapsody.jersey1.model.Person;
import com.automationrhapsody.jersey1.rules.PersonServiceJerseyClient;

import java.util.List;

import org.assertj.core.api.SoftAssertions;
import org.junit.After;
import org.junit.Before;
import org.junit.ClassRule;
import org.junit.Test;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.core.Is.is;

public class PersonServiceTest {

	@ClassRule
	public static final PersonServiceJerseyClient CLIENT 
		= new PersonServiceJerseyClient();

	private SoftAssertions softAssertions;

	private Person person;

	@Before
	public void setUp() {
		person = new Person();
		person.setId(123);
		person.setFirstName("First Name");
		person.setLastName("Last Name");
		person.setEmail("Email");

		softAssertions = new SoftAssertions();
	}

	@After
	public void tearDown() {
		softAssertions.assertAll();
	}

	@Test
	public void testAllOperations() {
		String saveResult = CLIENT.save(person);
		assertThat(saveResult, is("Added Person with id=123"));

		Person actual = CLIENT.get(person.getId());
		softAssertions
			.assertThat(actual.getId()).isEqualTo(person.getId());
		softAssertions
			.assertThat(actual.getFirstName()).isEqualTo(person.getFirstName());
		softAssertions
			.assertThat(actual.getLastName()).isEqualTo(person.getLastName());
		softAssertions
			.assertThat(actual.getEmail()).isEqualTo(person.getEmail());

		String result = CLIENT.remove();
		assertThat(result, is("Last person remove. Total count: 4"));
	}
}

Conclusion

Soft assertions are needed in case of functional tests being run with JUnit. Since such are not available out of the box because JUnit is targeted for unit tests soft assertions can be used from external libraries such as AssertJ.

Read more...

Manage and automatically select needed WebDriver in Java 8 Selenium project

Last Updated on by

Post summary: Example code how to efficiently manage and automatically select needed local WebDriver using Java 8 method reference used as lambda expression.

Code examples in current post can be found in GitHub selenium-samples-java/design-patterns repository.

Java 8 features

In this example lambda expression and method reference Java 8 features are used. More in Java 8 features can be found in Java 8 features – Lambda expressions, Interface changes, Stream API, DateTime API post.

Functional interface

Before explaining lambda it is needed to understand the idea of functional interface as they are leveraged for use with lambda expressions. Functional interface is interface that has only one abstract method that is to be implemented. Functional interface may or may not have default or static methods (again new Java 8 feature). Although not mandatory good practice is to annotate functional interface with @FunctionalInterface.

Lambda expressions

There is not such term in Java, but you can think of lambda expression as anonymous method. Lambda expression is piece of code that provides inline implementation of a functional interface, eliminating the need of using anonymous classes. Lambda expressions facilitate functional programming and ease development by reducing the amount of code needed.

Method reference

Sometimes when using lambda expression all you do is call a method by name. Method reference provides easy way to call the method making code more readable.

Managing WebDriver

Proposed solution of managing WebDriver has enumeration called Browser and class called WebDriverFactory. Another important thing is web drivers should be placed in folder with name webdrivers and named with special pattern.

Browser enum

Code is shown bellow:

package com.automationrhapsody.designpatterns;

import java.util.Arrays;
import java.util.function.Supplier;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.ie.InternetExplorerDriver;

public enum Browser {
	FIREFOX("gecko", FirefoxDriver::new),
	CHROME("chrome", ChromeDriver::new),
	IE("ie", InternetExplorerDriver::new);

	private String name;
	private Supplier<WebDriver> driverSupplier;

	Browser(String name, Supplier<WebDriver> driverSupplier) {
		this.name = name;
		this.driverSupplier = driverSupplier;
	}

	public String getName() {
		return name;
	}

	public WebDriver getDriver() {
		return driverSupplier.get();
	}

	public static Browser fromString(String value) {
		for (Browser browser : values()) {
			if (value != null && value.toLowerCase().equals(browser.getName())) {
				return browser;
			}
		}
		System.out.println("Invalid driver name passed as 'browser' property. "
			+ "One of: " + Arrays.toString(values()) + " is expected.");
		return FIREFOX;
	}
}

Enumeration’s constructor has Supplier functional interface as parameter. When constructor is called method reference FirefoxDriver::new is called as lambda expression which purpose is to instantiate new Firefox driver. If only lambda expression is used is would be: () -> new FirefoxDriver(). Notice that method reference is much shorter and easy to read. getDriver() method invokes Supplier’s get() method which is implemented by the lambda expression, so lambda expression is executed hence instantiating new web driver. With this approach Firefox web dirver object is created only when getDriver() method is called.

WebDriverFactory

Code is:

package com.automationrhapsody.designpatterns;

import java.io.File;

import org.openqa.selenium.WebDriver;

class WebDriverFactory {

	private static final String WEB_DRIVER_FOLDER = "webdrivers";

		public static WebDriver createWebDriver() {
		Browser browser = Browser.fromString(System.getProperty("browser"));
		String arch = System.getProperty("os.arch").contains("64") ? "64" : "32";
		String os = System.getProperty("os.name").toLowerCase().contains("win") 
				? "win.exe" : "linux";
		String driverFileName = browser.getName() + "driver-" + arch + "-" + os;
		String driverFilePath = driversFolder(new File("").getAbsolutePath());
		System.setProperty("webdriver." + browser.getName() + ".driver", 
				driverFilePath + driverFileName);
		return browser.getDriver();
	}

	private static String driversFolder(String path) {
		File file = new File(path);
		for (String item : file.list()) {
			if (WEB_DRIVER_FOLDER.equals(item)) {
				return file.getAbsolutePath() + "/" + WEB_DRIVER_FOLDER + "/";
			}
		}
		return driversFolder(file.getParent());
	}
}

This code recursively searches for folder named webdrivers in the project. This is done because when you have multi-module project running from IDE and from Maven has different root folder and finding web drivers is not possible from both simultaneously. Once folder is found then proper web driver is selected based on OS and architecture. Code reads browser system property which can be passed from outside hence making selection of web driver easy to configure. Important part is to have web drivers with special naming convention.

Web drivers naming convention

In order code above to work web drivers should be place in webdrivers folder in the project and their names should match the pattern: {DIVER_NAME}-{ARCHITECTURE}-{OS}, e.g. geckodriver-64-win.exe for Windows 64 bit and geckodriver-64-linux for Linux 64 bit.

Conclusion

Proposed solution is very elegant way to manage your web drivers and select proper one just by passing -Dbrowser={BROWSER} Java system property.

Read more...

Retry JUnit failed tests immediately

Last Updated on by

Post summary: How to retry failed JUnit tests immediately and if retry is OK report test as passed.

Approaches

There are mainly three approaches to make JUnit retry failed tests.

  • Maven Surefire or Failsafe plugins – follow plugin name links for more details how to use and configure plugins
  • JUnit rules – code listed in current post can be used as a rule. See more for rules in Use JUnit rules to debug failed API tests post. Problem is @Rule annotation works for test methods only. In order to have retry logic in @BeforeClass then @ClassRule object should be instantiated.
  • JUnit custom runner – this post is dedicated on creating own JUnit retry runner and run tests with it.

Custom JUnit retry runner

Custom runner can be created by extending org.junit.runners.BlockJUnit4ClassRunner class and override public void run(final RunNotifier notifier) and protected void runChild(final FrameworkMethod method, RunNotifier notifier) methods. run() is accessed when test class is instantiated, runChild() is accessed when test method is run. Bellow is the code for custom JUnit retry runner class:

package com.automationrhapsody.junit.runners;

import org.junit.Ignore;
import org.junit.internal.AssumptionViolatedException;
import org.junit.internal.runners.model.EachTestNotifier;
import org.junit.runner.Description;
import org.junit.runner.notification.RunNotifier;
import org.junit.runner.notification.StoppedByUserException;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.Statement;

public class RetryRunner extends BlockJUnit4ClassRunner {

	private static final int RETRY_COUNT = 2;

	public RetryRunner(Class<?> clazz) throws InitializationError {
		super(clazz);
	}

	@Override
	public void run(final RunNotifier notifier) {
		EachTestNotifier testNotifier = new EachTestNotifier(notifier, getDescription());
		Statement statement = classBlock(notifier);
		try {
			statement.evaluate();
		} catch (AssumptionViolatedException ave) {
			testNotifier.fireTestIgnored();
		} catch (StoppedByUserException sbue) {
			throw sbue;
		} catch (Throwable t) {
			System.out.println("Retry class: " + getDescription().getDisplayName());
			retry(testNotifier, statement, t, getDescription());
		}
	}

	@Override
	protected void runChild(final FrameworkMethod method, RunNotifier notifier) {
		Description description = describeChild(method);
		if (method.getAnnotation(Ignore.class) != null) {
			notifier.fireTestIgnored(description);
		} else {
			runTest(methodBlock(method), description, notifier);
		}
	}

	private void runTest(Statement statement, Description description, RunNotifier notifier) {
		EachTestNotifier eachNotifier = new EachTestNotifier(notifier, description);
		eachNotifier.fireTestStarted();
		try {
			statement.evaluate();
		} catch (AssumptionViolatedException e) {
			eachNotifier.addFailedAssumption(e);
		} catch (Throwable e) {
			System.out.println("Retry test: " + description.getDisplayName());
			retry(eachNotifier, statement, e, description);
		} finally {
			eachNotifier.fireTestFinished();
		}
	}

	private void retry(EachTestNotifier notifier, Statement statement, Throwable currentThrowable, Description info) {
		int failedAttempts = 0;
		Throwable caughtThrowable = currentThrowable;
		while (RETRY_COUNT > failedAttempts) {
			try {
				System.out.println("Retry attempt " + (failedAttempts + 1) + " for " + info.getDisplayName());
				statement.evaluate();
				return;
			} catch (Throwable t) {
				failedAttempts++;
				caughtThrowable = t;
			}
		}
		notifier.addFailure(caughtThrowable);
	}
}

Code shown above is available in GitHub java-samples/junit repository.

Using JUnit RetryRunner

In order to configure JUnit test to use the runner, class holding tests should be annotated with @RunWith:

@RunWith(RetryRunner.class)
public class RetryRunnerTests {
	@Test
	public void testRetrySuccessFirstTime() {
		assertTrue(true);
	}
}

Conclusion

Making JUnit to rerun is easy, the harder thing to do is to fix your tests so they pass from first time. Generally it is not a good to have tests that are flaky.

Read more...

Complete guide to email verifications with Automation SMTP Server

Last Updated on by

Post summary: How to do complete email verification with your own Automation SMTP server.

SMTP is protocol initially defined in 1982 and is still used nowadays. In order to automate application which sends out emails you need SMTP server which reads messages and saves them to disk for further processing. Note that this is only in case when you application sends emails.

Windows SMTP server

One option is to use SMTP server provided by Windows. Problems here are two. First is that from Vista SMTP server is no more supported. There is SMTP server in Windows Server distributions but licence for them is more expensive. Second problem comes from configuration of the server. You might have several machines and configurations should be maintained on all of them. It is an feasible option to use Windows SMTP server but current post is not dedicated to it.

Automation SMTP Server

What I offer in this post is your own Automation SMTP Server. It is located in following GitHub project. Solution is actually mixture of two open source projects. For server I use Antix SMTP Server For Developers, which is really good SMTP server. It is windows application and is more suitable for manual SMTP testing rather than automation. I’ve extracted the SMTP core with some modifications as a console application which saves emails as EML file on disk. For reading of emails I use source code of Easily Retrieve Email Information from .EML Files article with several modifications. What you need to do in order to make successful email verification is download executable from GitHub and follow instructions bellow. More info for it can be fond on its home page Automation SMTP Server.

Automation SMTP Server usage

In GitHub AutomationSMTPServer repository there is example that shows how to use Automation SMTP Server. Server should be added as reference to your automation project. Since it is a reference it gets copied into compiled executables folder.

Delete recent emails

Before doing anything in your tests it is good to delete old emails. Automation SMTP Server is saving mail into folder named “temp”. This is how it works and cannot be changed.

private string currentDir =
	Directory.GetCurrentDirectory() + Path.DirectorySeparatorChar;
private string mailsDir = currentDir + "temp";

if (Directory.Exists(mailsDir))
{
	Directory.Delete(mailsDir, true);
}

Start Automation SMTP Server

Server is a console application. It receives emails and saves them to disk. If counter party send QUIT message to disconnect server gets restarted to wait for next connection. Server should be started as a process. Port should be provided as arguments. If not provided it can be configured in SMTP Server config file. If not configured there it gives message and takes 25 for default port.

Process smtpServer = new Process();
smtpServer.StartInfo.FileName = currentDir + "AutomationSMTPServer.exe";
smtpServer.StartInfo.Arguments = "25";
smtpServer.Start();

Send emails

This is the point where your application under test is sending emails which you will later verify.

Read emails

Once emails have been sent out from application under test you are ready to read and process them.

string[] files = Directory.GetFiles(mailsDir);
List<EMLFile> mails = new List<EMLFile>();

foreach (string file in files)
{
	EMLFile mail = new EMLFile(file);
	mails.Add(mail);
	File.Delete(file);
}

Verify emails

Here you can use EMLFile class which is parsing the EML file and is representing is as object so you can do operations on it. Once you have the mail as object you can access all its attributes and verify some of them. It all depends on your testing strategy. Another option is to define on expected EML file, read it and compare both actual and expected. EMLFile class has predefined Equals method which is comparing all the attributes of the emails.

bool compare1 = mails[0].Equals(mails[1]);
bool compare2 = mails[0].Equals(mails[2]);
bool compare3 = mails[1].Equals(mails[2]);

Stop Automation SMTP Server

This part is important. If not stopped server will continue to work and will block the port. Its architecture is defined in such manner that only way to stop it is it to terminate console application. In case where you have started it from C# code as process way to stop it is to kill the process.

smtpServer.Kill();

Conclusion

Proper email verification can be challenge. In case your application under tests send emails I would say it is crucial to have correct email testing as mail is what customers receive. And in the end it is all about customers! So give it a try and enjoy this easy way of email verification.

Read more...

Extract and verify text from PDF with C#

Last Updated on by

Post summary: How to extract text from PDF in C#.

PDF verification is pretty rare case in automation testing. Still it could happen.

iTextSharp

iTextSharp is library that allows you to manipulate PDF files. We need very small of this library. It has build in reader that iterates through pages and returns only text.

using iTextSharp.text.pdf;
using iTextSharp.text.pdf.parser;
using System.Text;

namespace PDFExtractor
{
	public class PDFExtractor
	{
		public static string ExtractTextFromPDF(string pdfFileName)
		{
			StringBuilder result = new StringBuilder();
			// Create a reader for the given PDF file
			using (PdfReader reader = new PdfReader(pdfFileName))
			{
				// Read pages
				for (int page = 1; page <= reader.NumberOfPages; page++)
				{
					SimpleTextExtractionStrategy strategy =
						new SimpleTextExtractionStrategy();
					string pageText =
						PdfTextExtractor.GetTextFromPage(reader, page, strategy);
					result.Append(pageText);
				}
			}
			return result.ToString();
		}
	}
}

Verification

Once extracted text can be verified against expected as described in Text verification post.

Read more...

Text verification

Last Updated on by

Post summary: Verify actual text with expected one by ignoring what is not relevant during compare.

In automation testing there is no definitive way what text verification is best to be done. One strategy is to check that an expected word or a phrase exists in actual text shown in application under tests. Other strategy is to prepare large amount of text to verify. Later strategy is expensive in case of effort for preparation and maintenance. First strategy might not be sufficient to do correct verifications.

In between

What I suggest here is something in between. Not too much but not too less. Problem with paragraph of text to be verified is it might contain data we do not have control over, e.g. date, time, unique values, etc.

Example

Imagine an e-commerce website. When you place the order there is order confirmation page. You want to verify not only that you are on this page but also that text is correct as per specification. Most likely text will contain data you do not have control over – order number and date. Breaking verification is small chunks is an option. Another option is to manipulate the actual text. Third option is to define the text as expected with special strings that will get ignored during compare.

Actual vs Expected

Actual text could be: “Order 123456 has been successfully placed on 01.01.1970! Thank you for your order. ”
Expected text could be: “Order ~SKIP~ has been successfully placed on ~SKIP~! Thank you for your order. ”
And then you can compare both where ~SKIP~ will be ignored during compare.

Compare code

Code to do the compare shown above is incorporated in NTestsRunner also:

public const string IgnoreDuringCompare = "~SKIP~";

public static bool EqualsWithIgnore(this string value1, string value2)
{
	string regexPattern = "(.*?)";
	// If value is null set it to empty
	value1 = value1 ?? string.Empty;
	value2 = value2 ?? string.Empty;
	string input = string.Empty;
	string pattern = string.Empty;
	// Unify new lines symbols
	value1 = value1.Replace("\r\n", "\n");
	value2 = value2.Replace("\r\n", "\n");
	// If no one conains ignore string then compare directly
	if (!value1.Contains(IgnoreDuringCompare) &&
		!value2.Contains(IgnoreDuringCompare))
	{
		return value1.Equals(value2);
	}
	else if (value1.Contains(IgnoreDuringCompare))
	{
		pattern = Regex.Escape(value1).Replace(IgnoreDuringCompare, regexPattern);
		input = value2;
	}
	else if (value2.Contains(IgnoreDuringCompare))
	{
		pattern = Regex.Escape(value2).Replace(IgnoreDuringCompare, regexPattern);
		input = value1;
	}

	Match match = Regex.Match(input, pattern);
	return match.Success;
}

Use in tests

In your tests you will do something like:

string actual = OrderConfirmationPage.GetConfirmationText();
string expected = "Order " + ExtensionMethods.IgnoreDuringCompare +
	" has been successfully placed on " + ExtensionMethods.IgnoreDuringCompare +
	"! Thank you for your order. ";
Assert.IsTrue(actual.EqualsWithIgnore(expected));

Conclusion

It might take little bit more effort to prepare expected strings but verification will be more complete and correct rather than just to expect a word or a phrase.

Read more...

Multilingual automation testing with enumerations

Last Updated on by

Post summary: Solution for automated testing of multilingual sites by using string values in all supported languages for enumerations.

In efficiently use of enumerations with string values in C# post I’ve described how you can add text to an enumeration element and then use it. Current post is elaboration with code samples for testing multilingual applications.

The challenge

Multilingual automation is always a challenge. If you use text to locate elements or verify condition then trying to run test with different language will fail. Enumerations with language dependant string values is pretty good solution. How to do it is described bellow.

Define attribute

StringValue class is extending System.Attribute. It has two properties for text and language. It should have AllowMultiple = true in order to be applied as many times as many languages you have.

namespace System
{
	[AttributeUsage(AttributeTargets.Field, AllowMultiple = true)]
	public class StringValue : Attribute
	{
		public string Value { get; private set; }
		public string Lang { get; private set; }

		public StringValue(string lang, string value)
		{
			Lang = lang;
			Value = value;
		}
	}
}

Read attribute

With reflection read all StringValue attributes. Iterate them and return the one that matches language given as parameter.

using System.Reflection;

namespace System
{
	public static class ExtensionMethods
	{
		public static string GetStringValue(this Enum value, string lang)
		{
			string stringValue = value.ToString();
			Type type = value.GetType();
			FieldInfo fieldInfo = type.GetField(value.ToString());
			StringValue[] attrs = fieldInfo.
				GetCustomAttributes(typeof(StringValue), false) as StringValue[];
			foreach (StringValue attr in attrs)
			{
				if (attr.Lang == lang)
				{
					return attr.Value;
				}
			}
			return stringValue;
		}
	}
}

Apply to enumerations

All supported languages can be defined as string constants. It will be pretty cool if can define enumeration with languages and pass it in StringValue constructor as language but it is not possible as it is not a compile time constant.

public class Constants
{
	public const string LangEn = "en";
	public const string LangFr = "fr";
	public const string LangDe = "de";
}

public enum Messages
{
	[StringValue(Constants.LangEn, "Problem occured, try again later")]
	[StringValue(Constants.LangFr, "Problème survenu, réessayer plus tard")]
	[StringValue(Constants.LangDe, "Problem aufgetreten, " +
		"versuchen Sie es später erneut")]
	ProblemOccured,
	[StringValue(Constants.LangEn, "Successfully done")]
	[StringValue(Constants.LangFr, "Fait avec succès")]
	[StringValue(Constants.LangDe, "Erfolgreich durchgeführt")]
	Success
}

Use in code

Somewhere at a top level of your tests you should have property or field which most likely will be read from conflagration and will define for which locate is current test run.

string lang = Constants.LangFr;

This is then used to read correct text value for given enumeration element.

Assert.AreEqual(Messages.ProblemOccured.GetStringValue(lang), 
	App.MessageBox.GetText());

Conclusion

Multilingual testing is a challenge. Be smart and use all tricks you might get. In this post I’ve revealed pretty good trick to do the automation. Challenge with this approach will be initial set up of enumerations with all the translations.

Read more...

Efficiently use of enumerations with string values in C#

Last Updated on by

Post summary: Using enumerations or specialised classes makes your automation tests easy to understand and maintain. Show with code samples how to define and read string value to enumeration elements.

When you do automation tests and have to pass value to a method it is so easy and natural to just use strings. There are many cases where string is the correct solution. There are also many cases where string can be solution, but enumeration or specialised class are better and more efficient solution.

Why not strings

Having the following example – web application with drop down which has several options. We are using Page objects pattern to model the page. Page object has a method which accepts the option to be selected. String seems as a natural solution but is wrong. Although string will work enumeration is the only right solution. Drop down has limited and already defined options that can be selected. Exposing just string may cause misinterpretations for consumer of your method. It is much more easy to limit the consumer to several enumeration values. In this way consumer knows what data to provide and this automatically keeps code clean from magic strings. If changes are needed they will be done only in the enumeration making code easier to maintain.

Problem with enumerations in C#

Using enumerations for example given above will not work. Unlike Java enumerations in C# are wrappers for int or other numeric types value. You are not able to use text with enumeration element.

Using string values with enumerations

Only way to use string values in enumerations is by adding it as an attribute to each enumeration’s element. It takes several steps in order to accomplish this.

  1. Create the attribute class that will be applied to enumeration element
  2. Create extension method that is responsible for reading string value from enumeration element
  3. Apply string value attribute to enumeration element
  4. Use in code

Bellow are code samples how to use string values with enumerations in C#. Defining and reading of the attribute is functionality built in NTestsRunner.

Define attribute

First step is to create class that extends System.Attribute. It has only one string property to hold the text in it. Text is passed in constructor. Note that this class is defined in System name space in order to have it by default skipping the need of importing name space you might not be aware of.

namespace System
{
	public class StringValue : Attribute
	{
		public string Value { get; private set; }

		public StringValue(string value)
		{
			Value = value;
		}
	}
}

Read the attribute

C# provides so called extension methods, a great way to add new functionality to existing type without creating new derived type. Reading of string value from enumeration element is done with GetStringValue extension method. With reflection all StringValue custom attributes of element are obtained. If some found text of first is returned. If not then string representation of element is returned.

using System.Reflection;

namespace System
{
	public static class ExtensionMethods
	{
		public static string GetStringValue(this Enum value)
		{
			string stringValue = value.ToString();
			Type type = value.GetType();
			FieldInfo fieldInfo = type.GetField(value.ToString());
			StringValue[] attrs = fieldInfo.
				GetCustomAttributes(typeof(StringValue), false) as StringValue[];
			if (attrs.Length > 0)
			{
				stringValue = attrs[0].Value;
			}
			return stringValue;
		}
	}
}

Apply to enumerations

Once StringValue class is ready it can be applied as attribute to any enumeration.

public enum Messages
{
	[StringValue("Problem occured, try again later")]
	ProblemOccured,
	[StringValue("Successfully done")]
	Success
}

Use in code

In code string value can be obtained from enumeration’s element with GetStringValue method.

Assert.AreEqual(Messages.ProblemOccured.GetStringValue(), App.MessageBox.GetText());

Conclusion

Using enumerations is mandatory to make readable and maintainable automation. Working effectively with enumerations will increase your value as automation specialist.

Read more...

NTestsRunner for functional automated tests

Last Updated on by

Post summary: NTestsRunner implementation details and features.

In previous post I’ve described unit testing frameworks and why they are not suitable for running functional automated tests. I introduced NTestsRunner – very simple runner that can be used for running your automation tests. This topic is dedicated for implementation details of the NTestsRunner.

Verifications

It is important in functional testing to be able to place several verification points in one test. For this purpose abstract class Verification is implemented. It has two properties to store more details about verification and time it was taken. Constructor receives comma separated string values. In case of zero strings are passed then result is empty string. If one string is passed then this is the result. If more than one string is added then first string is taken as formatting string and others are used to build up the result. Logic is similar to string.Format(String, Object[]) method.

public abstract class Verification
{
	public string Result { get; private set; }
	public DateTime ExecutedAt { get; private set; }

	public Verification(params object[] args)
	{
		...
	}
}

Passed or Failed

In automation test may have two conditions – passed or failed. This is why two concrete classes are extending Verification: VerificationPassed and VerificationFailed. They do not add any other functionality. Those classes use parent’s class constructor. This is example how to instantiate object from those classes:

string value = "number";
int number = 1;
Verification result =
	new VerificationFailed("This is formatting string {0} {1}. ",
		value,
		number);

Test case result

Test case is generally a set of conditions to verify whether given scenario works are per user requirements. In automation world test case is test method with several verification points inside. In NTestsRunner TestCaseResult is class representing the idea of a test case. It has properties for name, time to run and list of all verifications with count of passed and failed.

public class TestCaseResult
{
	public List<Verification> Verifications = new List<Verification>();
	public string Name { get; set; }
	public int VerificationsFailed { get; set; }
	public int VerificationsPassed { get; set; }
	public TimeSpan Time { get; set; }
}

Test plan result

TestPlanResult in NTestsRunner has nothing to do with test plan term from QA world. Here this is a representation of a test class with test methods inside. It has properties for name and time to run. Also there is list with all TestCaseResults, i.e. test methods in that class. There are counters for passed and failed test cases and also counters for all passed and failed verifications inside all TestCaseResults.

public class TestPlanResult
{
	public List<TestCaseResult> TestCases = new List<TestCaseResult>();
	public string Name { get; set; }
	public int TestCasesPassed { get; private set; }
	public int TestCasesFailed { get; private set; }
	public int VerificationsPassed { get; private set; }
	public int VerificationsFailed { get; private set; }
	public TimeSpan Time { get; private set; }

	public void Count()
	{
		...
	}
}

Class and method attributes

In order to make one class a test class it should have with [TestClass] attributes. To convert method to a test one it should have [TestMethod] attribute. Just the attribute is not enough though. Method should have special method signature. This is required by NTestsRunner.

Test method signature

In order to run without exception test method needs to conform to two rules:

  1. To have attribute [TestMethod]
  2. Method to receive parameter List verifications in its signature, i.e.
    [TestMethod]
    public void TestMethod1(List<Verification> verifications)
    

Configurations

Configurations can be found on NTestRunner home page.

Execution

Once object from NTestsRunner is instantiated and configured tests with Execute() method. In side this method all classes from calling assembly (the one that holds the tests) are taken. If TestsToExecute is configured then only those with name matching given values are taken. If no TestsToExecute is provided then all classes with attribute [TestClass] are taken. Methods from each class are taken by default in order of appearance in the class. If method has [TestClass] attribute then method is executed by passing List object to it. Inside the method Verifications are collected as list into TestCaseResult object. After method is run TestCaseResult is added to its parent TestPlanResult which is added to list with all results. In the end results are saved as XML and HTML.

Results in jUnit XML

In order to integrate with CI tools such as Jenkins or Bamboo results are exported to XML file after execution has finished. File is named Results.xml and is located in test results folder. XML format is implemented according to junit-4.xsd.

Results in HTML

Tests result are saved as HTML report for better readability. File is named Results.html and is located in test results folder.

Usage

In order to use NTestsRunner a console application project is needed. This project will hold test classes. As the one bellow. Take into consideration that this is very simplified usage pattern. In reality Page objects design pattern will be used. Page objects will make the verifications and return them.

[TestClass]
public class TestClass1
{
	[TestMethod]
	public void TestMethod1(List<Verification> verifications)
	{
		// Do some actions
		verifications.Add(new VerificationFailed("There is error"));
		// Do some actions
		verifications.Add(new VerificationPassed("Everythign is OK"));
	}
}

In its main method new instance of NTestsRunner is created. Configurations are done and test executions is started. It is that simple to use it.

class Program
{
	static void Main(string[] args)
	{
		NTestsRunnerSettings settings = new NTestsRunnerSettings();
		settings.TestResultsDir = @"C:\temp";
		settings.MaxTestCaseRuntimeMinutes = 2;
		settings.TestsToExecute.Add("TestClass1");
		settings.PreventScreenLock = true;

		NTestsRunner runner = new NTestsRunner(settings);
		runner.Execute();
	}
}

Pros and cons

NTestsRunner has its pros and cons.
Pros are:

  • Pretty easy to use
  • Open source and can be customised to you specific needs
  • Gives you ability to make several verifications in one test and in case of fail it doesn’t break current test method
  • Tests are stored into console application that can be easily run
  • Results are saved in jUnit XML for CI integration
  • Results are saved in HTML

Cons are:

  • Test methods should have specific signature
  • It is not easy to migrate existing tests to new format

Conclusions

This is pretty good tool for running functional automated tests. It is very easy to use and is made especially for running functional automated tests. You can definitely give it a try.

Read more...

Running functional automation tests

Last Updated on by

Post summary: Unit testing frameworks are not very suitable for running functional tests. NTestsRunner is an alternative way of running functional automated tests.

Unit testing

Unit testing is focused on testing code on low level. Methods, sets of methods or modules are being tested by writing test code which invokes those methods with specific arguments. In unit testing all external dependencies (database, file system, network, etc.) are removed. Those resources are simulated in unit tests by using so called mock objects. Mock objects are controlled by tests designer and have predictive behaviour. Running piece of code which doesn’t have external dependencies happens almost immediately. So unit tests are executed for very low amount of time. It is considered set of unit tests taking longer than 5 minutes is not well designed. Unit tests are strictly focused. One test tests only one condition. Each test is not related in anyhow to other tests.

Unit testing frameworks

Unit testing frameworks conform to unit tests purpose and design. Tests should not depend on each other. For this reason unit testing frameworks execute tests in random order (xUnit.net or MS Unit Testing Framework), other like NUnit in alphabetic order of tests method names. Checking for given conditions are done with assertions. If one assert fails current test execution is stopped and test is marked as failed.

Functional testing

Functional testing is focused on ensuring that software product works as per user requirements. Real life software has external dependencies. Most software products has some kind of user interface. Automation tests are focused on verifying that UI works correctly. Transferring and rendering data to UI takes time, database operations take time, file and network operations also take time, etc. In general functional tests are more complex and take much longer to execute. In order to be efficient checking of several conditions are defined in each test. Functional tests can be manual and automated. In current post when I mention functional tests I mean only automated.

Requirement for running functional test – many verifications

In a perfect situation functional one test should verify only one condition. In reality because of too many external dependencies time for tests execution is large. An time matters. In order to shorten this time we make several checks in one test. For e.g. in e-commerce web site order is placed. We verify order confirmation page that there is order number, that address is same used during checkout, that user’s email is correct. We also can verify inside user’s email box that received mail is correct. We can check in database for some properties of the order. If we have to do one order for each check tests will take significantly long time. To be efficient we do all checks with one order in one test case. We need a framework which allows you to have multiple verifications in one test. Further more if verification is failed test execution should continue.

Requirement for running functional test – controlled sequence

What is really good to be avoided but sometimes cannot is test dependency. Sometimes one tests needs another to have done something before continuing. As I said this should be avoided but in order to be efficient you should make trade off between good tests design and time to execute. For e.g. you may want to cancel and then refund order placed in e-commerce web site. Generally it is best to place new order for this test but if placing of order requires too much time then an option is to reuse already existing order from previous test. We need to be able to control test case execution order.

NTestsRunner

In order to have control over tests and use many verifications I’ve created NTestsRunner. Its code is in GitHub NTestsRunner repository. This is .NET library. You create console application with your tests and use the library within. Tests are annotated is similar fashion as with unit testing frameworks. Tests are executed in sequential order. There could be many verifications. Results are saved as HTML and XML in jUnit format. NTests runner is described in more details in this post NTestsRunner for functional automated tests.

Read more...

Complete guide how to use design patterns in automation

Last Updated on by

Post summary: Complete code example of how design patterns can be used in real life test automation.

With series of posts I’ve described 5 design patterns that each automation test engineer should know and use. I’ve started with brief description of the patterns. Then I’ve explained in details with code snippets following patterns: Page objects, Facade, Factory, Singleton, Null object. Code examples are located in GitHub for C# and Java.

Overview

This post is intended to bond all together in a complete guide how to do write better automation code. Generally automation project consists of two parts. Automation framework project and tests project. Current guide is intended to describe how to build your automation testing framework. How to structure your tests is different topic. Remember once having correctly designed framework then tests will be much more clean, maintainable and easy to write. To keep post shorter some of the code that is not essential for representing the idea is removed. Whole code is in GitHub.

Page objects

Everything start by defining proper page objects. There is no fixed recipe for this. It all depends on the structure of application under test. General rule is that repeating elements (header, footer, menu, widget, etc) are extracted as separate objects. The whole idea is to have one element defined in only one place (stay DRY)! Bellow is our HomePage object. What you can do generally is make search and clear search terms. Note that clearing is done with jQuery code. This is because of a bug I’ve described with workaround in Selenium WebDriver cannot click UTF-8 icons post.

using OpenQA.Selenium;

namespace AutomationRhapsody.DesignPatterns
{
	class HomePageObject
	{
		private WebDriverFacade webDriver;
		public HomePageObject(WebDriverFacade webDriver)
		{
			this.webDriver = webDriver;
		}

		private IWebElement SearchField
		{
			get { return webDriver.FindElement(By.Id("search")); }
		}

		public void SearchFor(string text)
		{
			SearchField.SendKeys(text);
		}

		public void ClearSearch()
		{
			webDriver.ExecuteJavaScript("$('span.cancel').click()");
		}
	}
}

WebDriver factory

WebDriver factory will be responsible for instantiating the WebDriver based on condition which browser we want to run our tests with.

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium.IE;

namespace AutomationRhapsody.DesignPatterns
{
	public class WebDriverFactory
	{
		public IWebDriver CreateInstance(Browsers browser)
		{
			if (Browsers.Chrome == browser)
			{
				return new ChromeDriver();
			}
			else if (Browsers.IE == browser)
			{
				return new InternetExplorerDriver();
			}
			else
			{
				return new FirefoxDriver();
			}
		}
	}
}

Constructor take an argument browser type. Browser type is defined as enumeration. This is very important. Avoid passing back and forth strings. Always stick to enums or special purpose classes. This will save you time investigating bugs in your automation.

public enum Browsers
{
	Chrome, IE, Firefox
}

NullWebElement

This is null object pattern and implements IWebElement. There is a NULL property that is used to compare is given element is not found or no.

using OpenQA.Selenium;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Drawing;

namespace AutomationRhapsody.DesignPatterns
{
	public class NullWebElement : IWebElement
	{
		private const string nullWebElement = "NullWebElement";
		public bool Displayed { get { return false; } }
		public bool Enabled { get { return false; } }
		public Point Location { get { return new Point(0, 0); } }
		public bool Selected { get { return false; } }
		public Size Size { get { return new Size(0, 0); } }
		public string TagName { get { return nullWebElement; } }
		public string Text { get { return nullWebElement; } }
		public void Clear() { }
		public void Click() { }
		public string GetAttribute(string attributeName) { return nullWebElement; }
		public string GetCssValue(string propertyName) { return nullWebElement; }
		public void SendKeys(string text) { }
		public void Submit() { }
		public IWebElement FindElement(By by) { return this; }
		public ReadOnlyCollection<IWebElement> FindElements(By by)
		{
			return new ReadOnlyCollection<IWebElement>(new List<IWebElement>());
		}

		private NullWebElement() { }

		private static NullWebElement instance;
		public static NullWebElement NULL
		{
			get
			{
				if (instance == null)
				{
					instance = new NullWebElement();
				}
				return instance;
			}
		}
	}
}

WebDriver facade

WebDriver facade main responsibility is to define custom behaviour on elements location. This gives you centralised control over elements location. Constructor takes browser type and uses the factory to create WebDriver instance which is used internally in the facade. FindElement method defines explicit wait. If element is not found then NullWebElement which is actually implementation of Null object pattern. The idea is to safely locate elements with try/catch and then just use them skipping checks for null.

using OpenQA.Selenium;
using OpenQA.Selenium.Support.UI;
using System;
using System.Collections.ObjectModel;

namespace AutomationRhapsody.DesignPatterns
{
	public class WebDriverFacade
	{
		private IWebDriver webDriver = null;
		private TimeSpan waitForElement = TimeSpan.FromSeconds(5);

		public WebDriverFacade(Browsers browser)
		{
			WebDriverFactory factory = new WebDriverFactory();
			webDriver = factory.CreateInstance(browser);
		}

		public void Start(string url)
		{
			webDriver.Url = url;
			webDriver.Navigate();
		}

		public void Stop()
		{
			webDriver.Quit();
		}

		public object ExecuteJavaScript(string script)
		{
			return ((IJavaScriptExecutor)webDriver).
				ExecuteScript("return " + script);
		}

		public IWebElement FindElement(By by)
		{
			try
			{
				WebDriverWait wait = new WebDriverWait(webDriver, waitForElement);
				return wait.Until(ExpectedConditions.ElementIsVisible(by));
			}
			catch
			{
				return NullWebElement.NULL;
			}
		}
	}
}

Tests

As I mention initially this post is about using efficiently design patterns in your framework automation project. Tests design are not being discussed here. Once you have designed the framework one simple test (without asserts) that makes search will look like code bellow.

Browsers browser = Browsers.Chrome;
WebDriverFacade webDriver = new WebDriverFacade(browser);
webDriver.Start("http://automationrhapsody.com/examples/utf8icons.html");

HomePageObject homePage = new HomePageObject(webDriver);
homePage.ClearSearch();
homePage.SearchFor("automation");

webDriver.Stop();

Conclusion

Design patterns are enabling you to write maintainable code. They increase value of your code. As shown in this series of posts they are not so complicated and you can easily adopt them in your automation. Most important is design patterns increase your value as a professional. So make the time to learn them!

Read more...

Singleton and Null object patterns

Last Updated on by

Post summary: Description with code samples of Singleton and Null object design patterns.

Singleton

Singleton pattern is used in a case where you want to restrict the instantiation of a class to only one object.

More details

The class should not require parameters for its construction. If creation of a class requires significant resources then singleton is a suitable solution. It is good that singletons are created when they are first needed, so called lazy initialisation.

Null object pattern

The idea is to make code more simpler and safe by skipping checks for null reference of a given object:

if (someVariable != null) {
	// Do something with someVariable
}

References

This post is part of design patterns guide series. Code examples are located in GitHub for C# and Java.

More details

Null object implements given interface and its methods are doing nothing. This makes the null object more predictable. You can safely invoke methods on the null object without the threat of a NullRefferenceException to break you application. This pattern is very well combined with singleton pattern where null object is actually a singleton. In this case you can check for reference or equality.

Example

Most common usage of Singleton pattern is to limit WebDriver to only one object instance in the whole automation test project.

private static IWebDriver webDriver;
public static IWebDriver WebDriver
{
	get
	{
		if (webDriver == null)
		{
			webDriver = new FirefoxDriver();
		}
		return webDriver;
	}
}

I’m going to give slightly different usage example where Singleton is used along with Null object pattern. NullWebElement implements IWebElement interface so it must implement all methods and properties defined by the interface. This is done on lines 4 to 21. Properties are returning some values, but not null! Methods are doing nothing. From line 23 to 36 is the Singleton definition. If Singleton is object you have defined then it should have private constructor so no one is able to instantiate it. There is a private field which actually hold the reference to the singleton. NULL is a property which instantiates the singleton if not already instantiated and returns it.

public class NullWebElement : IWebElement
{
	private const string nullWebElement = "NullWebElement";
	public bool Displayed { get { return false; } }
	public bool Enabled { get { return false; } }
	public Point Location { get { return new Point(0, 0); } }
	public bool Selected { get { return false; } }
	public Size Size { get { return new Size(0, 0); } }
	public string TagName { get { return nullWebElement; } }
	public string Text { get { return nullWebElement; } }
	public void Clear() { }
	public void Click() { }
	public string GetAttribute(string attributeName) { return nullWebElement; }
	public string GetCssValue(string propertyName) { return nullWebElement; }
	public void SendKeys(string text) { }
	public void Submit() { }
	public IWebElement FindElement(By by) { return this; }
	public ReadOnlyCollection<IWebElement> FindElements(By by)
	{
		return new ReadOnlyCollection<IWebElement>(new List<IWebElement>());
	}

	private NullWebElement() { }

	private static NullWebElement instance;
	public static NullWebElement NULL
	{
		get
		{
			if (instance == null)
			{
				instance = new NullWebElement();
			}
			return instance;
		}
	}
}

There are two main benefits of code above. First benefit comes from the Null object pattern. You can locate an element which doesn’t exists and call some of its methods without your tests to crash. You might say that comparing for null is not a a big overhead, but in big having a null check before each action is a waste of time.

IWebElement element = null;
try
{
	element = webDriver.FindElement(By.CssSelector("notExisting"));
}
catch
{
	element = NullWebElement.NULL;
}
element.Click();

It might be a discussion whether it is better to silent this failure error with null object or leave to massive crash. My opinion is you must fail the test and if you put some logging into Click method of the NullWebElement then you can easily trace the issue. The real benefit is you will have only one failed test instead of whole bunch failed because of the crash.

Second benefit comes from the Singleton pattern. You can easily compare some element against the NullWebElement.NULL. Most likely you will locate the element and use it safely because of null object pattern, but there might a case where you want to see if element is actually not found.

if (element == NullWebElement.NULL)
{
	Console.WriteLine("Element not found!");
}

Conclusion

Singleton is a pattern that you definitely should know. Null object with combination of Singleton can decrease the amount of code you write by skipping the checks for null. I would say if adopted those can simplify your code which is a benefit in a big projects.

Read more...

Factory design pattern

Last Updated on by

Post summary: Description with code samples of Factory design pattern.

Factory pattern is used to create objects based on specific rules.

More details

You may have several classes implementing one and the same interface. You might not know what object is suitable to get instantiated at given point in your code. Object creation may depend on different if/else or switch statements. You definitely do not want to make those is/else check every time you need and object. This is why you handle the object creation task to the factory which knows exactly what and how to create an object. The factory always returns a newly created object or re-initialised one.

References

This post is part of design patterns guide series. Code examples are located in GitHub for C# and Java.

Example

In example I’ve given here I’ve implemented a factory that creates IWebDriver object based on specific input. Object creation logic is encapsulated from outer world. This factory is very simple but is just fine for our purposes. More complex factories are defined by Factory method pattern and Abstract factory pattern but they are suitable for more complex applications. Factory in the example have only CreateInstance method. In it WebDriver is instantiated based on given from outside string with its name. It doesn’t store or re-use WebDriver object but always creates a new one.

public class WebDriverFactory
{
	public IWebDriver CreateInstance(string browser)
	{
		if ("Chrome".ToLower() == browser.ToLower())
		{
			return new ChromeDriver();
		}
		else if ("InternetExplorer".ToLower() == browser.ToLower())
		{
			return new InternetExplorerDriver();
		}
		else
		{
			return new FirefoxDriver();
		}
	}
}

Factory is used to instantiate WebDriver in tests. Condition which browser to instantiate may come from external parameter.

WebDriverFactory factory = new WebDriverFactory();
IWebDriver webDriver = factory.CreateInstance("Chrome");
webDriver.Url = "http://automationrhapsody.com";
webDriver.Navigate();

IWebElement logo = webDriver.FindElement(By.CssSelector("div#header-inner h1 a"));
logo.Click();

webDriver.Quit();

Conclusion

Factory is good to be used in automation testing as you have centralised control over WebDriver creation. This pattern will definitely keep you DRY. With the time being you may add more conditions on object creation (logging, different ports, etc.).

Read more...

Facade design pattern

Last Updated on by

Post summary: Description with code samples of Facade design pattern.

Facade design pattern provides simple and easy to use interface to a larger and more complex code, API or set of APIs.

More details

If you have to deal with complex or poorly designed API you might find yourself confused by all the functionality available in it. Most likely you don’t need everything provided by the API. So for ease and better maintainability only API features that are needed gets exposed outside the facade. In this way you simplify the API usage saving time for your colleagues when maintaining this code. Also you have control over how this external API is used and can prevent misunderstanding or misuse of it. If more functionality from API is needed this will require update of the facade or adding new facade which exposes it.

References

This post is part of design patterns guide series. Code examples are located in GitHub for C# and Java.

Example

It might not be a good idea to have the pattern in the name of the facade, but in the example I’ve put it to ease. In this example WebDriver instantiation is hard coded to Firefox, but in real life it will be instantiated based on some rules. Facade exposes Start, Stop and FindElement methods. All other WebDriver functionality is not accessible.

public class WebDriverFacade
{
	private IWebDriver webDriver = null;
	private TimeSpan waitForElement = TimeSpan.FromSeconds(5);

	public WebDriverFacade()
	{
		webDriver = new FirefoxDriver();
	}

	public void Start(string url)
	{
		webDriver.Url = url;
		webDriver.Navigate();
	}

	public void Stop()
	{
		webDriver.Quit();
	}

	public IWebElement FindElement(By by)
	{
		try
		{
			WebDriverWait wait = new WebDriverWait(webDriver, waitForElement);
			return wait.Until(ExpectedConditions.ElementIsVisible(by));
		}
		catch
		{
			return null;
		}
	}
}

Facade is used exactly the same way as WebDriver itself:

WebDriverFacade webDriver = new WebDriverFacade();
webDriver.Start("http://automationrhapsody.com");

IWebElement logo = webDriver.FindElement(By.CssSelector("div#header-inner h1 a"));
logo.Click();

webDriver.Stop();

Conclusion

WebDriver API is neither complex nor poorly designed, so maybe making a facade is not a mandatory thing. Still I think having control over its usage is a good approach. In given example you can see I use explicit wait so I locate exactly the same way all elements in my automation project.

Read more...

Page objects design pattern

Last Updated on by

Post summary: Description with code samples of Page Objects design pattern.

This is the most important pattern related to software automation. It enables you to create object repository with UI elements. Elements are separated from test logic. This pattern makes code much more maintainable and reusable.

References

This post is part of design patterns guide series. Code examples are located in GitHub for C# and Java.

Never do this

Without this pattern what you might do is start WebDriver, navigate to some test page. Locate element you need then click it.

IWebElement element = webDriver.FindElement(By.CssSelector("div.find label"));
element.Click();

This is good when you want to test the idea or do some quick demo. In a real life project this element might be needed in dozens of tests. What happens if the UI changes and CSS selector is not matching any more? Here comes the maintainability problem, you have to search and replace all of them.

The proper way

In software engineering there is a principle called DRY for short. Its idea is to have each element or action stored only once in a system. This avoids copy/paste and reduces the overhead for code maintenance. The same idea is used in Page Object pattern.

  • Each page or re-usable part of a page (i.e. header, footer, menu) is separate class.
  • Class constructor takes WebDriver as argument and uses it internally to locate elements.
  • Each element is a private property (or getter in Java).
  • Actions are public and internally operate with elements.

In code bellow SearchField is private property used only by SearchFor method which is exposed as available action on HomePage. Element can be located inside the action method but suggested approach gives better readability. And if element is needed more than once then defining it separately is a must.

public class HomePage
{
	private IWebDriver webDriver;

	public HomePageObject(IWebDriver webDriver)
	{
		this.webDriver = webDriver;
	}

	private IWebElement SearchField
	{
		get { return webDriver.FindElement(By.Id("search")); }
	}

	public void SearchFor(string text)
	{
		SearchField.SendKeys(text);
	}
}

Page object is instantiated in the test and actions are invoked.

HomePageObject homePage = new HomePageObject(webDriver);
homePage.SearchFor("automation");

With this approach you have one element defined on only one place. Only actions are exposed out of the page object. It is very clear what actions can be done on this page.

Original page objects

To be honest what I have described in this post is not actually the original Page Objects pattern. Although the idea is same I have modified it slightly. Originally each action returns a page object. In this way you can chain methods in one test and very easy to understand if a test will be broken when given action now returns different page object. I do not like chaining of methods though so I don’t need this extra. It doesn’t bother you at all to do

public HomePage SearchFor(string text)
{
	SearchField.SendKeys(text);
	return this;
}

Also I have separated elements location outside of actions because I want better maintainability and readability of the code.

Conclusion

Always use page objects in you test automation. ALWAYS!

Read more...

Design patterns every test automation engineer should know

Last Updated on by

Post summary: Description with examples of 5 design patterns that is good to know and use in our automation testing.

Design patterns are interesting topic. They are created to solve common problems in software design. Although design patterns are not reserved only for software development they seem not to be widely discussed in software automation. Maybe this is because the topic sounds complicated. Yes, there are really sophisticated design patterns used to solve complex issues in software development. But also there are easy to understand and adopt design patterns that can significantly improve readability and maintainability of our test automation code.

With series of posts I’m going to give overview with examples on some design patterns that can be very useful in our test automation projects. To keep blog neat in this first post I’ll just outline the list of patterns. Each one will be discusses in separate post with code samples.

Page Objects pattern

This is the most important pattern related to software automation. It enables you to create object repository with UI elements. Elements are separated from tests logic. This pattern makes code much more maintainable and reusable. Click for more details and code samples.

Facade pattern

The whole idea of facade design pattern is to provide simple and easy to use interface to a larger and more complex code, API or set of APIs. Most likely you don’t need everything provided by the API. So for ease and better maintainability only API features that are needed gets exposed outside the facade. In this way you simplify the API usage and you have control over how this external API is used and can prevent misunderstanding or misuse. Click for more details and code samples.

Factory pattern

Factory pattern is used to create objects based on specific rules. You may have several classes implementing an interface. In your code you do not want to bother defining which concrete class to instantiate or you might not know what object is suitable to get instantiated. This is why you handle the object creation task to the factory which knows exactly what object to create. With factory pattern object creation is encapsulated. Click for more details and code samples.

Singleton pattern

Singleton pattern is needed when you need exactly one object from a specific class in the whole application. Click for more details and code samples.

Null object pattern

The idea is to make code more simpler and safe by skipping null reference checks. Null object implements given interface and its methods are doing nothing. This makes the null object predictable. You can safely invoke methods on the null object without the threat of a NullReferenceException to break your application. Click for more details and code samples.

Putting all together

Once you have gone through patterns one by one now it is time to put everything in one project. Complete guide how to use design patterns in automation post is showing how.

Read more...

Selenium WebDriver cannot click UTF-8 icons

Last Updated on by

Post summary: Selenium WebDriver is not able to click UTF-8 icons. Solution is to use jQuery code.

In given example there is simple search form. Search value is cleared by “X” icon. This seems pretty forward case to be automated with Selenium WebDriver. It just seems! Element is properly located but when Selenium tries to click it an ElementNotVisibleException is thrown.

UTF-8 icon

When you inspect the code you can notice element is an empty SPAN. It has UTF-8 icon displayed over it by CSS “content” property. With this approach it is very easy to visualise very easy vast amount of interesting icons without being proficient in image editing.

Debug

I’ve setup very simple Selenium project for Visual Studio 2013 in my GitHub repository. On debug element can be located, but its width=0 so Selenium treats it as Displayed=false. This seems a good candidate for Selenium bug.

// Locate element
IWebElement element = webDriver.FindElement(By.CssSelector("span.cancel"));
// Debug info
bool isDisplayed = element.Displayed; // false
int width = element.Size.Width; // 0
int heigth = element.Size.Height; // 17

Solve it

Problem solved with execution of jQuery that will do the actual click of the element:

// jQuery workaround of the click
((IJavaScriptExecutor)webDriver).
	ExecuteScript("$('span.cancel').click()");

If jQuery is not available then try to do the click with function of JavaScript library used by the application you are automating. If none is used they you can just use DOM JavaScript call:

((IJavaScriptExecutor)webDriver).
	ExecuteScript("document.getElementsByClassName('cancel')[0].click();");

Bonus

There is one more interesting thing about jQuery. You can execute jQuery with some specific selector logic and this will return same IWebElement if you have located it with WebDriver.

// Locate element with jQuery
element = (IWebElement)((IJavaScriptExecutor)webDriver).
	ExecuteScript("return $('span.cancel')[0]");

One small detail – [0] is needed in the end of jQuery code otherwise a jQuery object is returned and Selenium is not able to cast it to IWebElement. And ofcourse element is yet not clickable. I’m just giving an alternative way of locating elements if Selenium way gets too complicated.

Conclusion

jQuery can be used inside Selenium WebDriver for tasks which are impossible to be done otherwise or are too hard and not worth wasting time on it. jQuery can be quite helpful in your automation. So it is worth improving your skill set with it. Remember, the essence of automation is about saving your company time and money, not wasting them on insignificant tasks.

Read more...

Efficient waiting for Ajax call data loading with Selenium WebDriver

Last Updated on by

Post summary: This post is about implementing an efficient mechanism for Selenium WebDriver wait for elements by execution of jQuery code.

Automating single page application with Selenium WebDriver could be sometimes a tricky task. You can get into the trap of timing issues. Although you set explicit waits you still can try to use element that is not yet loaded by the Ajax call. Remember Thread.Sleep() is never an option! You can use very tiny sleeps (100-200ms) in order to wait for initiation of given process, but never use sleep to wait for the end of the process.

Implement Selenium wrapper (Facade)

I good approach I like is to implement your own FindElement method which is basically a wrapper for Selenium’s methods (Facade pattern). With this approach you are hiding unneeded Selenium functionality and have centralised control over location of elements and explicit waits. Locate behaviour of your entire framework is controlled in just one method.

private static TimeSpan waitForElement = TimeSpan.FromSeconds(10);

public static IWebElement FindElement(By by)
{
	try
	{
		WaitForReady();
		WebDriverWait wait = new WebDriverWait(webDriver, waitForElement);
		return wait.Until(ExpectedConditions.ElementIsVisible(by));
	}
	catch
	{
		return null;
	}
}

Code above is C# one and is implementation of explicit wait with WebDriverWait class from OpenQA.Selenium.Support.UI. You can see an unknown (so far) method WaitForReady(). Note that ElementIsVisible is used instead of ElementExists because element might be on page but yet not ready to work with.

Wait for Ajax call to finish

Initially WaitForReady() was supposed to check that Ajax has finished loading by using jQuery.active property. This is in case jQuery is used in the application under test. If this property is 0 then there are no active Ajax request to server.

private static void WaitForReady()
{
	WebDriverWait wait = new WebDriverWait(webDriver, waitForElement);
	wait.Until(driver => (bool)((IJavaScriptExecutor)driver).
			ExecuteScript("return jQuery.active == 0"));
}

Wait for Ajax call to finish and data to load

You can realise that sometimes it is not enough to wait for Ajax to finish rather than to wait for data to be rendered. There is fancy loader in my application under which is a DIV shown when some action is being performed. If there is one on your application then you’d better wait not only for Ajax to finish, but to loader to hide.

private static void WaitForReady()
{
	WebDriverWait wait = new WebDriverWait(webDriver, waitForElement);
	wait.Until(driver =>
	{
		bool isAjaxFinished = (bool)((IJavaScriptExecutor)driver).
			ExecuteScript("return jQuery.active == 0");
		try
		{
			driver.FindElement(By.ClassName("spinner"));
			return false;
		}
		catch
		{
			return isAjaxFinished;
		}
	});
}

If “spinner” location gives exception then loader is not present and we can stop waiting. Good!

Improve the wait for data load

What about performance? When putting a timer the result was ~300ms for each Selenium search for loader. Not so good… Is 300ms long? Sure not, but taking into consideration this is called every time an element is located then this could make a huge difference in test execution times.

Why not making the same check for hidden loader, but this time with a JavaScript call to browser? I’m familiar with jQuery, then why not.

private static void WaitForReady()
{
	WebDriverWait wait = new WebDriverWait(webDriver, waitForElement);
	wait.Until(driver =>
	{
		bool isAjaxFinished = (bool)((IJavaScriptExecutor)driver).
			ExecuteScript("return jQuery.active == 0");
		bool isLoaderHidden = (bool)((IJavaScriptExecutor)driver).
			ExecuteScript("return $('.spinner').is(':visible') == false");
		return isAjaxFinished & isLoaderHidden;
	});
}

Conclusion

Same logic to check that element with class=”spinner” is not visible on page but this time at a cost of ~30ms. I like it much better this way!

Read more...

Automating SignalR applications with Selenium WebDriver

Last Updated on by

Post summary: This post is about an “No response from server for url” issue during automation of web application using SignalR with Selenium WebDriver. Issue was resolved by configuring application to connect to server through WebSocket protocol.

I had to automate with Selenium WebDriver (version 2.43) a single page web application which was build with SignalR. First thing to do when you start an automation is to automate a smoke test scenario. So did I. It run fine on Internet Explorer (version 11) and Chrome (version 37). I was happy and confident I’m going to finish this project on time. The happy face was dramatically changed when I run the suite on Firefox (version 33). The driver timed out with “No response from server for url” issue. I searched the net for similar problems with no luck. I couldn’t find an end-to-end solution on issues with SignalR automation so I prepared this post.

About SignalR

ASP.NET SignalR is a library for building “real-time” applications that enables server to push events to client instead on relying client to request data from server. Once application is started SignalR client tries to connect to server with transport protocols in order shown in the list: WebSocket, Server-sent events, Forever Frame (IE only) and finally Ajax long polling.

My application was forced to connect to server with Long Polling transport (later on I discovered this was caused by a bug in application’s connection logic). Client (browser) opens connection to server. Server keeps this connection open for relatively long time (2 minutes in my case). Seems like this open connection is confusing Selenium and it doesn’t actually know when browser is ready. “Unstable” page loading strategy did not worked out:

FirefoxProfile profile = new FirefoxProfile();
profile.setPreference("webdriver.load.strategy", "unstable");
WebDriver driver = new FirefoxDriver(profile);

The only solution to make Selenium working with SignalR in all three browsers was to make application under test working with WebSocket transport protocol.

It is also beneficial for application itself to work with WebSockets. There are several SignalR prerequisites in order to use it with WebSockets: IIS 8 or IIS 8 Express with enabled WebSockets. In order to get enabled Web Sockets are installed additionally to IIS as a Windows feature.

Read more...