Monthly Archives: February 2015

NTestsRunner for functional automated tests

Last Updated on by

Post summary: NTestsRunner implementation details and features.

In previous post I’ve described unit testing frameworks and why they are not suitable for running functional automated tests. I introduced NTestsRunner – very simple runner that can be used for running your automation tests. This topic is dedicated for implementation details of the NTestsRunner.

Verifications

It is important in functional testing to be able to place several verification points in one test. For this purpose abstract class Verification is implemented. It has two properties to store more details about verification and time it was taken. Constructor receives comma separated string values. In case of zero strings are passed then result is empty string. If one string is passed then this is the result. If more than one string is added then first string is taken as formatting string and others are used to build up the result. Logic is similar to string.Format(String, Object[]) method.

public abstract class Verification
{
	public string Result { get; private set; }
	public DateTime ExecutedAt { get; private set; }

	public Verification(params object[] args)
	{
		...
	}
}

Passed or Failed

In automation test may have two conditions – passed or failed. This is why two concrete classes are extending Verification: VerificationPassed and VerificationFailed. They do not add any other functionality. Those classes use parent’s class constructor. This is example how to instantiate object from those classes:

string value = "number";
int number = 1;
Verification result =
	new VerificationFailed("This is formatting string {0} {1}. ",
		value,
		number);

Test case result

Test case is generally a set of conditions to verify whether given scenario works are per user requirements. In automation world test case is test method with several verification points inside. In NTestsRunner TestCaseResult is class representing the idea of a test case. It has properties for name, time to run and list of all verifications with count of passed and failed.

public class TestCaseResult
{
	public List<Verification> Verifications = new List<Verification>();
	public string Name { get; set; }
	public int VerificationsFailed { get; set; }
	public int VerificationsPassed { get; set; }
	public TimeSpan Time { get; set; }
}

Test plan result

TestPlanResult in NTestsRunner has nothing to do with test plan term from QA world. Here this is a representation of a test class with test methods inside. It has properties for name and time to run. Also there is list with all TestCaseResults, i.e. test methods in that class. There are counters for passed and failed test cases and also counters for all passed and failed verifications inside all TestCaseResults.

public class TestPlanResult
{
	public List<TestCaseResult> TestCases = new List<TestCaseResult>();
	public string Name { get; set; }
	public int TestCasesPassed { get; private set; }
	public int TestCasesFailed { get; private set; }
	public int VerificationsPassed { get; private set; }
	public int VerificationsFailed { get; private set; }
	public TimeSpan Time { get; private set; }

	public void Count()
	{
		...
	}
}

Class and method attributes

In order to make one class a test class it should have with [TestClass] attributes. To convert method to a test one it should have [TestMethod] attribute. Just the attribute is not enough though. Method should have special method signature. This is required by NTestsRunner.

Test method signature

In order to run without exception test method needs to conform to two rules:

  1. To have attribute [TestMethod]
  2. Method to receive parameter List verifications in its signature, i.e.
    [TestMethod]
    public void TestMethod1(List<Verification> verifications)
    

Configurations

Configurations can be found on NTestRunner home page.

Execution

Once object from NTestsRunner is instantiated and configured tests with Execute() method. In side this method all classes from calling assembly (the one that holds the tests) are taken. If TestsToExecute is configured then only those with name matching given values are taken. If no TestsToExecute is provided then all classes with attribute [TestClass] are taken. Methods from each class are taken by default in order of appearance in the class. If method has [TestClass] attribute then method is executed by passing List object to it. Inside the method Verifications are collected as list into TestCaseResult object. After method is run TestCaseResult is added to its parent TestPlanResult which is added to list with all results. In the end results are saved as XML and HTML.

Results in jUnit XML

In order to integrate with CI tools such as Jenkins or Bamboo results are exported to XML file after execution has finished. File is named Results.xml and is located in test results folder. XML format is implemented according to junit-4.xsd.

Results in HTML

Tests result are saved as HTML report for better readability. File is named Results.html and is located in test results folder.

Usage

In order to use NTestsRunner a console application project is needed. This project will hold test classes. As the one bellow. Take into consideration that this is very simplified usage pattern. In reality Page objects design pattern will be used. Page objects will make the verifications and return them.

[TestClass]
public class TestClass1
{
	[TestMethod]
	public void TestMethod1(List<Verification> verifications)
	{
		// Do some actions
		verifications.Add(new VerificationFailed("There is error"));
		// Do some actions
		verifications.Add(new VerificationPassed("Everythign is OK"));
	}
}

In its main method new instance of NTestsRunner is created. Configurations are done and test executions is started. It is that simple to use it.

class Program
{
	static void Main(string[] args)
	{
		NTestsRunnerSettings settings = new NTestsRunnerSettings();
		settings.TestResultsDir = @"C:\temp";
		settings.MaxTestCaseRuntimeMinutes = 2;
		settings.TestsToExecute.Add("TestClass1");
		settings.PreventScreenLock = true;

		NTestsRunner runner = new NTestsRunner(settings);
		runner.Execute();
	}
}

Pros and cons

NTestsRunner has its pros and cons.
Pros are:

  • Pretty easy to use
  • Open source and can be customised to you specific needs
  • Gives you ability to make several verifications in one test and in case of fail it doesn’t break current test method
  • Tests are stored into console application that can be easily run
  • Results are saved in jUnit XML for CI integration
  • Results are saved in HTML

Cons are:

  • Test methods should have specific signature
  • It is not easy to migrate existing tests to new format

Conclusions

This is pretty good tool for running functional automated tests. It is very easy to use and is made especially for running functional automated tests. You can definitely give it a try.

Read more...

Running functional automation tests

Last Updated on by

Post summary: Unit testing frameworks are not very suitable for running functional tests. NTestsRunner is an alternative way of running functional automated tests.

Unit testing

Unit testing is focused on testing code on low level. Methods, sets of methods or modules are being tested by writing test code which invokes those methods with specific arguments. In unit testing all external dependencies (database, file system, network, etc.) are removed. Those resources are simulated in unit tests by using so called mock objects. Mock objects are controlled by tests designer and have predictive behaviour. Running piece of code which doesn’t have external dependencies happens almost immediately. So unit tests are executed for very low amount of time. It is considered set of unit tests taking longer than 5 minutes is not well designed. Unit tests are strictly focused. One test tests only one condition. Each test is not related in anyhow to other tests.

Unit testing frameworks

Unit testing frameworks conform to unit tests purpose and design. Tests should not depend on each other. For this reason unit testing frameworks execute tests in random order (xUnit.net or MS Unit Testing Framework), other like NUnit in alphabetic order of tests method names. Checking for given conditions are done with assertions. If one assert fails current test execution is stopped and test is marked as failed.

Functional testing

Functional testing is focused on ensuring that software product works as per user requirements. Real life software has external dependencies. Most software products has some kind of user interface. Automation tests are focused on verifying that UI works correctly. Transferring and rendering data to UI takes time, database operations take time, file and network operations also take time, etc. In general functional tests are more complex and take much longer to execute. In order to be efficient checking of several conditions are defined in each test. Functional tests can be manual and automated. In current post when I mention functional tests I mean only automated.

Requirement for running functional test – many verifications

In a perfect situation functional one test should verify only one condition. In reality because of too many external dependencies time for tests execution is large. An time matters. In order to shorten this time we make several checks in one test. For e.g. in e-commerce web site order is placed. We verify order confirmation page that there is order number, that address is same used during checkout, that user’s email is correct. We also can verify inside user’s email box that received mail is correct. We can check in database for some properties of the order. If we have to do one order for each check tests will take significantly long time. To be efficient we do all checks with one order in one test case. We need a framework which allows you to have multiple verifications in one test. Further more if verification is failed test execution should continue.

Requirement for running functional test – controlled sequence

What is really good to be avoided but sometimes cannot is test dependency. Sometimes one tests needs another to have done something before continuing. As I said this should be avoided but in order to be efficient you should make trade off between good tests design and time to execute. For e.g. you may want to cancel and then refund order placed in e-commerce web site. Generally it is best to place new order for this test but if placing of order requires too much time then an option is to reuse already existing order from previous test. We need to be able to control test case execution order.

NTestsRunner

In order to have control over tests and use many verifications I’ve created NTestsRunner. Its code is in GitHub NTestsRunner repository. This is .NET library. You create console application with your tests and use the library within. Tests are annotated is similar fashion as with unit testing frameworks. Tests are executed in sequential order. There could be many verifications. Results are saved as HTML and XML in jUnit format. NTests runner is described in more details in this post NTestsRunner for functional automated tests.

Read more...

WPF automation – running the tests

Last Updated on by

Post summary: How to sell your automation to management. Guide for running the tests unattended.

References

This post is part of Automation of WPF applications with Telerik Testing Framework and TestStack White series. Sample application can be found in GitHub SampleApp repository.

Test frameworks need mouse and keyboard

As you have noticed when running the examples it is not possible to do anything else while tests are running because both Telerik Testing Framework and TestStack White are using mouse and keyboard in order to click and type text. This is how both frameworks are designed and work. It doesn’t seem very effective if you are wasting you time watching tests running on your workstation instead of doing something productive. This is not a good argument when advocating your automation.

The essence of automation is being effective

This topic is not about Automation vs. Manual testing so I will not go in that direction. I’ll just say that still there are companies lacking management willpower to support and embrace automation. So we need to be a good salesmen!

How to drive you automation to success

As I said we need to be wise when promoting our automation to get more time and resources. Sales strategy is pretty easy and straight forward not requiring huge investments:

  • Free to use and pretty easy to work with frameworks.
  • Some enthusiasm to make the first automation. Remember, be smart and first automate most repeating scenarios. This will show real results and help you buy some time for further automation.
  • Virtual (or real) machine with test running on it during the night
  • Mail with results is sent to management with results in the morning

Run the tests unattended

You get the machine, set up the framework, set up scheduled tasks for deploying latest application under test and latest test code. Run the tests and get into trouble. Tests does not run! This is because there are special requirements to schedule unattended run. There must be an active Windows session in order mouse and keyboard to be used. Once session is interrupted tests stop. This is detailed KB article on the topic with several possible solutions.

Working solution

You can try solutions in the article to see which works best for you. For me solution is to have remote desktop in remote desktop. This requires Windows Server installation. Only server provides two simultaneous remote desktop sessions. There are unofficial patches for non-server versions which I haven’t tried and cannot comment. Two local users are needed on the Windows Server with no desktop locking or screen saver (domain users most likely will have desktop locked after a time). Log from your machine to test machine with first user. From first user’s session log to same testing machine with second user. Tests are started from second user’s session. Once tests are started you can freely close desktop (not log out!). You can create scheduled task which runs only when user is logged in and just wait results in the morning. If two accounts are overhead there is option to use software that prevents your computer to lock. See prevent screen lock thread if something works for you.

Conclusions

Automation is exciting field of career development for test engineers. This blog is dedicated to automation testing. You will find very useful and interesting topics in it. I would definitely encourage you just to give it a try. Good luck!

Read more...

WPF automation – using the elements

Last Updated on by

Post summary: Use already created Page Objects and build up test framework.

References

This post is part of Automation of WPF applications with Telerik Testing Framework and TestStack White series. Sample application can be found in GitHub SampleApp repository.

Page Objects holder

Bellow is App.cs which is representation of application under test.

using ArtOfTest.WebAii.Wpf;
using White.Core;
using White.Core.UIItems.WindowItems;

namespace SampleApp.Tests.Framework.Elements
{
	public class App
	{
		public WpfApplication ApplicationWebAii { get; private set; }
		public Application ApplicationWhite { get; private set; }

		public App(WpfApplication webAiiApp, Application whiteApp)
		{
			ApplicationWebAii = webAiiApp;
			ApplicationWhite = whiteApp;
		}

		public MainWindow MainWindow
		{
			get
			{
				return new MainWindow(ApplicationWebAii
					.WaitForWindow(MainWindow.WINDOW_NAME).Find);
			}
		}

		public OpenFile OpenFile
		{
			get
			{
				return new OpenFile(GetWindowByName("Open"));
			}
		}

		public MessageBox MessageBox
		{
			get
			{
				return new MessageBox(GetWindowByName(""));
			}
		}

		private Window GetWindowByName(string windowName)
		{
			// Workaround as method GetWindow(string title) is not working
			foreach (Window window in ApplicationWhite.GetWindows())
			{
				if (windowName.Equals(window.Name))
				{
					return window;
				}
			}
			return null;
		}
	}
}

Constructor takes instance of Telerik Testing Framework’s application (WpfApplication) and TestStack White’s application (Application). Those are stored inside the App instance.

Access the Page Objects

Each window in real application is represented by a property in the App class. When accessed a new object of this page object class is created and its elements can be accessed.

WPF page objects require VisualFind in order to be instantiated. It is obtained by first locating the window with Telerik’s

public WpfWindow WaitForWindow(string caption);

From located window we need only the VisualFind which is used internally to locate elements on that particular window.

WinForms page objects require White’s Window instance in order to be instantiated. Window is located by

public virtual Window GetWindow(string title);

I found this method not always working so I’ve made a workaround method

private Window GetWindowByName(string windowName);

New Page Objects vs. Cached Page Objects

In the example above every time an action is required a new page object is instantiated. In some cases instantiating the object may require longer time or you might need some properties in this object preserved during tests. In such cases you may use Singleton design pattern and instantiate only one object.

private MessageBox messageBox = null;
public MessageBox MessageBox
{
	get
	{
		if (messageBox == null)
		{
			messageBox = new MessageBox(GetWindowByName(""));
		}
		return messageBox;
	}
}

Both approaches have pros and cons. In case of new page object you always work with a fresh instance without any previous state saved. This might require more time to instantiate the objects and you are not able to save previous states. Cached objects may be much faster as a performance but having internal state may lead to unexpected bugs in your automation.

Base test

Finally to make all work we need an instance of App. Instance is created in BaseTest.cs class.

using ArtOfTest.WebAii.Core;
using SampleApp.Tests.Framework.Elements;
using White.Core;

namespace SampleApp.Tests.Framework.Tests
{
	public class BaseTest
	{
		protected App App { get; set; }
		private string applicationPath =
			"C:\\SampleApp\\SampleApp\\bin\\Debug\\SampleApp.exe";

		protected void Start()
		{
			if (App == null)
			{
				Application appWhite = Application.Launch(applicationPath);
				Manager manager = new Manager(false);
				manager.Start();
				App = new App(
					manager.ConnectToApplication(appWhite.Process), appWhite);
			}
		}

		protected void Stop()
		{
			if (App != null && App.ApplicationWhite != null)
			{
				App.ApplicationWhite.Kill();
			}
			App = null;
		}
	}
}

All tests inherit from the base test class. Initialise and clean up code is added in base test. In our case Start() method is the initialiser. It must be called in order to instantiate App class. App property is protected so every extending class has access to it.

Initialise the frameworks

In order to start the application under test we need full path to the exe file. In this example this is hard coded but in real life it will be configurable. Start the application with White’s

public static Application Launch(string executable);

Once started connect to it with Telerik framework by creating a Manager and use its

public WpfApplication ConnectToApplication(Process proc, string pid = null);

Process is obtained out of White Application.Process property. Opposite launch order order is not working. White is not able to Attach to running process.

Use page objects

Once Start() method is called, application under test is started and both frameworks are connected to it you can simply do in your test:

App.MainWindow.ClickBrowseButton();

This will find and create new instance of MainWindow and then it will find and click Browse button. Your framework defines the actions on elements which are later used in actual tests. Once all the work on framework has been done it is that simple to build your tests.

Clean up

Stop() method is called in the end of the test in order to close the application under test by killing underlying process.

The tests

This is unit test created with MS Unit Testing Framework in order to demonstrate real testing on application.

using Microsoft.VisualStudio.TestTools.UnitTesting;
using SampleApp.Tests.Framework.Tests;

namespace SampleApp.Tests
{
    [TestClass]
    public class UnitTest1 : BaseTest
    {
		[TestInitialize]
		public void Initialise()
		{
			Start();
		}

		[TestMethod]
		public void OpenFile_OnCancel_GivesMessage()
		{
			App.MainWindow.ClickBrowseButton();
			App.OpenFile.ClickCancelButton();
			Assert.AreEqual("Problem occured, try again later",
				App.MessageBox.GetText());
			App.MessageBox.ClickOkButton();
		}

		[TestMethod]
		public void OpenFile_OnAttachFile_GivesMessageAndFileIsShown()
		{
			string filePath = @"C:\SampleApp\SampleApp\bin\Debug\HappyFace.jpg";
			App.MainWindow.ClickBrowseButton();
			App.OpenFile.EnterFileName(filePath);
			App.OpenFile.ClickOpenButton();
			Assert.AreEqual("Successfully done", App.MessageBox.GetText());
			App.MessageBox.ClickOkButton();
			Assert.AreEqual(filePath, App.MainWindow.GetFilePathAtIndex(1));
		}

		[TestCleanup]
		public void CleanUp()
		{
			Stop();
		}
    }
}

Unit testing frameworks

Unit testing frameworks are designed to run tests in random order. Before each test method annotated with [TestInitialize] is run. In our case application is started. After each test method annotated with [TestCleanup] is run. In our case application is stopped. For this simple application running tests with unit testing framework is OK. We are not doing unit tests but functional once. So for bigger and more complex tests unit testing frameworks are not very convenient. I’ve created very simple tests runner. This post describing the need of such tests runner.

This post shows how to build up the framework based on page objects. Next post is WPF automation – running the tests.

Read more...

WPF automation – locating and structure of WinForms elements

Last Updated on by

Post summary: Guide how to locate WinForms elements with TestStack White.

References

This post is part of Automation of WPF applications with Telerik Testing Framework and TestStack White series. Sample application can be found in GitHub SampleApp repository.

WinForms elements manipulation with TestStack White

MessageBox.cs class is representation of a MsgBox of our sample application which is WinForms.

using System.Windows.Automation;
using White.Core.UIItems;
using White.Core.UIItems.Finders;
using White.Core.UIItems.WindowItems;

namespace SampleApp.Tests.Framework.Elements
{
	public class MessageBox
	{
		private Window win;
		public MessageBox(Window window)
		{
			win = window;
		}

		private Button Button_OK
		{
			get
			{
				return win.Get<Button>(
					SearchCriteria.ByControlType(ControlType.Button).
					AndByText("OK"));
			}
		}
		private Label Label_Text
		{
			get
			{
				return win.Get<Label>(
					SearchCriteria.ByControlType(ControlType.Text));
			}
		}

		public void ClickOkButton()
		{
			Button_OK.Click();
		}

		public string GetText()
		{
			return Label_Text.Text;
		}
	}
}

MessageBox is following Page Object design pattern. Elements are private properties. Actions are public methods acting on elements. Again: one element MUST be defined in only one place!

Constructor receives instance of current window object. Elements are located by White framework method:

public virtual T Get<T>(SearchCriteria searchCriteria) where T : UIItem;

SearchCriteria

SearchCriteria is used for locating of elements. Search is possible with combination of following criteria: Text, ControlType, AutomationId, NativeProperty, Framework. And here comes the hardness of working with White in complex UIs. There may be several TextBox-es (ByControlType == Edit) AND with no text inside (AndByText == “”). You can take them all and then to access by index the one which is proper for your test. And what happens when you fill text in one of them. Next find has one less window. If you think it is easy then enjoy, I would rather not do it.

UISpy

Finding of correct SearchCriteria is done with powerful tool UISpy.exe. Unfortunately this one is deprecated from Windows 8. MS have added a replacement called Inspect.exe. I don’t find it useful this is why I have added UISpy.exe in the GitHub repository.

UISpy

“Dialog” is the message box window which is child of “MainWindow”. It has empty title and two elements: OK button and message text. Current example shows location of the text element. Its MS ControlType == ControlType.Text. This is what we need to locate it since there is only one such element on message box. Once we have it we take the text and test compares current vs. expected text. OK button is located by ControlType and text. In case of different localisations we may leave just the ControlType, but if there are more buttons text is mandatory to distinguish the correct button.

Once you have defined your page objects it is time to assemble them. Next post is WPF automation – locating and structure of WinForms elements.

Read more...

WPF automation – locating and structure of WPF elements

Last Updated on by

Post summary: Guide how to locate WPF elements with Telerik Testing Framework.

References

This post is part of Automation of WPF applications with Telerik Testing Framework and TestStack White series. Sample application can be found in GitHub SampleApp repository.

MainWindow Page Object

MainWindow.cs class is representation of a main window of our sample application which is WPF. Note that code here is shortened. Full code can be found in GitHub repository.

using ArtOfTest.WebAii.Controls.Xaml.Wpf;
using ArtOfTest.WebAii.Silverlight;
using ArtOfTest.WebAii.TestTemplates;
using System.Threading;

namespace SampleApp.Tests.Framework.Elements
{
	public class MainWindow : XamlElementContainer
	{
		public static string WINDOW_NAME = &quot;MainWindow&quot;;
		private string mainPath =
			&quot;XamlPath=/Border[0]/AdornerDecorator[0]/ContentPresenter[0]/Grid[0]/&quot;;
		public MainWindow(VisualFind find) : base(find) { }

		private Button Button_Browse
		{
			get
			{
				return Get&lt;Button&gt;(mainPath + &quot;Button[0]&quot;);
			}
		}

		public void ClickBrowseButton()
		{
			Button_Browse.User.Click();
			Thread.Sleep(500);
		}
	}
}

MainWindow is following Page Object design pattern. Elements are private Properties. Actions on elements are public methods that are the actual building blocks of the tests. Do not just add action methods for the sake of adding them. Add them if only the tests need such action.

If element is used in only one action then it could be inside the action method, but if element is used more than one it is mandatory to keep is as separate property. The main idea is to avoid duplications in order to improve maintainability. Remember to stay DRY: one element MUST be defined in only one place!

MainWindow class extends XamlElementContainer which comes from the framework. Constructor takes VisualFind object and passes it to XamlElementContainer’s constructor. This allows us to use framework’s methods for location of elements. In our case

public TControl Get&lt;TControl&gt;(params string[] clauses)
	where TControl : ArtOfTest.WebAii.Controls.Xaml.IFrameworkElement;

This is the strategy used by the Test Studio generated tests and I prefer it.

Different approach for MainWindow

Another element structure could be not to extend XamlElementContainer but pass VisualFind object through MainWindow’s constructor and use it internally to locate elements.

using ArtOfTest.WebAii.Controls.Xaml.Wpf;
using ArtOfTest.WebAii.Silverlight;
using System.Threading;

namespace SampleApp.Tests.Framework.Elements
{
	public class MainWindow
	{
		public static string WINDOW_NAME = &quot;MainWindow&quot;;
		private string mainPath =
			&quot;XamlPath=/Border[0]/AdornerDecorator[0]/ContentPresenter[0]/Grid[0]/&quot;;
		private VisualFind visualFind;
		public MainWindow(VisualFind find)
		{
			visualFind = find;
		}

		private Button Button_Browse
		{
			get
			{
				return visualFind.ByExpression(
					new XamlFindExpression(mainPath + &quot;Button[0]&quot;)).
					CastAs&lt;Button&gt;();
			}
		}

		public void ClickBrowseButton()
		{
			Button_Browse.User.Click();
			Thread.Sleep(500);
		}
	}
}

How to locate elements is described in finding page elements article.

XamlPath

As you can see I’m using exact XamlPath find expression in order to locate my elements because I find it consistent and easy to maintain. XamlPath syntax is pretty similar to XPath. Difference is its indexes are zero based (Button[0] is the first element) and it doesn’t provide predicates and axes. The hard part is to get the XamlPath. I have found this lovely tool WPF Inspector. It gives a lot of information about structure of WPF application.

WPF Inspector

WPF Inspector

WPF Inspector

This is screen shot of the XamlPath for “Browse” button. In more complex UIs even with this great tool it is not that easy to get the XamlPath. If you find it hard you may consider other find expressions.

All elements on MainWindow are only WPF thus instance of White is not needed. In case of WinForms content hosted inside WPF you can pass instance of White through elements’ constructor and use it in actions inside.

public MainWindow(VisualFind find, Application applicationWhite) : base(find)
{
	appWhite = applicationWhite;
}

Having instance of White inside class you can work with it as explained in next post WPF automation – locating and structure of WPF elements.

Read more...

WPF automation – projects structure

Last Updated on by

Post summary: How to structure projects in the course of WPF automation.

References

This post is part of Automation of WPF applications with Telerik Testing Framework and TestStack White series. Sample application can be found in GitHub SampleApp repository.

Overview

For better maintainability and good design automation testing is separated in two projects. One is the so called framework project. It has knowledge about third used party frameworks (Telerik Testing Framework, TestStack White, Selenium, Watij, etc.). It operates on elements via those third party frameworks. Framework exposes actions that are building blocks of the tests. The other project are the tests. It doesn’t know anything about used automation frameworks. It uses methods exposed by internal framework and builds the tests with them. The idea is that one team/person with development knowledge can be responsible for the internal framework, others can be responsible for writing well designed tests using the internal framework only. Another benefit from two projects is ability to completely change internal framework project without affecting test logic at all. Here is the structure of my sample projects:

Projects structure

Projects structure

SampleApp – This is the application under test. Generally it is not in the test project, you receive it from developers or from nightly build or from some other place. For the purpose of this demo I’ve created very basic application that uploads and image and shows it inside. There is attached image (HappyFace.jpg) that is used in the tests.

SampleApp.Tests – Here is the most important artefact – the tests, the purpose of all this fuss. In the demo this is made as UnitTest with MS Unit Testing Framework. You can use what ever run strategy you like – another unit testing framework (xUnit, nUnit, etc.) or you can build you own tests runner. I have build my own tests running because by definition unit tests should be independent from each other thus MS Unit Testing Framework runs them in random order. It might be possible to manage order, but this will require managing of some sort of external files. It gives too much overhead to me.

SampleApp.Tests.Framework – This is where the interesting part is. Here are the elements – main application form (WPF), message boxes shown on success or error (WinForms) and open file dialogue (WinForms). MainWindow is manipulated with Telerik Testing Framework, other two are manipulated with White. App.cs is a holder which represent application itself and elements are access through it. BaseTest holds and instance of App. It is extended by UnitTests and App and its elements are accessible.

Next post is WPF automation – locating and structure of WPF elements in code.

Read more...

Automation of WPF applications

Last Updated on by

Post summary: Overview how to successfully automate WPF application with Telerik Testing Framework and TestStack White.

I’ve spent lots of time in automation of a WPF application. Nowadays world is dealing with web application and desktop application are fading away. Nevertheless desktop applications still has their own private space and I hope those series of posts will be useful to someone.

Preface

Application under test is created with WPF but has WinFoms grids included on different windows presenting lots of information. Performance is the main reason to use WinFoms grids inside a WPF application. WPF grid with 40 columns and 100 rows lags a lot. After research and proof of concept on commercial and open source tools and frameworks two were chosen: Telerik Testing Framework and TestStack White. They provide flexibility and maintainability which is so vital in fast changing application. Both are used with C#. Every one has its own weaknesses and strengths but both made a perfect synergy to provide you possibility to automate almost everything.

Telerik Testing Framework

Extremely powerful and free to use framework. You can purchase additionally support which is at reasonable prices. Telerik also provide Test Studio, a tool and MS Visual Studio plug-in, build on top of the framework. This is also an option, but for me using a tool gives more restrictions than benefits. Their framework is the best available for testing WPF applications. It is very easy to use. Elements are easily located by XamlPath (very similar to XPath). Of course there are other locate options. Framework provides very rich API with lots of operations over the elements. Still its powerfulness is limited to WPF. It is not that good with WinForms elements. Telerik Testing Framework requires installation as it copies lots of assemblies in GAC.

TestStack White

Extremely powerful open source framework. Build as a wrapper of Microsoft UI Automation framework you can automate almost everything with it. But this comes at a price. It is hard to locate elements. I use it only where Telerik Testing Framework is not capable to do the work – WinForm grids, context menus, closing windows with small X in the top right corner?! Yes, those are WPF inside, but are hosted in operation system window and Telerik has no access to it.

In GitHub I have uploaded a MS Visual Studio 2013 project with very simple application taken from this example. I have added very basic tests to it to illustrate how it is done. With upcoming posts I’ll explain in details the idea and implementation.

Bellow is list of posts included in the series:

  1. Introduction (current post)
  2. WPF automation – projects structure
  3. WPF automation – locating and structure of WPF elements
  4. WPF automation – locating and structure of WinForms elements
  5. WPF automation – using the elements
  6. WPF automation – running the tests
Read more...

Complete guide how to use design patterns in automation

Last Updated on by

Post summary: Complete code example of how design patterns can be used in real life test automation.

With series of posts I’ve described 5 design patterns that each automation test engineer should know and use. I’ve started with brief description of the patterns. Then I’ve explained in details with code snippets following patterns: Page objects, Facade, Factory, Singleton, Null object. Code examples are located in GitHub for C# and Java.

Overview

This post is intended to bond all together in a complete guide how to do write better automation code. Generally automation project consists of two parts. Automation framework project and tests project. Current guide is intended to describe how to build your automation testing framework. How to structure your tests is different topic. Remember once having correctly designed framework then tests will be much more clean, maintainable and easy to write. To keep post shorter some of the code that is not essential for representing the idea is removed. Whole code is in GitHub.

Page objects

Everything start by defining proper page objects. There is no fixed recipe for this. It all depends on the structure of application under test. General rule is that repeating elements (header, footer, menu, widget, etc) are extracted as separate objects. The whole idea is to have one element defined in only one place (stay DRY)! Bellow is our HomePage object. What you can do generally is make search and clear search terms. Note that clearing is done with jQuery code. This is because of a bug I’ve described with workaround in Selenium WebDriver cannot click UTF-8 icons post.

using OpenQA.Selenium;

namespace AutomationRhapsody.DesignPatterns
{
	class HomePageObject
	{
		private WebDriverFacade webDriver;
		public HomePageObject(WebDriverFacade webDriver)
		{
			this.webDriver = webDriver;
		}

		private IWebElement SearchField
		{
			get { return webDriver.FindElement(By.Id(&quot;search&quot;)); }
		}

		public void SearchFor(string text)
		{
			SearchField.SendKeys(text);
		}

		public void ClearSearch()
		{
			webDriver.ExecuteJavaScript(&quot;$('span.cancel').click()&quot;);
		}
	}
}

WebDriver factory

WebDriver factory will be responsible for instantiating the WebDriver based on condition which browser we want to run our tests with.

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium.IE;

namespace AutomationRhapsody.DesignPatterns
{
	public class WebDriverFactory
	{
		public IWebDriver CreateInstance(Browsers browser)
		{
			if (Browsers.Chrome == browser)
			{
				return new ChromeDriver();
			}
			else if (Browsers.IE == browser)
			{
				return new InternetExplorerDriver();
			}
			else
			{
				return new FirefoxDriver();
			}
		}
	}
}

Constructor take an argument browser type. Browser type is defined as enumeration. This is very important. Avoid passing back and forth strings. Always stick to enums or special purpose classes. This will save you time investigating bugs in your automation.

public enum Browsers
{
	Chrome, IE, Firefox
}

NullWebElement

This is null object pattern and implements IWebElement. There is a NULL property that is used to compare is given element is not found or no.

using OpenQA.Selenium;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Drawing;

namespace AutomationRhapsody.DesignPatterns
{
	public class NullWebElement : IWebElement
	{
		private const string nullWebElement = &quot;NullWebElement&quot;;
		public bool Displayed { get { return false; } }
		public bool Enabled { get { return false; } }
		public Point Location { get { return new Point(0, 0); } }
		public bool Selected { get { return false; } }
		public Size Size { get { return new Size(0, 0); } }
		public string TagName { get { return nullWebElement; } }
		public string Text { get { return nullWebElement; } }
		public void Clear() { }
		public void Click() { }
		public string GetAttribute(string attributeName) { return nullWebElement; }
		public string GetCssValue(string propertyName) { return nullWebElement; }
		public void SendKeys(string text) { }
		public void Submit() { }
		public IWebElement FindElement(By by) { return this; }
		public ReadOnlyCollection&lt;IWebElement&gt; FindElements(By by)
		{
			return new ReadOnlyCollection&lt;IWebElement&gt;(new List&lt;IWebElement&gt;());
		}

		private NullWebElement() { }

		private static NullWebElement instance;
		public static NullWebElement NULL
		{
			get
			{
				if (instance == null)
				{
					instance = new NullWebElement();
				}
				return instance;
			}
		}
	}
}

WebDriver facade

WebDriver facade main responsibility is to define custom behaviour on elements location. This gives you centralised control over elements location. Constructor takes browser type and uses the factory to create WebDriver instance which is used internally in the facade. FindElement method defines explicit wait. If element is not found then NullWebElement which is actually implementation of Null object pattern. The idea is to safely locate elements with try/catch and then just use them skipping checks for null.

using OpenQA.Selenium;
using OpenQA.Selenium.Support.UI;
using System;
using System.Collections.ObjectModel;

namespace AutomationRhapsody.DesignPatterns
{
	public class WebDriverFacade
	{
		private IWebDriver webDriver = null;
		private TimeSpan waitForElement = TimeSpan.FromSeconds(5);

		public WebDriverFacade(Browsers browser)
		{
			WebDriverFactory factory = new WebDriverFactory();
			webDriver = factory.CreateInstance(browser);
		}

		public void Start(string url)
		{
			webDriver.Url = url;
			webDriver.Navigate();
		}

		public void Stop()
		{
			webDriver.Quit();
		}

		public object ExecuteJavaScript(string script)
		{
			return ((IJavaScriptExecutor)webDriver).
				ExecuteScript(&quot;return &quot; + script);
		}

		public IWebElement FindElement(By by)
		{
			try
			{
				WebDriverWait wait = new WebDriverWait(webDriver, waitForElement);
				return wait.Until(ExpectedConditions.ElementIsVisible(by));
			}
			catch
			{
				return NullWebElement.NULL;
			}
		}
	}
}

Tests

As I mention initially this post is about using efficiently design patterns in your framework automation project. Tests design are not being discussed here. Once you have designed the framework one simple test (without asserts) that makes search will look like code bellow.

Browsers browser = Browsers.Chrome;
WebDriverFacade webDriver = new WebDriverFacade(browser);
webDriver.Start(&quot;http://automationrhapsody.com/examples/utf8icons.html&quot;);

HomePageObject homePage = new HomePageObject(webDriver);
homePage.ClearSearch();
homePage.SearchFor(&quot;automation&quot;);

webDriver.Stop();

Conclusion

Design patterns are enabling you to write maintainable code. They increase value of your code. As shown in this series of posts they are not so complicated and you can easily adopt them in your automation. Most important is design patterns increase your value as a professional. So make the time to learn them!

Read more...

Singleton and Null object patterns

Last Updated on by

Post summary: Description with code samples of Singleton and Null object design patterns.

Singleton

Singleton pattern is used in a case where you want to restrict the instantiation of a class to only one object.

More details

The class should not require parameters for its construction. If creation of a class requires significant resources then singleton is a suitable solution. It is good that singletons are created when they are first needed, so called lazy initialisation.

Null object pattern

The idea is to make code more simpler and safe by skipping checks for null reference of a given object:

if (someVariable != null) {
	// Do something with someVariable
}

References

This post is part of design patterns guide series. Code examples are located in GitHub for C# and Java.

More details

Null object implements given interface and its methods are doing nothing. This makes the null object more predictable. You can safely invoke methods on the null object without the threat of a NullRefferenceException to break you application. This pattern is very well combined with singleton pattern where null object is actually a singleton. In this case you can check for reference or equality.

Example

Most common usage of Singleton pattern is to limit WebDriver to only one object instance in the whole automation test project.

private static IWebDriver webDriver;
public static IWebDriver WebDriver
{
	get
	{
		if (webDriver == null)
		{
			webDriver = new FirefoxDriver();
		}
		return webDriver;
	}
}

I’m going to give slightly different usage example where Singleton is used along with Null object pattern. NullWebElement implements IWebElement interface so it must implement all methods and properties defined by the interface. This is done on lines 4 to 21. Properties are returning some values, but not null! Methods are doing nothing. From line 23 to 36 is the Singleton definition. If Singleton is object you have defined then it should have private constructor so no one is able to instantiate it. There is a private field which actually hold the reference to the singleton. NULL is a property which instantiates the singleton if not already instantiated and returns it.

public class NullWebElement : IWebElement
{
	private const string nullWebElement = "NullWebElement";
	public bool Displayed { get { return false; } }
	public bool Enabled { get { return false; } }
	public Point Location { get { return new Point(0, 0); } }
	public bool Selected { get { return false; } }
	public Size Size { get { return new Size(0, 0); } }
	public string TagName { get { return nullWebElement; } }
	public string Text { get { return nullWebElement; } }
	public void Clear() { }
	public void Click() { }
	public string GetAttribute(string attributeName) { return nullWebElement; }
	public string GetCssValue(string propertyName) { return nullWebElement; }
	public void SendKeys(string text) { }
	public void Submit() { }
	public IWebElement FindElement(By by) { return this; }
	public ReadOnlyCollection<IWebElement> FindElements(By by)
	{
		return new ReadOnlyCollection<IWebElement>(new List<IWebElement>());
	}

	private NullWebElement() { }

	private static NullWebElement instance;
	public static NullWebElement NULL
	{
		get
		{
			if (instance == null)
			{
				instance = new NullWebElement();
			}
			return instance;
		}
	}
}

There are two main benefits of code above. First benefit comes from the Null object pattern. You can locate an element which doesn’t exists and call some of its methods without your tests to crash. You might say that comparing for null is not a a big overhead, but in big having a null check before each action is a waste of time.

IWebElement element = null;
try
{
	element = webDriver.FindElement(By.CssSelector("notExisting"));
}
catch
{
	element = NullWebElement.NULL;
}
element.Click();

It might be a discussion whether it is better to silent this failure error with null object or leave to massive crash. My opinion is you must fail the test and if you put some logging into Click method of the NullWebElement then you can easily trace the issue. The real benefit is you will have only one failed test instead of whole bunch failed because of the crash.

Second benefit comes from the Singleton pattern. You can easily compare some element against the NullWebElement.NULL. Most likely you will locate the element and use it safely because of null object pattern, but there might a case where you want to see if element is actually not found.

if (element == NullWebElement.NULL)
{
	Console.WriteLine("Element not found!");
}

Conclusion

Singleton is a pattern that you definitely should know. Null object with combination of Singleton can decrease the amount of code you write by skipping the checks for null. I would say if adopted those can simplify your code which is a benefit in a big projects.

Read more...

Factory design pattern

Last Updated on by

Post summary: Description with code samples of Factory design pattern.

Factory pattern is used to create objects based on specific rules.

More details

You may have several classes implementing one and the same interface. You might not know what object is suitable to get instantiated at given point in your code. Object creation may depend on different if/else or switch statements. You definitely do not want to make those is/else check every time you need and object. This is why you handle the object creation task to the factory which knows exactly what and how to create an object. The factory always returns a newly created object or re-initialised one.

References

This post is part of design patterns guide series. Code examples are located in GitHub for C# and Java.

Example

In example I’ve given here I’ve implemented a factory that creates IWebDriver object based on specific input. Object creation logic is encapsulated from outer world. This factory is very simple but is just fine for our purposes. More complex factories are defined by Factory method pattern and Abstract factory pattern but they are suitable for more complex applications. Factory in the example have only CreateInstance method. In it WebDriver is instantiated based on given from outside string with its name. It doesn’t store or re-use WebDriver object but always creates a new one.

public class WebDriverFactory
{
	public IWebDriver CreateInstance(string browser)
	{
		if ("Chrome".ToLower() == browser.ToLower())
		{
			return new ChromeDriver();
		}
		else if ("InternetExplorer".ToLower() == browser.ToLower())
		{
			return new InternetExplorerDriver();
		}
		else
		{
			return new FirefoxDriver();
		}
	}
}

Factory is used to instantiate WebDriver in tests. Condition which browser to instantiate may come from external parameter.

WebDriverFactory factory = new WebDriverFactory();
IWebDriver webDriver = factory.CreateInstance("Chrome");
webDriver.Url = "http://automationrhapsody.com";
webDriver.Navigate();

IWebElement logo = webDriver.FindElement(By.CssSelector("div#header-inner h1 a"));
logo.Click();

webDriver.Quit();

Conclusion

Factory is good to be used in automation testing as you have centralised control over WebDriver creation. This pattern will definitely keep you DRY. With the time being you may add more conditions on object creation (logging, different ports, etc.).

Read more...

Facade design pattern

Last Updated on by

Post summary: Description with code samples of Facade design pattern.

Facade design pattern provides simple and easy to use interface to a larger and more complex code, API or set of APIs.

More details

If you have to deal with complex or poorly designed API you might find yourself confused by all the functionality available in it. Most likely you don’t need everything provided by the API. So for ease and better maintainability only API features that are needed gets exposed outside the facade. In this way you simplify the API usage saving time for your colleagues when maintaining this code. Also you have control over how this external API is used and can prevent misunderstanding or misuse of it. If more functionality from API is needed this will require update of the facade or adding new facade which exposes it.

References

This post is part of design patterns guide series. Code examples are located in GitHub for C# and Java.

Example

It might not be a good idea to have the pattern in the name of the facade, but in the example I’ve put it to ease. In this example WebDriver instantiation is hard coded to Firefox, but in real life it will be instantiated based on some rules. Facade exposes Start, Stop and FindElement methods. All other WebDriver functionality is not accessible.

public class WebDriverFacade
{
	private IWebDriver webDriver = null;
	private TimeSpan waitForElement = TimeSpan.FromSeconds(5);

	public WebDriverFacade()
	{
		webDriver = new FirefoxDriver();
	}

	public void Start(string url)
	{
		webDriver.Url = url;
		webDriver.Navigate();
	}

	public void Stop()
	{
		webDriver.Quit();
	}

	public IWebElement FindElement(By by)
	{
		try
		{
			WebDriverWait wait = new WebDriverWait(webDriver, waitForElement);
			return wait.Until(ExpectedConditions.ElementIsVisible(by));
		}
		catch
		{
			return null;
		}
	}
}

Facade is used exactly the same way as WebDriver itself:

WebDriverFacade webDriver = new WebDriverFacade();
webDriver.Start("http://automationrhapsody.com");

IWebElement logo = webDriver.FindElement(By.CssSelector("div#header-inner h1 a"));
logo.Click();

webDriver.Stop();

Conclusion

WebDriver API is neither complex nor poorly designed, so maybe making a facade is not a mandatory thing. Still I think having control over its usage is a good approach. In given example you can see I use explicit wait so I locate exactly the same way all elements in my automation project.

Read more...

Page objects design pattern

Last Updated on by

Post summary: Description with code samples of Page Objects design pattern.

This is the most important pattern related to software automation. It enables you to create object repository with UI elements. Elements are separated from test logic. This pattern makes code much more maintainable and reusable.

References

This post is part of design patterns guide series. Code examples are located in GitHub for C# and Java.

Never do this

Without this pattern what you might do is start WebDriver, navigate to some test page. Locate element you need then click it.

IWebElement element = webDriver.FindElement(By.CssSelector(&quot;div.find label&quot;));
element.Click();

This is good when you want to test the idea or do some quick demo. In a real life project this element might be needed in dozens of tests. What happens if the UI changes and CSS selector is not matching any more? Here comes the maintainability problem, you have to search and replace all of them.

The proper way

In software engineering there is a principle called DRY for short. Its idea is to have each element or action stored only once in a system. This avoids copy/paste and reduces the overhead for code maintenance. The same idea is used in Page Object pattern.

  • Each page or re-usable part of a page (i.e. header, footer, menu) is separate class.
  • Class constructor takes WebDriver as argument and uses it internally to locate elements.
  • Each element is a private property (or getter in Java).
  • Actions are public and internally operate with elements.

In code bellow SearchField is private property used only by SearchFor method which is exposed as available action on HomePage. Element can be located inside the action method but suggested approach gives better readability. And if element is needed more than once then defining it separately is a must.

public class HomePage
{
	private IWebDriver webDriver;

	public HomePageObject(IWebDriver webDriver)
	{
		this.webDriver = webDriver;
	}

	private IWebElement SearchField
	{
		get { return webDriver.FindElement(By.Id(&quot;search&quot;)); }
	}

	public void SearchFor(string text)
	{
		SearchField.SendKeys(text);
	}
}

Page object is instantiated in the test and actions are invoked.

HomePageObject homePage = new HomePageObject(webDriver);
homePage.SearchFor(&quot;automation&quot;);

With this approach you have one element defined on only one place. Only actions are exposed out of the page object. It is very clear what actions can be done on this page.

Original page objects

To be honest what I have described in this post is not actually the original Page Objects pattern. Although the idea is same I have modified it slightly. Originally each action returns a page object. In this way you can chain methods in one test and very easy to understand if a test will be broken when given action now returns different page object. I do not like chaining of methods though so I don’t need this extra. It doesn’t bother you at all to do

public HomePage SearchFor(string text)
{
	SearchField.SendKeys(text);
	return this;
}

Also I have separated elements location outside of actions because I want better maintainability and readability of the code.

Conclusion

Always use page objects in you test automation. ALWAYS!

Read more...

Design patterns every test automation engineer should know

Last Updated on by

Post summary: Description with examples of 5 design patterns that is good to know and use in our automation testing.

Design patterns are interesting topic. They are created to solve common problems in software design. Although design patterns are not reserved only for software development they seem not to be widely discussed in software automation. Maybe this is because the topic sounds complicated. Yes, there are really sophisticated design patterns used to solve complex issues in software development. But also there are easy to understand and adopt design patterns that can significantly improve readability and maintainability of our test automation code.

With series of posts I’m going to give overview with examples on some design patterns that can be very useful in our test automation projects. To keep blog neat in this first post I’ll just outline the list of patterns. Each one will be discusses in separate post with code samples.

Page Objects pattern

This is the most important pattern related to software automation. It enables you to create object repository with UI elements. Elements are separated from tests logic. This pattern makes code much more maintainable and reusable. Click for more details and code samples.

Facade pattern

The whole idea of facade design pattern is to provide simple and easy to use interface to a larger and more complex code, API or set of APIs. Most likely you don’t need everything provided by the API. So for ease and better maintainability only API features that are needed gets exposed outside the facade. In this way you simplify the API usage and you have control over how this external API is used and can prevent misunderstanding or misuse. Click for more details and code samples.

Factory pattern

Factory pattern is used to create objects based on specific rules. You may have several classes implementing an interface. In your code you do not want to bother defining which concrete class to instantiate or you might not know what object is suitable to get instantiated. This is why you handle the object creation task to the factory which knows exactly what object to create. With factory pattern object creation is encapsulated. Click for more details and code samples.

Singleton pattern

Singleton pattern is needed when you need exactly one object from a specific class in the whole application. Click for more details and code samples.

Null object pattern

The idea is to make code more simpler and safe by skipping null reference checks. Null object implements given interface and its methods are doing nothing. This makes the null object predictable. You can safely invoke methods on the null object without the threat of a NullReferenceException to break your application. Click for more details and code samples.

Putting all together

Once you have gone through patterns one by one now it is time to put everything in one project. Complete guide how to use design patterns in automation post is showing how.

Read more...