Monthly Archives: February 2015

NTestsRunner for functional automated tests

Last Updated on by

Post summary: NTestsRunner implementation details and features.

In the previous post I’ve described unit testing frameworks and why they are not suitable for running functional automated tests. I introduced NTestsRunner – very simple runner that can be used for running your automation tests. This topic is dedicated to implementation details of the NTestsRunner.

Verifications

It is important in functional testing to be able to place several verification points in one test. For this purpose, abstract class Verification is implemented. It has two properties to store more details about verification and time it was taken. Constructor receives comma separated string values. In case of zero strings are passed then the result is an empty string. If one string is passed then this is the result. If more than one string is added then first string is taken as formatting string and others are used to build up the result. Logic is similar to string.Format(String, Object[]) method.

public abstract class Verification
{
	public string Result { get; private set; }
	public DateTime ExecutedAt { get; private set; }

	public Verification(params object[] args)
	{
		...
	}
}

Passed or Failed

In automation test may have two conditions – passed or failed. This is why two concrete classes are extending Verification: VerificationPassed and VerificationFailed. They do not add any other functionality. Those classes use parent’s class constructor. This is an example how to instantiate an object from those classes:

string value = "number";
int number = 1;
Verification result =
	new VerificationFailed("This is formatting string {0} {1}. ",
		value,
		number);

Test case result

Test case is generally a set of conditions to verify whether given scenario works are per user requirements. In automation, world test case is test method with several verification points inside. In NTestsRunner TestCaseResult is class representing the idea of a test case. It has properties for name, time to run and list of all verifications with a count of passed and failed.

public class TestCaseResult
{
	public List<Verification> Verifications = new List<Verification>();
	public string Name { get; set; }
	public int VerificationsFailed { get; set; }
	public int VerificationsPassed { get; set; }
	public TimeSpan Time { get; set; }
}

Test plan result

TestPlanResult in NTestsRunner has nothing to do with test plan term from QA world. Here this is a representation of a test class with test methods inside. It has properties for name and time to run. Also, there is a list of all TestCaseResults, i.e. test methods in that class. There are counters for passed and failed test cases and also counters for all passed and failed verifications inside all TestCaseResults.

public class TestPlanResult
{
	public List<TestCaseResult> TestCases = new List<TestCaseResult>();
	public string Name { get; set; }
	public int TestCasesPassed { get; private set; }
	public int TestCasesFailed { get; private set; }
	public int VerificationsPassed { get; private set; }
	public int VerificationsFailed { get; private set; }
	public TimeSpan Time { get; private set; }

	public void Count()
	{
		...
	}
}

Class and method attributes

In order to make one class a test class, it should have with [TestClass] attributes. To convert method to a test one it should have [TestMethod] attribute. Just the attribute is not enough though. The method should have special method signature. This is required by NTestsRunner.

Test method signature

In order to run without exception test method needs to conform to two rules:

  1. To have attribute [TestMethod]
  2. Method to receive parameter List verifications in its signature, i.e.
    [TestMethod]
    public void TestMethod1(List<Verification> verifications)
    

Configurations

Configurations can be found on NTestRunner home page.

Execution

Once object from NTestsRunner is instantiated and configured tests with Execute() method. Inside this method, all classes from calling assembly (the one that holds the tests) are taken. If TestsToExecute is configured then only those with name matching is given values are taken. If no TestsToExecute is provided then all classes with the attribute [TestClass] are taken. Methods from each class are taken by default in order of appearance in the class. If the method has [TestClass] attribute then the method is executed by passing List object to it. Inside the method, Verifications are collected as a list into a TestCaseResult object. After the method is run TestCaseResult is added to its parent TestPlanResult which is added to list with all results. In the end, results are saved as XML and HTML.

Results in JUnit XML

In order to integrate with CI tools such as Jenkins or Bamboo results are exported to XML file after execution has finished. The file is named Results.xml and is located in test results folder. XML format is implemented according to junit-4.xsd.

Results in HTML

Tests result are saved as HTML report for better readability. The file is named Results.html and is located in test results folder.

Usage

In order to use NTestsRunner a console application project is needed. This project will hold test classes. As the one below. Take into consideration that this is very simplified usage pattern. In reality, Page objects design pattern will be used. Page objects will make the verifications and return them.

[TestClass]
public class TestClass1
{
	[TestMethod]
	public void TestMethod1(List<Verification> verifications)
	{
		// Do some actions
		verifications.Add(new VerificationFailed("There is error"));
		// Do some actions
		verifications.Add(new VerificationPassed("Everythign is OK"));
	}
}

In its main method, a new instance of NTestsRunner is created. Configurations are done and test executions are started. It is that simple to use it.

class Program
{
	static void Main(string[] args)
	{
		NTestsRunnerSettings settings = new NTestsRunnerSettings();
		settings.TestResultsDir = @"C:\temp";
		settings.MaxTestCaseRuntimeMinutes = 2;
		settings.TestsToExecute.Add("TestClass1");
		settings.PreventScreenLock = true;

		NTestsRunner runner = new NTestsRunner(settings);
		runner.Execute();
	}
}

Pros and cons

NTestsRunner has its pros and cons.
Pros are:

  • Pretty easy to use
  • Open source and can be customized to your specific needs
  • Gives you ability to make several verifications in one test and in case of failure it doesn’t break current test method
  • Tests are stored in console application that can be easily run
  • Results are saved in JUnit XML for CI integration
  • Results are saved in HTML

Cons are:

  • Test methods should have a specific signature
  • It is not easy to migrate existing tests to a new format

Conclusions

This is a pretty good tool for running functional automated tests. It is very easy to use and is made especially for running functional automated tests. You can definitely give it a try.

Related Posts

Read more...

Running functional automation tests

Last Updated on by

.Post summary: Unit testing frameworks are not very suitable for running functional tests. NTestsRunner is an alternative way of running functional automated tests.

Unit testing

Unit testing is focused on testing of code on a low level. Methods, sets of methods or modules are being tested by writing test code which invokes those methods with specific arguments. In unit testing, all external dependencies (database, file system, network, etc.) are removed. Those resources are simulated in unit tests by using so-called mock objects. Mock objects are controlled by tests designer and have predictive behavior. Running a piece of code which doesn’t have external dependencies happens almost immediately. So unit tests are executed for a very low amount of time. It is considered a set of unit tests taking longer than 5 minutes is not well designed. Unit tests are strictly focused. One test tests only one condition. Each test is not related in anyhow to other tests.

Unit testing frameworks

Unit testing frameworks conform to unit tests purpose and design. Tests should not depend on each other. For this reason, unit testing frameworks execute tests in random order (xUnit.net or MS Unit Testing Framework), other like NUnit in alphabetic order of tests method names. Checking for given conditions are done with assertions. If one assert fails current test execution is stopped and the test is marked as failed.

Functional testing

Functional testing is focused on ensuring that software product works as per user requirements. Real life software has external dependencies. Most software products have some kind of user interface. Automation tests are focused on verifying that UI works correctly. Transferring and rendering data to UI takes time, database operations take time, file and network operations also take time, etc. In general functional tests are more complex and take much longer to execute. In order to be efficient checking of several conditions are defined in each test. Functional tests can be manual and automated. In current post when I mention functional tests I mean only automated.

A requirement for running functional test – many verifications

In a perfect situation, a functional one test should verify only one condition. In reality, because of too many external dependencies, time for tests execution is large. A time matters. In order to shorten this time we make several checks in one test. For e.g. in an e-commerce website, an order is placed. We verify order confirmation page that there is order number, that address is same used during checkout, that user’s email is correct. We also can verify inside user’s email box that received mail is correct. We can check in a database for some properties of the order. If we have to do one order for each check tests will take significantly longer time. To be efficient we do all checks with one order in one test case. We need a framework which allows you to have multiple verifications in one test. Furthermore, if verification is failed test execution should continue.

A requirement for running functional test – controlled sequence

What is really good to be avoided but sometimes cannot is test dependency. Sometimes one test needs another to have done something before continuing. As I said this should be avoided but in order to be efficient you should make a trade-off between good tests design and time to execute. For e.g. you may want to cancel and then refund order placed in an e-commerce website. Generally, it is best to place a new order for this test but if the placing of order requires too much time then an option is to reuse already existing order from the previous test. We need to be able to control test case execution order.

NTestsRunner

In order to have control over tests and use many verifications, I’ve created NTestsRunner. Its code is in GitHub NTestsRunner repository. This is a .NET library. You create a console application with your tests and use the library within. Tests are annotated is similar fashion as with unit testing frameworks. Tests are executed in sequential order. There could be many verifications. Results are saved as HTML and XML in JUnit format. NTestsRunner is described in more details in NTestsRunner for functional automated tests post.

Related Posts

Read more...

WPF automation – running the tests

Last Updated on by

Post summary: How to sell your automation to management. Guide for running the tests unattended.

References

This post is part of Automation of WPF applications with Telerik Testing Framework and TestStack White series. The sample application can be found in GitHub SampleApp repository.

Test frameworks need mouse and keyboard

As you have noticed when running the examples it is not possible to do anything else while tests are running because both Telerik Testing Framework and TestStack White are using mouse and keyboard in order to click and type text. This is how both frameworks are designed and work. It doesn’t seem very effective if you are wasting your time watching tests running on your workstation instead of doing something productive. This is not a good argument when advocating your automation.

The essence of automation is being effective

This topic is not about Automation vs. Manual testing so I will not go in that direction. I’ll just say that still there are companies lacking management willpower to support and embrace automation. So we need to be good salesmen!

How to drive your automation to success

As I said we need to be wise when promoting our automation to get more time and resources. Sales strategy is pretty easy and straightforward not requiring huge investments:

  • Free to use and pretty easy to work with frameworks.
  • Some enthusiasm to make the first automation. Remember, be smart and first automate most repeating scenarios. This will show real results and help you buy some time for further automation.
  • Virtual (or real) machine with test running on it during the night
  • Mail with results is sent to management with results in the morning

Run the tests unattended

You get the machine, set up the framework, set up scheduled tasks for deploying the latest application under test and latest test code. Run the tests and get into trouble. Tests do not run! This is because there are special requirements to schedule unattended run. There must be an active Windows session in order mouse and keyboard to be used. Once the session is interrupted tests stop. This is detailed KB article on the topic with several possible solutions.

Working solution

You can try solutions in the article to see which works best for you. For me, the solution is to have a remote desktop in remote desktop. This requires Windows Server installation. The only server provides two simultaneous remote desktop sessions. There are unofficial patches for non-server versions which I haven’t tried and cannot comment. Two local users are needed on the Windows Server with no desktop locking or screen saver (domain users most likely will have desktop locked after a time). Log from your machine to test machine with the first user. From first user’s session log to same testing machine with the second user. Tests are started from second user’s session. Once tests are started you can freely close desktop (not log out!). You can create a scheduled task which runs only when the user is logged in and just wait for results in the morning. If two accounts are overhead there is an option to use software that prevents your computer to lock. See prevent screen lock thread if something works for you.

Conclusions

Automation is an exciting field of career development for test engineers. This blog is dedicated to automation testing. You will find very useful and interesting topics in it. I would definitely encourage you just to give it a try. Good luck!

Related Posts

Read more...

WPF automation – using the elements

Last Updated on by

Post summary: Use already created Page Objects and build up test framework.

References

This post is part of Automation of WPF applications with Telerik Testing Framework and TestStack White series. The sample application can be found in GitHub SampleApp repository.

Page Objects holder

Below is App.cs which is a representation of application under test.

using ArtOfTest.WebAii.Wpf;
using White.Core;
using White.Core.UIItems.WindowItems;

namespace SampleApp.Tests.Framework.Elements
{
	public class App
	{
		public WpfApplication ApplicationWebAii { get; private set; }
		public Application ApplicationWhite { get; private set; }

		public App(WpfApplication webAiiApp, Application whiteApp)
		{
			ApplicationWebAii = webAiiApp;
			ApplicationWhite = whiteApp;
		}

		public MainWindow MainWindow
		{
			get
			{
				return new MainWindow(ApplicationWebAii
					.WaitForWindow(MainWindow.WINDOW_NAME).Find);
			}
		}

		public OpenFile OpenFile
		{
			get
			{
				return new OpenFile(GetWindowByName("Open"));
			}
		}

		public MessageBox MessageBox
		{
			get
			{
				return new MessageBox(GetWindowByName(""));
			}
		}

		private Window GetWindowByName(string windowName)
		{
			// Workaround as method GetWindow(string title) is not working
			foreach (Window window in ApplicationWhite.GetWindows())
			{
				if (windowName.Equals(window.Name))
				{
					return window;
				}
			}
			return null;
		}
	}
}

The constructor takes an instance of Telerik Testing Framework’s application (WpfApplication) and TestStack White’s application (Application). Those are stored in the App instance.

Access the Page Objects

Each window in a real application is represented by a property in the App class. When accessed a new object of this page object class is created and its elements can be accessed.

WPF page objects require VisualFind in order to be instantiated. It is obtained by first locating the window with Telerik’s

public WpfWindow WaitForWindow(string caption);

From the located window we need only the VisualFind which is used internally to locate elements on that particular window.

WinForms page objects require White’s Window instance in order to be instantiated. The window is located by

public virtual Window GetWindow(string title);

I found this method not always working so I’ve made a workaround method

private Window GetWindowByName(string windowName);

New Page Objects vs. Cached Page Objects

In the example above every time, an action is required a new page object is instantiated. In some cases instantiating the object may require longer time or you might need some properties in this object preserved during tests. In such cases, you may use Singleton design pattern and instantiate only one object.

private MessageBox messageBox = null;
public MessageBox MessageBox
{
	get
	{
		if (messageBox == null)
		{
			messageBox = new MessageBox(GetWindowByName(""));
		}
		return messageBox;
	}
}

Both approaches have pros and cons. In case of new page object, you always work with a fresh instance without any previous state saved. This might require more time to instantiate the objects and you are not able to save previous states. Cached objects may be much faster as a performance but having internal state may lead to unexpected bugs in your automation.

Base test

Finally to make all work we need an instance of App. The instance is created in BaseTest.cs class.

using ArtOfTest.WebAii.Core;
using SampleApp.Tests.Framework.Elements;
using White.Core;

namespace SampleApp.Tests.Framework.Tests
{
	public class BaseTest
	{
		protected App App { get; set; }
		private string applicationPath =
			"C:\\SampleApp\\SampleApp\\bin\\Debug\\SampleApp.exe";

		protected void Start()
		{
			if (App == null)
			{
				Application appWhite = Application.Launch(applicationPath);
				Manager manager = new Manager(false);
				manager.Start();
				App = new App(
					manager.ConnectToApplication(appWhite.Process), appWhite);
			}
		}

		protected void Stop()
		{
			if (App != null && App.ApplicationWhite != null)
			{
				App.ApplicationWhite.Kill();
			}
			App = null;
		}
	}
}

All tests inherit from the base test class. Initialise and clean up code is added in the base test. In our case Start() method is the initializer. It must be called in order to instantiate App class. App property is protected so every extending class has access to it.

Initialise the frameworks

In order to start the application under test, we need the full path to the exe file. In this example, this is hardcoded but in real life, it will be configurable. Start the application with White:

public static Application Launch(string executable);

Once started then connect to it with Telerik framework by creating a Manager and use its

public WpfApplication ConnectToApplication(Process proc, string pid = null);

The process is obtained out of White Application.Process property. Opposite launch order is not working. White is not able to Attach to running process.

Use page objects

Once Start() method is called, the application under test is started and both frameworks are connected to it you can simply do in your test:

App.MainWindow.ClickBrowseButton();

This will find and create a new instance of MainWindow and then it will find and click Browse button. Your framework defines the actions on elements which are later used in actual tests. Once all the work on framework has been done it is that simple to build your tests.

Clean up

Stop() method is called at the end of the test in order to close the application under test by killing the underlying process.

The tests

This is unit test created with MS Unit Testing Framework in order to demonstrate real testing on the application.

using Microsoft.VisualStudio.TestTools.UnitTesting;
using SampleApp.Tests.Framework.Tests;

namespace SampleApp.Tests
{
    [TestClass]
    public class UnitTest1 : BaseTest
    {
		[TestInitialize]
		public void Initialise()
		{
			Start();
		}

		[TestMethod]
		public void OpenFile_OnCancel_GivesMessage()
		{
			App.MainWindow.ClickBrowseButton();
			App.OpenFile.ClickCancelButton();
			Assert.AreEqual("Problem occured, try again later",
				App.MessageBox.GetText());
			App.MessageBox.ClickOkButton();
		}

		[TestMethod]
		public void OpenFile_OnAttachFile_GivesMessageAndFileIsShown()
		{
			string filePath = @"C:\SampleApp\SampleApp\bin\Debug\HappyFace.jpg";
			App.MainWindow.ClickBrowseButton();
			App.OpenFile.EnterFileName(filePath);
			App.OpenFile.ClickOpenButton();
			Assert.AreEqual("Successfully done", App.MessageBox.GetText());
			App.MessageBox.ClickOkButton();
			Assert.AreEqual(filePath, App.MainWindow.GetFilePathAtIndex(1));
		}

		[TestCleanup]
		public void CleanUp()
		{
			Stop();
		}
    }
}

Unit testing frameworks

Unit testing frameworks are designed to run tests in random order. Before each test method annotated with [TestInitialize] is run. In our case application is started. After each test method annotated with [TestCleanup] is run. In our case application is stopped. For this simple application running tests with a unit testing framework is OK. We are not doing unit tests but functional once. So for bigger and more complex tests unit testing frameworks are not very convenient. I’ve created very simple tests runner. This post describing the need of such tests runner.

This post shows how to build up the framework based on page objects. Next post is WPF automation – running the tests.

Related Posts

Read more...

WPF automation – locating and structure of WinForms elements

Last Updated on by

Post summary: Guide how to locate WinForms elements with TestStack White.

References

This post is part of Automation of WPF applications with Telerik Testing Framework and TestStack White series. The sample application can be found in GitHub SampleApp repository.

WinForms elements manipulation with TestStack White

MessageBox.cs class is a representation of a MsgBox of our sample application which is WinForms.

using System.Windows.Automation;
using White.Core.UIItems;
using White.Core.UIItems.Finders;
using White.Core.UIItems.WindowItems;

namespace SampleApp.Tests.Framework.Elements
{
	public class MessageBox
	{
		private Window win;
		public MessageBox(Window window)
		{
			win = window;
		}

		private Button Button_OK
		{
			get
			{
				return win.Get<Button>(
					SearchCriteria.ByControlType(ControlType.Button).
					AndByText("OK"));
			}
		}
		private Label Label_Text
		{
			get
			{
				return win.Get<Label>(
					SearchCriteria.ByControlType(ControlType.Text));
			}
		}

		public void ClickOkButton()
		{
			Button_OK.Click();
		}

		public string GetText()
		{
			return Label_Text.Text;
		}
	}
}

MessageBox is following Page Object design pattern. Elements are private properties. Actions are public methods acting on elements. Again: one element must be defined in only one place.

Constructor receives an instance of the current window object. Elements are located by White framework method:

public virtual T Get<T>(SearchCriteria searchCriteria) where T : UIItem;

SearchCriteria

SearchCriteria is used for locating elements. Search is possible with a combination of following criteria: Text, ControlType, AutomationId, NativeProperty, Framework. And here comes the hardness of working with White in complex UIs. There may be several TextBox-es (ByControlType == Edit) AND with no text inside (AndByText == “”). You can take them all and then access by index the one which is proper for your test. And what happens when you fill text in one of them. Next find has one less window. If you think it is easy then enjoy, I would rather not do it.

UISpy

Finding of the proper SearchCriteria is done with powerful tool UISpy.exe. Unfortunately, this one is deprecated from Windows 8. MS have added a replacement called Inspect.exe. I don’t find it useful this is why I have added UISpy.exe in the GitHub repository.

UISpy

Dialog is the message box window which is a child of MainWindow. It has empty title and two elements: OK button and message text. Current example shows the location of the text element. Its MS ControlType == ControlType.Text. This is what we need to locate it since there is only one such element in the message box. Once we have it we take the text and test compares current vs. expected text. OK button is located by ControlType and text. In case of different localisations we may leave just the ControlType, but if there are more buttons text is mandatory to distinguish the correct button.

Once you have defined your page objects it is time to assemble them. Next post is WPF automation – locating and structure of WPF elements.

Related Posts

Read more...

WPF automation – locating and structure of WPF elements

Last Updated on by

Post summary: Guide how to locate WPF elements with Telerik Testing Framework.

References

This post is part of Automation of WPF applications with Telerik Testing Framework and TestStack White series. The sample application can be found in GitHub SampleApp repository.

MainWindow Page Object

MainWindow.cs class is a representation of the main window of our sample application which is WPF. Note that code here is shortened. The full code can be found in GitHub repository.

using ArtOfTest.WebAii.Controls.Xaml.Wpf;
using ArtOfTest.WebAii.Silverlight;
using ArtOfTest.WebAii.TestTemplates;
using System.Threading;

namespace SampleApp.Tests.Framework.Elements
{
	public class MainWindow : XamlElementContainer
	{
		public static string WINDOW_NAME = "MainWindow";
		private string mainPath =
			"XamlPath=/Border[0]/AdornerDecorator[0]/ContentPresenter[0]/Grid[0]/";
		public MainWindow(VisualFind find) : base(find) { }

		private Button Button_Browse
		{
			get
			{
				return Get<Button>(mainPath + "Button[0]");
			}
		}

		public void ClickBrowseButton()
		{
			Button_Browse.User.Click();
			Thread.Sleep(500);
		}
	}
}

MainWindow is following Page Object design pattern. Elements are private Properties. Actions on elements are public methods that are the actual building blocks of the tests. Do not just add action methods for the sake of adding them. Add them if only the tests need such action.

If the element is used in only one action then it could be inside the action method, but if the element is used more than one it is mandatory to keep it as separate property. The main idea is to avoid duplications in order to improve maintainability. Remember to stay DRY: one element must be defined in only one place.

MainWindow class extends XamlElementContainer which comes from the framework. The constructor takes VisualFind object and passes it to XamlElementContainer‘s constructor. This allows us to use framework’s methods for location of elements. In our case

public TControl Get<TControl>(params string[] clauses)
	where TControl : ArtOfTest.WebAii.Controls.Xaml.IFrameworkElement;

This is the strategy used by the Test Studio generated tests and I prefer it.

A different approach for MainWindow

Another element structure could be not to extend XamlElementContainer but pass a VisualFind object through MainWindow’s constructor and use it internally to locate elements.

using ArtOfTest.WebAii.Controls.Xaml.Wpf;
using ArtOfTest.WebAii.Silverlight;
using System.Threading;

namespace SampleApp.Tests.Framework.Elements
{
	public class MainWindow
	{
		public static string WINDOW_NAME = "MainWindow";
		private string mainPath =
			"XamlPath=/Border[0]/AdornerDecorator[0]/ContentPresenter[0]/Grid[0]/";
		private VisualFind visualFind;
		public MainWindow(VisualFind find)
		{
			visualFind = find;
		}

		private Button Button_Browse
		{
			get
			{
				return visualFind.ByExpression(
					new XamlFindExpression(mainPath + "Button[0]")).
					CastAs<Button>();
			}
		}

		public void ClickBrowseButton()
		{
			Button_Browse.User.Click();
			Thread.Sleep(500);
		}
	}
}

How to locate elements is described in finding page elements article.

XamlPath

As you can see I’m using exact XamlPath find expression in order to locate my elements because I find it consistent and easy to maintain. XamlPath syntax is pretty similar to XPath. The difference is its indexes are zero-based (Button[0] is the first element) and it doesn’t provide predicates and axes. The hard part is to get the XamlPath. I have found this lovely tool WPF Inspector. It gives a lot of information about the structure of WPF application.

WPF Inspector

WPF Inspector

This is a screenshot of the XamlPath for Browse button. In more complex UIs even with this great tool, it is not that easy to get the XamlPath. If you find it hard you may consider other find expressions.

All elements on MainWindow are only WPF thus an instance of White is not needed. In case of WinForms content hosted inside WPF, you can pass an instance of White through elements’ constructor and use it in actions inside.

public MainWindow(VisualFind find, Application applicationWhite) : base(find)
{
	appWhite = applicationWhite;
}

Having an instance of White inside the class you can work with it as explained in WPF automation – locating and structure of WinForms elements post.

Related Posts

Read more...

WPF automation – projects structure

Last Updated on by

Post summary: How to structure projects in the course of WPF automation.

References

This post is part of Automation of WPF applications with Telerik Testing Framework and TestStack White series. The sample application can be found in GitHub SampleApp repository.

Overview

For better maintainability and good design, automation testing is separated into two projects. One is the so-called framework project. It has knowledge about third used party frameworks (Telerik Testing Framework, TestStack White, Selenium, Watij, etc.). It operates on elements via those third-party frameworks. Framework exposes actions that are building blocks of the tests. The other project is the tests. It doesn’t know anything about used automation frameworks. It uses methods exposed by an internal framework and builds the tests with them. The idea is that one team/person with development knowledge can be responsible for the internal framework, others can be responsible for writing well-designed tests using the internal framework only. Another benefit of two projects is the ability to completely change internal framework project without affecting test logic at all. Here is the structure of my sample projects:

Projects structure

Projects structure

SampleApp – This is the application under test. Generally, it is not in the test project, you receive it from developers or from nightly build or from some other place. For the purpose of this demo, I’ve created a very basic application that uploads an image and shows it inside. There is attached image (HappyFace.jpg) that is used in the tests.

SampleApp.Tests – Here is the most important artifact – the tests, the purpose of all this fuss. In the demo, this is made as UnitTest with MS Unit Testing Framework. You can use whatever run strategy you like – another unit testing framework (xUnit, nUnit, etc.) or you can build your own tests runner. I have built my own tests running because by definition unit tests should be independent of each other thus MS Unit Testing Framework runs them in random order. It might be possible to manage order, but this will require managing of some sort of external files. It gives too much overhead to me.

SampleApp.Tests.Framework – This is where the interesting part is. Here are the elements – main application form (WPF), message boxes shown on success or error (WinForms) and open file dialogue (WinForms). MainWindow is manipulated with Telerik Testing Framework, other two are manipulated with White. App.cs is a holder which represent application itself and elements are accessible through it. BaseTest holds an instance of App. It is extended by UnitTests and App and its elements are accessible.

Next post is WPF automation – locating and structure of WPF elements in the code.

Related Posts

Read more...

Automation of WPF applications

Last Updated on by

Post summary: Overview how to successfully automate WPF application with Telerik Testing Framework and TestStack White.

I’ve spent lots of time in automation of a WPF application. Nowadays the world is dealing with web application and desktop application are fading away. Nevertheless, desktop applications still have their own private space and I hope these series of posts will be useful to someone.

Preface

An application under test is created with WPF but has WinForms grids included on different windows presenting lots of information. Performance is the main reason to use WinForms grids inside a WPF application. WPF grid with 40 columns and 100 rows lags a lot. After research and proof of concept of commercial and open source tools and frameworks, two were chosen: Telerik Testing Framework and TestStack White. They provide flexibility and maintainability which is so vital in a fast-changing application. Both are used with C#. Every one has its own weaknesses and strengths but both made a perfect synergy to provide you the possibility to automate almost everything.

Telerik Testing Framework

Extremely powerful and free to use framework. You can purchase additional support which is at reasonable prices. Telerik also provides Test Studio, a tool and MS Visual Studio plug-in, build on top of the framework. This is also an option, but for me using a tool gives more restrictions than benefits. Their framework is the best available for testing WPF applications. It is very easy to use. Elements are easily located by XamlPath (very similar to XPath). Of course, there are other locate options. The framework provides very rich API with lots of operations over the elements. Still, its powerfulness is limited to WPF. It is not that good with WinForms elements. Telerik Testing Framework requires installation as it copies lots of assemblies in GAC.

TestStack White

Extremely powerful open source framework. Build as a wrapper of Microsoft UI Automation framework you can automate almost everything with it. But this comes at a price. It is hard to locate elements. I use it only where Telerik Testing Framework is not capable to do the work – WinForm grids, context menus, closing windows with small X in the top right corner?! Yes, those are WPF inside but are hosted in operation system window and Telerik has no access to it.

In GitHub, I have uploaded an MS Visual Studio 2013 project with very simple application taken from this example. I have added very basic tests to it to illustrate how it is done. With upcoming posts, I’ll explain in details the idea and implementation.

Below is a list of posts included in the series:

  1. Introduction (current post)
  2. WPF automation – projects structure
  3. WPF automation – locating and structure of WPF elements
  4. WPF automation – locating and structure of WinForms elements
  5. WPF automation – using the elements
  6. WPF automation – running the tests

Related Posts

Read more...

Complete guide how to use design patterns in automation

Last Updated on by

Post summary: Complete code example of how design patterns can be used in real life test automation.

With series of posts, I’ve described 5 design patterns that each automation test engineer should know and use. I’ve started with a brief description of the patterns. Then I’ve explained in details with code snippets following patterns: Page objects, Facade, Factory, Singleton, Null object. Code examples are located in GitHub for C# and Java.

Overview

This post is intended to bond all together in a complete guide how to do write better automation code. Generally, automation project consists of two parts. Automation framework project and tests project. Current guide is intended to describe how to build your automation testing framework. How to structure your tests is a different topic. Remember once having correctly designed framework then tests will be much more clean, maintainable and easy to write. To keep post shorter some of the code that is not essential for representing the idea is removed. The whole code is on GitHub.

Page objects

Everything starts by defining proper page objects. There is no fixed recipe for this. It all depends on the structure of application under test. The general rule is that repeating elements (header, footer, menu, widget, etc) are extracted as separate objects. The whole idea is to have one element defined in only one place (stay DRY)! Below is our HomePage object. What you can do generally is make search and clear search terms. Note that clearing is done with jQuery code. This is because of a bug I’ve described with a workaround in Selenium WebDriver cannot click UTF-8 icons post.

using OpenQA.Selenium;

namespace AutomationRhapsody.DesignPatterns
{
	class HomePageObject
	{
		private WebDriverFacade webDriver;
		public HomePageObject(WebDriverFacade webDriver)
		{
			this.webDriver = webDriver;
		}

		private IWebElement SearchField
		{
			get { return webDriver.FindElement(By.Id("search")); }
		}

		public void SearchFor(string text)
		{
			SearchField.SendKeys(text);
		}

		public void ClearSearch()
		{
			webDriver.ExecuteJavaScript("$('span.cancel').click()");
		}
	}
}

WebDriver factory

WebDriver factory will be responsible for instantiating the WebDriver based on a condition which browser we want to run our tests with.

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium.IE;

namespace AutomationRhapsody.DesignPatterns
{
	public class WebDriverFactory
	{
		public IWebDriver CreateInstance(Browsers browser)
		{
			if (Browsers.Chrome == browser)
			{
				return new ChromeDriver();
			}
			else if (Browsers.IE == browser)
			{
				return new InternetExplorerDriver();
			}
			else
			{
				return new FirefoxDriver();
			}
		}
	}
}

The constructor takes an argument browser type. Browser type is defined as an enumeration. This is very important. Avoid passing back and forth strings. Always stick to enums or special purpose classes. This will save you time investigating bugs in your automation.

public enum Browsers
{
	Chrome, IE, Firefox
}

NullWebElement

This is null object pattern and implements IWebElement. There is a NULL property that is used to compare is given element is not found or no.

using OpenQA.Selenium;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Drawing;

namespace AutomationRhapsody.DesignPatterns
{
	public class NullWebElement : IWebElement
	{
		private const string nullWebElement = "NullWebElement";
		public bool Displayed { get { return false; } }
		public bool Enabled { get { return false; } }
		public Point Location { get { return new Point(0, 0); } }
		public bool Selected { get { return false; } }
		public Size Size { get { return new Size(0, 0); } }
		public string TagName { get { return nullWebElement; } }
		public string Text { get { return nullWebElement; } }
		public void Clear() { }
		public void Click() { }
		public string GetAttribute(string attributeName) { return nullWebElement; }
		public string GetCssValue(string propertyName) { return nullWebElement; }
		public void SendKeys(string text) { }
		public void Submit() { }
		public IWebElement FindElement(By by) { return this; }
		public ReadOnlyCollection<IWebElement> FindElements(By by)
		{
			return new ReadOnlyCollection<IWebElement>(new List<IWebElement>());
		}

		private NullWebElement() { }

		private static NullWebElement instance;
		public static NullWebElement NULL
		{
			get
			{
				if (instance == null)
				{
					instance = new NullWebElement();
				}
				return instance;
			}
		}
	}
}

WebDriver facade

WebDriver facade main responsibility is to define custom behavior on elements location. This gives you centralized control over elements location. The constructor takes browser type and uses the factory to create WebDriver instance which is used internally in the facade. FindElement method defines explicit wait. If the element is not found then NullWebElement which is actual implementation of Null object pattern. The idea is to safely locate elements with try/catch and then just use them skipping checks for null.

using OpenQA.Selenium;
using OpenQA.Selenium.Support.UI;
using System;
using System.Collections.ObjectModel;

namespace AutomationRhapsody.DesignPatterns
{
	public class WebDriverFacade
	{
		private IWebDriver webDriver = null;
		private TimeSpan waitForElement = TimeSpan.FromSeconds(5);

		public WebDriverFacade(Browsers browser)
		{
			WebDriverFactory factory = new WebDriverFactory();
			webDriver = factory.CreateInstance(browser);
		}

		public void Start(string url)
		{
			webDriver.Url = url;
			webDriver.Navigate();
		}

		public void Stop()
		{
			webDriver.Quit();
		}

		public object ExecuteJavaScript(string script)
		{
			return ((IJavaScriptExecutor)webDriver).
				ExecuteScript("return " + script);
		}

		public IWebElement FindElement(By by)
		{
			try
			{
				WebDriverWait wait = new WebDriverWait(webDriver, waitForElement);
				return wait.Until(ExpectedConditions.ElementIsVisible(by));
			}
			catch
			{
				return NullWebElement.NULL;
			}
		}
	}
}

Tests

As I mentioned initially this post is about using efficiently design patterns in your framework automation project. Tests design are not being discussed here. Once you have designed the framework one simple test (without asserts) that makes search will look like the code below.

Browsers browser = Browsers.Chrome;
WebDriverFacade webDriver = new WebDriverFacade(browser);
webDriver.Start("https://automationrhapsody.com/examples/utf8icons.html");

HomePageObject homePage = new HomePageObject(webDriver);
homePage.ClearSearch();
homePage.SearchFor("automation");

webDriver.Stop();

Conclusion

Design patterns are enabling you to write maintainable code. They increase the value of your code. As shown in this series of posts they are not so complicated and you can easily adopt them in your automation. Most important is design patterns increase your value as a professional. So make the time to learn them!

Related Posts

Read more...

Singleton and Null object patterns

Last Updated on by

Post summary: Description with code samples of Singleton and Null object design patterns.

Singleton

Singleton pattern is used in a case where you want to restrict the instantiation of a class to only one object.

More details

The class should not require parameters for its construction. If the creation of a class requires significant resources then a singleton is a suitable solution. It is good that singletons are created when they are first needed, so-called lazy initialization.

Null object pattern

The idea is to make the code simpler and safe by skipping checks for null reference of a given object:

if (someVariable != null) {
	// Do something with someVariable
}

References

This post is part of design patterns guide series. Code examples are located in GitHub for C# and Java.

More details

Null object implements given interface and its methods are doing nothing. This makes the null object more predictable. You can safely invoke methods on the null object without the threat of a NullRefferenceException to break your application. This pattern is very well combined with singleton pattern where a null object is actually a singleton. In this case, you can check for reference or equality.

Example

Most common usage of Singleton pattern is to limit WebDriver to only one object instance in the whole automation test project.

private static IWebDriver webDriver;
public static IWebDriver WebDriver
{
	get
	{
		if (webDriver == null)
		{
			webDriver = new FirefoxDriver();
		}
		return webDriver;
	}
}

I’m going to give slightly different usage example where Singleton is used along with Null object pattern. NullWebElement implements IWebElement interface so it must implement all methods and properties defined by the interface. This is done on lines 4 to 21. Properties are returning some values, but not null! Methods are doing nothing. From line 23 to 36 is the Singleton definition. If Singleton is an object you have defined then it should have a private constructor so no one is able to instantiate it. There is a private field which actually holds the reference to the singleton. NULL is a property which instantiates the singleton if not already instantiated and returns it.

public class NullWebElement : IWebElement
{
	private const string nullWebElement = "NullWebElement";
	public bool Displayed { get { return false; } }
	public bool Enabled { get { return false; } }
	public Point Location { get { return new Point(0, 0); } }
	public bool Selected { get { return false; } }
	public Size Size { get { return new Size(0, 0); } }
	public string TagName { get { return nullWebElement; } }
	public string Text { get { return nullWebElement; } }
	public void Clear() { }
	public void Click() { }
	public string GetAttribute(string attributeName) { return nullWebElement; }
	public string GetCssValue(string propertyName) { return nullWebElement; }
	public void SendKeys(string text) { }
	public void Submit() { }
	public IWebElement FindElement(By by) { return this; }
	public ReadOnlyCollection<IWebElement> FindElements(By by)
	{
		return new ReadOnlyCollection<IWebElement>(new List<IWebElement>());
	}

	private NullWebElement() { }

	private static NullWebElement instance;
	public static NullWebElement NULL
	{
		get
		{
			if (instance == null)
			{
				instance = new NullWebElement();
			}
			return instance;
		}
	}
}

There are two main benefits of code above. The first benefit comes from the Null object pattern. You can locate an element which doesn’t exist and call some of its methods without your tests to crash. You might say that comparing for null is not a big overhead, but in big having a null check before each action is a waste of time.

IWebElement element = null;
try
{
	element = webDriver.FindElement(By.CssSelector("notExisting"));
}
catch
{
	element = NullWebElement.NULL;
}
element.Click();

It might be a discussion whether it is better to silent this failure error with a null object or leave to a massive crash. My opinion is you must fail the test and if you put some logging into Click method of the NullWebElement then you can easily trace the issue. The real benefit is you will have only one failed test instead of the whole bunch failed because of the crash.

The second benefit comes from the Singleton pattern. You can easily compare some element against the NullWebElement.NULL. Most likely you will locate the element and use it safely because of null object pattern, but there might a case where you want to see if the element is actually not found.

if (element == NullWebElement.NULL)
{
	Console.WriteLine("Element not found!");
}

Conclusion

Singleton is a pattern that you definitely should know. A null object with a combination of Singleton can decrease the amount of code you write by skipping the checks for null. I would say if adopted those can simplify your code which is a benefit in big projects.

Related Posts

Read more...

Factory design pattern

Last Updated on by

Post summary: Description of code samples of Factory design pattern.

Factory pattern is used to create objects based on specific rules.

More details

You may have several classes implementing one and the same interface. You might not know what object is suitable to get instantiated at given point in your code. Object creation may depend on different if/else or switch statements. You definitely do not want to make those is/else check every time you need an object. This is why you handle the object creation task to the factory which knows exactly what and how to create an object. The factory always returns a newly created object or re-initialized one.

References

This post is part of design patterns guide series. Code examples are located in GitHub for C# and Java.

Example

In the example I’ve given here, I’ve implemented a factory that creates IWebDriver object based on a specific input. Object creation logic is encapsulated from the outer world. This factory is very simple but is just fine for our purposes. More complex factories are defined by Factory method pattern and Abstract factory pattern but they are suitable for more complex applications. The factory in the example has an only CreateInstance method. In it, WebDriver is instantiated based on given from outside string with its name. It doesn’t store or re-use WebDriver object but always creates a new one.

public class WebDriverFactory
{
	public IWebDriver CreateInstance(string browser)
	{
		if ("Chrome".ToLower() == browser.ToLower())
		{
			return new ChromeDriver();
		}
		else if ("InternetExplorer".ToLower() == browser.ToLower())
		{
			return new InternetExplorerDriver();
		}
		else
		{
			return new FirefoxDriver();
		}
	}
}

A factory is used to instantiate WebDriver in tests. The condition which browser to instantiate may come from external parameter.

WebDriverFactory factory = new WebDriverFactory();
IWebDriver webDriver = factory.CreateInstance("Chrome");
webDriver.Url = "https://automationrhapsody.com";
webDriver.Navigate();

IWebElement logo = webDriver.FindElement(By.CssSelector("div#header-inner h1 a"));
logo.Click();

webDriver.Quit();

Conclusion

A factory is good to be used in automation testing as you have centralized control over WebDriver creation. This pattern will definitely keep you DRY. With the time being, you may add more conditions on object creation (logging, different ports, etc.).

Related Posts

Read more...

Facade design pattern

Last Updated on by

Post summary: Description with code samples of Facade design pattern.

Facade design pattern provides simple and easy to use interface to a larger and more complex code, API or set of APIs.

More details

If you have to deal with complex or poorly designed API you might find yourself confused by all the functionality available in it. Most likely you don’t need everything provided by the API. So for ease and better maintainability, only API features that are needed gets exposed outside the facade. In this way, you simplify the API usage saving time for your colleagues when maintaining this code. Also, you have control over how this external API is used and can prevent misunderstanding or misuse of it. If more functionality from API is needed this will require an update of the facade or adding new facade which exposes it.

References

This post is part of design patterns guide series. Code examples are located in GitHub for C# and Java.

Example

It might not be a good idea to have the pattern in the name of the facade, but in the example, I’ve put it to ease. In this example, WebDriver instantiation is hardcoded to Firefox, but in real life, it will be instantiated based on some rules. Facade exposes Start, Stop and FindElement methods. All other WebDriver functionality is not accessible.

public class WebDriverFacade
{
	private IWebDriver webDriver = null;
	private TimeSpan waitForElement = TimeSpan.FromSeconds(5);

	public WebDriverFacade()
	{
		webDriver = new FirefoxDriver();
	}

	public void Start(string url)
	{
		webDriver.Url = url;
		webDriver.Navigate();
	}

	public void Stop()
	{
		webDriver.Quit();
	}

	public IWebElement FindElement(By by)
	{
		try
		{
			WebDriverWait wait = new WebDriverWait(webDriver, waitForElement);
			return wait.Until(ExpectedConditions.ElementIsVisible(by));
		}
		catch
		{
			return null;
		}
	}
}

A facade is used exactly the same way as WebDriver itself:

WebDriverFacade webDriver = new WebDriverFacade();
webDriver.Start("https://automationrhapsody.com");

IWebElement logo = webDriver.FindElement(By.CssSelector("div#header-inner h1 a"));
logo.Click();

webDriver.Stop();

Conclusion

WebDriver API is neither complex nor poorly designed, so maybe making a facade is not a mandatory thing. Still, I think having control over its usage is a good approach. In given example, you can see I use explicit wait so I locate exactly the same way all elements in my automation project.

Related Posts

Read more...

Page objects design pattern

Last Updated on by

Post summary: Description of code samples of Page Objects design pattern.

This is the most important pattern related to software automation. It enables you to create object repository with UI elements. Elements are separated from test logic. This pattern makes code much more maintainable and reusable.

References

This post is part of design patterns guide series. Code examples are located in GitHub for C# and Java.

Never do this

Without this pattern what you might do is start WebDriver, navigate to some test page. Locate element you need then click it.

IWebElement element = webDriver.FindElement(By.CssSelector("div.find label"));
element.Click();

This is good when you want to test the idea or do some quick demo. In a real-life project, this element might be needed in dozens of tests. What happens if the UI changes and CSS selector is not matching anymore? Here comes the maintainability problem, you have to search and replace all of them.

The proper way

In software engineering, there is a principle called DRY for short. Its idea is to have each element or action stored only once in a system. This avoids copy/paste and reduces the overhead for code maintenance. The same idea is used in Page Object pattern.

  • Each page or re-usable part of a page (i.e. header, footer, menu) is a separate class.
  • Class constructor takes WebDriver as an argument and uses it internally to locate elements.
  • Each element is a private property (or getter in Java).
  • Actions are public and internally operate with elements.

In the code below SearchField is private property used only by SearchFor method which is exposed to available action on HomePage. An element can be located inside the action method but suggested approach gives better readability. And if an element is needed more than once then defining it separately is a must.

public class HomePage
{
	private IWebDriver webDriver;

	public HomePageObject(IWebDriver webDriver)
	{
		this.webDriver = webDriver;
	}

	private IWebElement SearchField
	{
		get { return webDriver.FindElement(By.Id("search")); }
	}

	public void SearchFor(string text)
	{
		SearchField.SendKeys(text);
	}
}

The page object is instantiated in the test and actions are invoked.

HomePageObject homePage = new HomePageObject(webDriver);
homePage.SearchFor("automation");

With this approach, you have one element defined in only one place. Only actions are exposed out of the page object. It is very clear what actions can be done on this page.

Original page objects

To be honest what I have described in this post is not actually the original Page Objects pattern. Although the idea is same I have modified it slightly. Originally each action returns a page object. In this way, you can chain methods in one test and very easy to understand if a test will be broken when given action now returns different page object. I do not like chaining of methods though so I don’t need this extra. It doesn’t bother you at all to do

public HomePage SearchFor(string text)
{
	SearchField.SendKeys(text);
	return this;
}

Also, I have separated elements located outside of actions because I want better maintainability and readability of the code.

Conclusion

Always use page objects in your test automation. ALWAYS!

Related Posts

Read more...

Design patterns every test automation engineer should know

Last Updated on by

Post summary: Description with examples of 5 design patterns that is good to know and use in our automation testing.

Design patterns are an interesting topic. They are created to solve common problems in software design. Although design patterns are not reserved only for software development they seem not to be widely discussed in software automation. Maybe this is because the topic sounds complicated. Yes, there are really sophisticated design patterns used to solve complex issues in software development. But also there are easy to understand and adopt design patterns that can significantly improve readability and maintainability of our test automation code.

With series of posts, I’m going to give an overview with examples of some design patterns that can be very useful in our test automation projects. To keep blog neat in this first post I’ll just outline the list of patterns. Each one will be discussed in a separate post with code samples.

Page Objects pattern

This is the most important pattern related to software automation. It enables you to create object repository with UI elements. Elements are separated from tests logic. This pattern makes code much more maintainable and reusable. Page objects design pattern

Facade pattern

The whole idea of facade design pattern is to provide simple and easy to use interface to a larger and more complex code, API or set of APIs. Most likely you don’t need everything provided by the API. So for ease and better maintainability, only API features that are needed gets exposed outside the facade. In this way, you simplify the API usage and you have control over how this external API is used and can prevent misunderstanding or misuse. Facade design pattern

Factory pattern

Factory pattern is used to create objects based on specific rules. You may have several classes implementing an interface. In your code, you do not want to bother defining which concrete class to instantiate or you might not know what object is suitable to get instantiated. This is why you handle the object creation task to the factory which knows exactly what object to create. With factory pattern, object creation is encapsulated. Factory design pattern

Singleton pattern

Singleton pattern is needed when you need exactly one object from a specific class in the whole application. Singleton and Null object patterns

Null object pattern

The idea is to make code simpler and safe by skipping null reference checks. Null object implements given interface and its methods are doing nothing. This makes the null object predictable. You can safely invoke methods on the null object without the threat of a NullReferenceException to break your application. Singleton and Null object patterns

Putting all together

Once you have gone through patterns one by one now it is time to put everything into one project. Complete guide how to use design patterns in automation post is showing how.

Related Posts

Read more...