Advanced WPF automation – memory usage

Last Updated on by

Post summary: Highlight eventual memory issue when using Telerik Testing Framework and TestStack White for desktop automation.

Memory is an important aspect. When you have several test cases it is not a problem. But on large projects with many tests memory turn out to be a serious issue.

Reference

This post is part of Advanced WPF desktop automation with Telerik Testing Framework and TestStack White series. The sample application can be found on GitHub.

Problem

Like every demo on certain technology automating WPF applications looks cool. And also like every technology problems occur when you start to use it on a large scale. Problem with WFP automation with Telerik Testing Framework and TestStack White is the memory. When your tests’ number grows frameworks start to use too much memory. By too much, I mean over 1GB which might not seem a lot but for a single process actually is. Increasing RAM of test machine is an only temporary solution and is not one that can be scaled.

Why so much memory

I have a project with 580 tests and 7300 verification points spread in 50 test classes. I’ve spent lots of hours debugging and profiling with several .NET profiling tools. In the end, all profilers show that a large amount of memory used is in unmanaged objects. So generally there is nothing you can do. It seems like some of the frameworks or both have memory issues and do not free all memory they use.

Solution

The solution is pretty simple to suggest but harder to implement – run each test class in the separate process. I’m using a much more enhanced version of NTestsRunner. It is running each test class in separate windows process. Once the test is finished results are serialized in results directory, the process is exited and all memory and object used for this test are released.

Conclusion

The memory could be a crucial factor in an automation project. Be prepared to have a solution for it. At this point, I’m not planning to put running tests in a separate process in NTestsRunner. If there is demand it is a pretty easy task to do it.

Related Posts

Read more...

Advanced WPF automation – working with WinForms grid

Last Updated on by

Post summary: Example how to work with WinForms grid with TestStack White.

TestStack White is a really powerful framework. It works on top of Windows UI Automation framework hiding its complexity. If White is not able to locate element you have access to underlying UI Automation and you can do almost anything you need.

Reference

This post is part of Advanced WPF desktop automation with Telerik Testing Framework and TestStack White series. The sample application can be found on GitHub.

MainGrid

For single responsibility separation grid logic is in separate class MainGrid.cs. The constructor takes White.Core.UIItems.WindowItems.Window object. Inside the window, we search for an element with control type ControlType.Table. It is the only one of its kind. If there are more we should narrow down the SearchCriteria.

public class MainGrid
{
	private Table table;
	public MainGrid(Window window)
	{
		SearchCriteria search = SearchCriteria.ByControlType(ControlType.Table);
		table = window.Get<Table>(search);
	}

	public string GetCellText(int index)
	{
		TableCell cell = GetCell(index);
		string value = cell.Value as string;
		return value;
	}

	public void ClickAtRow(int row)
	{
		TableCell cell = GetCell(row);
		Point topLeft = cell.Bounds.TopLeft;
		topLeft.X += 5;
		topLeft.Y += 5;
		Mouse.instance.Click(topLeft);
	}

	private TableCell GetCell(int index)
	{
		TableRows rows = table.Rows;
		TableCells cells = rows[index - 1].Cells;
		return cells[0];
	}
}

Access the grid

MainGrid is property inside MainWindow page object. On access to the property new object is instantiated. This might lead to performance issues if grid search and instantiation is slow. So, in this case, you can use Singleton design pattern. Singleton might lead to issues with old object state which will be hard to debug. It depends what your priorities are.

public class MainWindow : XamlElementContainer
{
	public static string WINDOW_NAME = "MainWindow";
	private Application app;
	private string mainPath =
		"XamlPath=/Border[0]/AdornerDecorator[0]/ContentPresenter[0]/Grid[0]/";
	public MainWindow(VisualFind find, Application application)
		: base(find)
	{
		app = application;
	}

	private MainGrid MainGrid
	{
		get
		{
			return new MainGrid(app.GetWindowByName(WINDOW_NAME));
		}
	}

	public void ClickTableAtRow(int row)
	{
		MainGrid.ClickAtRow(row);
	}

	public Verification VerifyTableCell(int index, string text)
	{
		return BaseTest.VerifyText(text, MainGrid.GetCellText(index));
	}
}

Conclusion

TestStack White is a powerful framework. It will be perfect if you can do the job without it. If you cannot you are lucky it exists.

Related Posts

Read more...

Advanced WPF desktop automation

Last Updated on by

Post summary: In this series of posts I’ll expand the examples and ideas started in Automation of WPF applications series.

Telerik Testing Framework and TestStack White are powerful tools for desktop automation. You can automate almost everything with a combination of those frameworks. This series of posts will give more details how to automate more complex applications.

Reference

Code samples are located in GitHub SampleAppPlus repository. Telerik Testing Framework requires installation as it copies lots of assemblies in GAC.

SampleAppPlusThere is SampleAppPlus which is actually a dummy application with only one purpose to be used to demonstrate automation principles. With this application, you can upload an image file. Once uploaded image is visualized. The image path is listed in a table. The image path is also visualized as an image in a custom control in the bottom of the main window. The user is able to add more text which is added to the table as long as editing already existing text. Add and edit are reflected on custom image element.

Topics

  • Page objects inheritance of similar windows
  • Working with WinForms grid
  • Windows themes and XamlPath
  • Read dependency property
  • NTestsRunner in action
  • Extension methods
  • Memory usage

Page objects inheritance

It is common to have similar windows in an application. Each window is modeled as page object in automation code. If windows are also similar in terms of internal structure it is efficient to re-use similar part and avoid duplications. Re-use is achieved with inheritance. Given SampleAppPlus application has very similar windows for adding and editing text. Code examples show how to optimize your effort and re-use what is possible to be re-used. More details can be found in Advanced WPF automation – page objects inheritance post.

Working with WinForms grid

As mentioned before Telerik Testing Framework is not very good with WinForms elements. This is the main reason to use TestStack White. It is not very likely to have WinForms elements in WPF application but in order to complete the big picture I’ve added such grid in a SampleAppPlus application. Code examples show how to manage WinForms grid. More details can be found in Advanced WPF automation – working with WinForms grid post.

Windows themes and XamlPath

In given examples elements are located with exact XamlPath find expression. This approach has a serious problem related to Windows themes. For complex user interfaces, XamlPath could be different on a different theme. Windows Classic theme sometimes produces different XamlPath in comparison with standard Windows themes. Yes, it is no more available from Windows 8 but Server editions are working only with Windows Classic theme. So one and the same tests could have differences. I couldn’t find a way to automatically detect which is current theme. The solution is to have different XamlPath for both standard and classic themes. Once you have it you can switch them manually with some configuration or you can try to automate the switch by locating element for which you know is different and save variable based on its location result.

Read dependency property

A dependency property is a way in C# to extend the standard provided functionality. It can happen in a real application that developers use such functionality. Given SampleAppPlus application has a special element with dependency property. Code examples show how to extract property value and use it in your tests. More details can be found in Advanced WPF automation – read dependency property post.

NTestsRunner in action

I’ve introduced NTestsRunner which is a custom way for running functional automated tests. Code samples show how to use it and create good tests that are run only with this tool.

Extension methods

Extension methods are one extremely good feature of .NET framework. I personally like them very much. I assume everyone writing code in C# is aware of them. Still, in code examples show how they can be used.

Memory usage

Memory is not a problem on small projects. But when the number of tests continue to grow it actually becomes a problem. More details can be found in Advanced WPF automation – memory usage post.

Related Posts

Read more...

WPF automation – running the tests

Last Updated on by

Post summary: How to sell your automation to management. Guide for running the tests unattended.

References

This post is part of Automation of WPF applications with Telerik Testing Framework and TestStack White series. The sample application can be found in GitHub SampleApp repository.

Test frameworks need mouse and keyboard

As you have noticed when running the examples it is not possible to do anything else while tests are running because both Telerik Testing Framework and TestStack White are using mouse and keyboard in order to click and type text. This is how both frameworks are designed and work. It doesn’t seem very effective if you are wasting your time watching tests running on your workstation instead of doing something productive. This is not a good argument when advocating your automation.

The essence of automation is being effective

This topic is not about Automation vs. Manual testing so I will not go in that direction. I’ll just say that still there are companies lacking management willpower to support and embrace automation. So we need to be good salesmen!

How to drive your automation to success

As I said we need to be wise when promoting our automation to get more time and resources. Sales strategy is pretty easy and straightforward not requiring huge investments:

  • Free to use and pretty easy to work with frameworks.
  • Some enthusiasm to make the first automation. Remember, be smart and first automate most repeating scenarios. This will show real results and help you buy some time for further automation.
  • Virtual (or real) machine with test running on it during the night
  • Mail with results is sent to management with results in the morning

Run the tests unattended

You get the machine, set up the framework, set up scheduled tasks for deploying the latest application under test and latest test code. Run the tests and get into trouble. Tests do not run! This is because there are special requirements to schedule unattended run. There must be an active Windows session in order mouse and keyboard to be used. Once the session is interrupted tests stop. This is detailed KB article on the topic with several possible solutions.

Working solution

You can try solutions in the article to see which works best for you. For me, the solution is to have a remote desktop in remote desktop. This requires Windows Server installation. The only server provides two simultaneous remote desktop sessions. There are unofficial patches for non-server versions which I haven’t tried and cannot comment. Two local users are needed on the Windows Server with no desktop locking or screen saver (domain users most likely will have desktop locked after a time). Log from your machine to test machine with the first user. From first user’s session log to same testing machine with the second user. Tests are started from second user’s session. Once tests are started you can freely close desktop (not log out!). You can create a scheduled task which runs only when the user is logged in and just wait for results in the morning. If two accounts are overhead there is an option to use software that prevents your computer to lock. See prevent screen lock thread if something works for you.

Conclusions

Automation is an exciting field of career development for test engineers. This blog is dedicated to automation testing. You will find very useful and interesting topics in it. I would definitely encourage you just to give it a try. Good luck!

Related Posts

Read more...

WPF automation – using the elements

Last Updated on by

Post summary: Use already created Page Objects and build up test framework.

References

This post is part of Automation of WPF applications with Telerik Testing Framework and TestStack White series. The sample application can be found in GitHub SampleApp repository.

Page Objects holder

Below is App.cs which is a representation of application under test.

using ArtOfTest.WebAii.Wpf;
using White.Core;
using White.Core.UIItems.WindowItems;

namespace SampleApp.Tests.Framework.Elements
{
	public class App
	{
		public WpfApplication ApplicationWebAii { get; private set; }
		public Application ApplicationWhite { get; private set; }

		public App(WpfApplication webAiiApp, Application whiteApp)
		{
			ApplicationWebAii = webAiiApp;
			ApplicationWhite = whiteApp;
		}

		public MainWindow MainWindow
		{
			get
			{
				return new MainWindow(ApplicationWebAii
					.WaitForWindow(MainWindow.WINDOW_NAME).Find);
			}
		}

		public OpenFile OpenFile
		{
			get
			{
				return new OpenFile(GetWindowByName("Open"));
			}
		}

		public MessageBox MessageBox
		{
			get
			{
				return new MessageBox(GetWindowByName(""));
			}
		}

		private Window GetWindowByName(string windowName)
		{
			// Workaround as method GetWindow(string title) is not working
			foreach (Window window in ApplicationWhite.GetWindows())
			{
				if (windowName.Equals(window.Name))
				{
					return window;
				}
			}
			return null;
		}
	}
}

The constructor takes an instance of Telerik Testing Framework’s application (WpfApplication) and TestStack White’s application (Application). Those are stored in the App instance.

Access the Page Objects

Each window in a real application is represented by a property in the App class. When accessed a new object of this page object class is created and its elements can be accessed.

WPF page objects require VisualFind in order to be instantiated. It is obtained by first locating the window with Telerik’s

public WpfWindow WaitForWindow(string caption);

From the located window we need only the VisualFind which is used internally to locate elements on that particular window.

WinForms page objects require White’s Window instance in order to be instantiated. The window is located by

public virtual Window GetWindow(string title);

I found this method not always working so I’ve made a workaround method

private Window GetWindowByName(string windowName);

New Page Objects vs. Cached Page Objects

In the example above every time, an action is required a new page object is instantiated. In some cases instantiating the object may require longer time or you might need some properties in this object preserved during tests. In such cases, you may use Singleton design pattern and instantiate only one object.

private MessageBox messageBox = null;
public MessageBox MessageBox
{
	get
	{
		if (messageBox == null)
		{
			messageBox = new MessageBox(GetWindowByName(""));
		}
		return messageBox;
	}
}

Both approaches have pros and cons. In case of new page object, you always work with a fresh instance without any previous state saved. This might require more time to instantiate the objects and you are not able to save previous states. Cached objects may be much faster as a performance but having internal state may lead to unexpected bugs in your automation.

Base test

Finally to make all work we need an instance of App. The instance is created in BaseTest.cs class.

using ArtOfTest.WebAii.Core;
using SampleApp.Tests.Framework.Elements;
using White.Core;

namespace SampleApp.Tests.Framework.Tests
{
	public class BaseTest
	{
		protected App App { get; set; }
		private string applicationPath =
			"C:\\SampleApp\\SampleApp\\bin\\Debug\\SampleApp.exe";

		protected void Start()
		{
			if (App == null)
			{
				Application appWhite = Application.Launch(applicationPath);
				Manager manager = new Manager(false);
				manager.Start();
				App = new App(
					manager.ConnectToApplication(appWhite.Process), appWhite);
			}
		}

		protected void Stop()
		{
			if (App != null && App.ApplicationWhite != null)
			{
				App.ApplicationWhite.Kill();
			}
			App = null;
		}
	}
}

All tests inherit from the base test class. Initialise and clean up code is added in the base test. In our case Start() method is the initializer. It must be called in order to instantiate App class. App property is protected so every extending class has access to it.

Initialise the frameworks

In order to start the application under test, we need the full path to the exe file. In this example, this is hardcoded but in real life, it will be configurable. Start the application with White:

public static Application Launch(string executable);

Once started then connect to it with Telerik framework by creating a Manager and use its

public WpfApplication ConnectToApplication(Process proc, string pid = null);

The process is obtained out of White Application.Process property. Opposite launch order is not working. White is not able to Attach to running process.

Use page objects

Once Start() method is called, the application under test is started and both frameworks are connected to it you can simply do in your test:

App.MainWindow.ClickBrowseButton();

This will find and create a new instance of MainWindow and then it will find and click Browse button. Your framework defines the actions on elements which are later used in actual tests. Once all the work on framework has been done it is that simple to build your tests.

Clean up

Stop() method is called at the end of the test in order to close the application under test by killing the underlying process.

The tests

This is unit test created with MS Unit Testing Framework in order to demonstrate real testing on the application.

using Microsoft.VisualStudio.TestTools.UnitTesting;
using SampleApp.Tests.Framework.Tests;

namespace SampleApp.Tests
{
    [TestClass]
    public class UnitTest1 : BaseTest
    {
		[TestInitialize]
		public void Initialise()
		{
			Start();
		}

		[TestMethod]
		public void OpenFile_OnCancel_GivesMessage()
		{
			App.MainWindow.ClickBrowseButton();
			App.OpenFile.ClickCancelButton();
			Assert.AreEqual("Problem occured, try again later",
				App.MessageBox.GetText());
			App.MessageBox.ClickOkButton();
		}

		[TestMethod]
		public void OpenFile_OnAttachFile_GivesMessageAndFileIsShown()
		{
			string filePath = @"C:\SampleApp\SampleApp\bin\Debug\HappyFace.jpg";
			App.MainWindow.ClickBrowseButton();
			App.OpenFile.EnterFileName(filePath);
			App.OpenFile.ClickOpenButton();
			Assert.AreEqual("Successfully done", App.MessageBox.GetText());
			App.MessageBox.ClickOkButton();
			Assert.AreEqual(filePath, App.MainWindow.GetFilePathAtIndex(1));
		}

		[TestCleanup]
		public void CleanUp()
		{
			Stop();
		}
    }
}

Unit testing frameworks

Unit testing frameworks are designed to run tests in random order. Before each test method annotated with [TestInitialize] is run. In our case application is started. After each test method annotated with [TestCleanup] is run. In our case application is stopped. For this simple application running tests with a unit testing framework is OK. We are not doing unit tests but functional once. So for bigger and more complex tests unit testing frameworks are not very convenient. I’ve created very simple tests runner. This post describing the need of such tests runner.

This post shows how to build up the framework based on page objects. Next post is WPF automation – running the tests.

Related Posts

Read more...

WPF automation – locating and structure of WinForms elements

Last Updated on by

Post summary: Guide how to locate WinForms elements with TestStack White.

References

This post is part of Automation of WPF applications with Telerik Testing Framework and TestStack White series. The sample application can be found in GitHub SampleApp repository.

WinForms elements manipulation with TestStack White

MessageBox.cs class is a representation of a MsgBox of our sample application which is WinForms.

using System.Windows.Automation;
using White.Core.UIItems;
using White.Core.UIItems.Finders;
using White.Core.UIItems.WindowItems;

namespace SampleApp.Tests.Framework.Elements
{
	public class MessageBox
	{
		private Window win;
		public MessageBox(Window window)
		{
			win = window;
		}

		private Button Button_OK
		{
			get
			{
				return win.Get<Button>(
					SearchCriteria.ByControlType(ControlType.Button).
					AndByText("OK"));
			}
		}
		private Label Label_Text
		{
			get
			{
				return win.Get<Label>(
					SearchCriteria.ByControlType(ControlType.Text));
			}
		}

		public void ClickOkButton()
		{
			Button_OK.Click();
		}

		public string GetText()
		{
			return Label_Text.Text;
		}
	}
}

MessageBox is following Page Object design pattern. Elements are private properties. Actions are public methods acting on elements. Again: one element must be defined in only one place.

Constructor receives an instance of the current window object. Elements are located by White framework method:

public virtual T Get<T>(SearchCriteria searchCriteria) where T : UIItem;

SearchCriteria

SearchCriteria is used for locating elements. Search is possible with a combination of following criteria: Text, ControlType, AutomationId, NativeProperty, Framework. And here comes the hardness of working with White in complex UIs. There may be several TextBox-es (ByControlType == Edit) AND with no text inside (AndByText == “”). You can take them all and then access by index the one which is proper for your test. And what happens when you fill text in one of them. Next find has one less window. If you think it is easy then enjoy, I would rather not do it.

UISpy

Finding of the proper SearchCriteria is done with powerful tool UISpy.exe. Unfortunately, this one is deprecated from Windows 8. MS have added a replacement called Inspect.exe. I don’t find it useful this is why I have added UISpy.exe in the GitHub repository.

UISpy

Dialog is the message box window which is a child of MainWindow. It has empty title and two elements: OK button and message text. Current example shows the location of the text element. Its MS ControlType == ControlType.Text. This is what we need to locate it since there is only one such element in the message box. Once we have it we take the text and test compares current vs. expected text. OK button is located by ControlType and text. In case of different localisations we may leave just the ControlType, but if there are more buttons text is mandatory to distinguish the correct button.

Once you have defined your page objects it is time to assemble them. Next post is WPF automation – locating and structure of WPF elements.

Related Posts

Read more...

WPF automation – projects structure

Last Updated on by

Post summary: How to structure projects in the course of WPF automation.

References

This post is part of Automation of WPF applications with Telerik Testing Framework and TestStack White series. The sample application can be found in GitHub SampleApp repository.

Overview

For better maintainability and good design, automation testing is separated into two projects. One is the so-called framework project. It has knowledge about third used party frameworks (Telerik Testing Framework, TestStack White, Selenium, Watij, etc.). It operates on elements via those third-party frameworks. Framework exposes actions that are building blocks of the tests. The other project is the tests. It doesn’t know anything about used automation frameworks. It uses methods exposed by an internal framework and builds the tests with them. The idea is that one team/person with development knowledge can be responsible for the internal framework, others can be responsible for writing well-designed tests using the internal framework only. Another benefit of two projects is the ability to completely change internal framework project without affecting test logic at all. Here is the structure of my sample projects:

Projects structure

Projects structure

SampleApp – This is the application under test. Generally, it is not in the test project, you receive it from developers or from nightly build or from some other place. For the purpose of this demo, I’ve created a very basic application that uploads an image and shows it inside. There is attached image (HappyFace.jpg) that is used in the tests.

SampleApp.Tests – Here is the most important artifact – the tests, the purpose of all this fuss. In the demo, this is made as UnitTest with MS Unit Testing Framework. You can use whatever run strategy you like – another unit testing framework (xUnit, nUnit, etc.) or you can build your own tests runner. I have built my own tests running because by definition unit tests should be independent of each other thus MS Unit Testing Framework runs them in random order. It might be possible to manage order, but this will require managing of some sort of external files. It gives too much overhead to me.

SampleApp.Tests.Framework – This is where the interesting part is. Here are the elements – main application form (WPF), message boxes shown on success or error (WinForms) and open file dialogue (WinForms). MainWindow is manipulated with Telerik Testing Framework, other two are manipulated with White. App.cs is a holder which represent application itself and elements are accessible through it. BaseTest holds an instance of App. It is extended by UnitTests and App and its elements are accessible.

Next post is WPF automation – locating and structure of WPF elements in the code.

Related Posts

Read more...

Automation of WPF applications

Last Updated on by

Post summary: Overview how to successfully automate WPF application with Telerik Testing Framework and TestStack White.

I’ve spent lots of time in automation of a WPF application. Nowadays the world is dealing with web application and desktop application are fading away. Nevertheless, desktop applications still have their own private space and I hope these series of posts will be useful to someone.

Preface

An application under test is created with WPF but has WinForms grids included on different windows presenting lots of information. Performance is the main reason to use WinForms grids inside a WPF application. WPF grid with 40 columns and 100 rows lags a lot. After research and proof of concept of commercial and open source tools and frameworks, two were chosen: Telerik Testing Framework and TestStack White. They provide flexibility and maintainability which is so vital in a fast-changing application. Both are used with C#. Every one has its own weaknesses and strengths but both made a perfect synergy to provide you the possibility to automate almost everything.

Telerik Testing Framework

Extremely powerful and free to use framework. You can purchase additional support which is at reasonable prices. Telerik also provides Test Studio, a tool and MS Visual Studio plug-in, build on top of the framework. This is also an option, but for me using a tool gives more restrictions than benefits. Their framework is the best available for testing WPF applications. It is very easy to use. Elements are easily located by XamlPath (very similar to XPath). Of course, there are other locate options. The framework provides very rich API with lots of operations over the elements. Still, its powerfulness is limited to WPF. It is not that good with WinForms elements. Telerik Testing Framework requires installation as it copies lots of assemblies in GAC.

TestStack White

Extremely powerful open source framework. Build as a wrapper of Microsoft UI Automation framework you can automate almost everything with it. But this comes at a price. It is hard to locate elements. I use it only where Telerik Testing Framework is not capable to do the work – WinForm grids, context menus, closing windows with small X in the top right corner?! Yes, those are WPF inside but are hosted in operation system window and Telerik has no access to it.

In GitHub, I have uploaded an MS Visual Studio 2013 project with very simple application taken from this example. I have added very basic tests to it to illustrate how it is done. With upcoming posts, I’ll explain in details the idea and implementation.

Below is a list of posts included in the series:

  1. Introduction (current post)
  2. WPF automation – projects structure
  3. WPF automation – locating and structure of WPF elements
  4. WPF automation – locating and structure of WinForms elements
  5. WPF automation – using the elements
  6. WPF automation – running the tests

Related Posts

Read more...