Advanced WPF automation – memory usage

Last Updated on by

Post summary: Highlight eventual memory issue when using Telerik Testing Framework and TestStack White for desktop automation.

Memory is important aspect. When you have several test cases it is not a problem. But on large projects with many tests memory turn out to be serious issue.

Reference

This post is part of Advanced WPF desktop automation with Telerik Testing Framework and TestStack White series. Sample application can be found in GitHub.

Problem

Like every demo on certain technology automating WPF applications looks cool. And also like every technology problems occur when you start to use it in a large scale. Problem with WFP automation with Telerik Testing Framework and TestStack White is memory. When your tests’ number grows frameworks start to use too much memory. By too much I mean over 1GB which might not seems a lot but for a single process actually is. Increasing RAM of test machine is only temporary solution and is not one that can be scaled.

Why so much memory

I have project with 580 tests and 7300 verification points spread in 50 test classes. I’ve spend lots of hours debugging and profiling with several .NET profiling tools. In the end all profilers show that the large amount of memory used is in unmanaged objects. So generally there is nothing you can do. It seems like some of the frameworks or both have memory issues and do not free all memory they use.

Solution

Solution is pretty simple to suggest but harder to implement – run each test class in separate process. I’m using much more enhanced version of NTestsRunner. It is running each test class in separate windows process. Once test is finished results are serialised in results directory, process is exited and all memory and object used for this test is released.

Conclusion

Memory could be a crucial factor in an automation project. Be prepared to have solution for it. At this point I’m not planing to put running tests in separate process in NTestsRunner. If there is demand it is pretty easy task to do it.

Read more...

Text verification

Last Updated on by

Post summary: Verify actual text with expected one by ignoring what is not relevant during compare.

In automation testing there is no definitive way what text verification is best to be done. One strategy is to check that an expected word or a phrase exists in actual text shown in application under tests. Other strategy is to prepare large amount of text to verify. Later strategy is expensive in case of effort for preparation and maintenance. First strategy might not be sufficient to do correct verifications.

In between

What I suggest here is something in between. Not too much but not too less. Problem with paragraph of text to be verified is it might contain data we do not have control over, e.g. date, time, unique values, etc.

Example

Imagine an e-commerce website. When you place the order there is order confirmation page. You want to verify not only that you are on this page but also that text is correct as per specification. Most likely text will contain data you do not have control over – order number and date. Breaking verification is small chunks is an option. Another option is to manipulate the actual text. Third option is to define the text as expected with special strings that will get ignored during compare.

Actual vs Expected

Actual text could be: “Order 123456 has been successfully placed on 01.01.1970! Thank you for your order. ”
Expected text could be: “Order ~SKIP~ has been successfully placed on ~SKIP~! Thank you for your order. ”
And then you can compare both where ~SKIP~ will be ignored during compare.

Compare code

Code to do the compare shown above is incorporated in NTestsRunner also:

public const string IgnoreDuringCompare = "~SKIP~";

public static bool EqualsWithIgnore(this string value1, string value2)
{
	string regexPattern = "(.*?)";
	// If value is null set it to empty
	value1 = value1 ?? string.Empty;
	value2 = value2 ?? string.Empty;
	string input = string.Empty;
	string pattern = string.Empty;
	// Unify new lines symbols
	value1 = value1.Replace("\r\n", "\n");
	value2 = value2.Replace("\r\n", "\n");
	// If no one conains ignore string then compare directly
	if (!value1.Contains(IgnoreDuringCompare) &&
		!value2.Contains(IgnoreDuringCompare))
	{
		return value1.Equals(value2);
	}
	else if (value1.Contains(IgnoreDuringCompare))
	{
		pattern = Regex.Escape(value1).Replace(IgnoreDuringCompare, regexPattern);
		input = value2;
	}
	else if (value2.Contains(IgnoreDuringCompare))
	{
		pattern = Regex.Escape(value2).Replace(IgnoreDuringCompare, regexPattern);
		input = value1;
	}

	Match match = Regex.Match(input, pattern);
	return match.Success;
}

Use in tests

In your tests you will do something like:

string actual = OrderConfirmationPage.GetConfirmationText();
string expected = "Order " + ExtensionMethods.IgnoreDuringCompare +
	" has been successfully placed on " + ExtensionMethods.IgnoreDuringCompare +
	"! Thank you for your order. ";
Assert.IsTrue(actual.EqualsWithIgnore(expected));

Conclusion

It might take little bit more effort to prepare expected strings but verification will be more complete and correct rather than just to expect a word or a phrase.

Read more...

Advanced WPF desktop automation

Last Updated on by

Post summary: In this series of posts I’ll expand the examples and ideas started in Automation of WPF applications series.

Telerik Testing Framework and TestStack White are powerful tools for desktop automation. You can automate almost everything with combination of those frameworks. This series of posts will give more details how to automate more complex applications.

Reference

Code samples are located in GitHub SampleAppPlus repository. Telerik Testing Framework requires installation as it copies lots of assemblies in GAC.

SampleAppPlusThere is SampleAppPlus which is actually a dummy application with only one purpose to be used to demonstrate automation principles. With this application you can upload image file. Once uploaded image is visualised. Image path is listed in a table. Image path is also visualised as image in a custom control in bottom of main window. User is able to add more text which is added to table as long with editing already existing text. Add and edit are reflected on custom image element.

Topics

  • Page objects inheritance of similar windows
  • Working with WinForms grid
  • Windows themes and XamlPath
  • Read dependency property
  • NTestsRunner in action
  • Extension methods
  • Memory usage

Page objects inheritance

It is common to have similar windows in an application. Each window is modelled as page object in automation code. If windows are also similar in terms of internal structure it is efficient to re-use similar part and avoid duplications. Re-use is achieved with inheritance. Given SampleAppPlus application has very similar windows for adding and editing text. Code examples show how to optimise your effort and re-use what is possible to be re-used. More details can be found in Advanced WPF automation – page objects inheritance post.

Working with WinForms grid

As mentioned before Telerik Testing Framework is not very good with WinForms elements. This is the main reason to use TestStack White. It is not very likely to have WinForms elements in WPF application but in order to complete the big picture I’ve added such grid in SampleAppPlus application. Code examples show how to manage WinForms grid. More details can be found in Advanced WPF automation – working with WinForms grid post.

Windows themes and XamlPath

In given examples elements are located with exact XamlPath find expression. This approach has serious problem related to Windows themes. For complex user interfaces XamlPath could be different on different theme. Windows Classic theme sometimes produces different XamlPath in comparison with standard Windows themes. Yes it is no more available from Windows 8 but Server editions are working only with Windows Classic theme. So one and the same tests could have differences. I couldn’t find a way to automatically detect which is current theme. Solution is to have different XamlPath for both standard and classic themes. Once you have it you can switch them manually with some configuration or you can try to automate the switch by locating element for which you know is different and save variable based on its location result.

Read dependency property

Dependency property is a way in C# to extend the standard provided functionality. It can happen in real application that developers use such functionality. Given SampleAppPlus application has special element with dependency property. Code examples show how to extract property value and use it in your tests. More details can be found in Advanced WPF automation – read dependency property post.

NTestsRunner in action

I’ve introduced NTestsRunner which is custom way for running functional automated tests. Code samples show how to use it and create good tests that are run only with this tool.

Extension methods

Extension methods are one extremely good feature of .NET framework. I personally like them very much. I assume everyone writing code in C# is aware of them. Still in code  examples show how they can be used.

Memory usage

Memory is not a problem on small projects. But when number of tests continue to grow it actually becomes a problem. More details can be found in Advanced WPF automation – memory usage post.

Read more...

Efficiently use of enumerations with string values in C#

Last Updated on by

Post summary: Using enumerations or specialised classes makes your automation tests easy to understand and maintain. Show with code samples how to define and read string value to enumeration elements.

When you do automation tests and have to pass value to a method it is so easy and natural to just use strings. There are many cases where string is the correct solution. There are also many cases where string can be solution, but enumeration or specialised class are better and more efficient solution.

Why not strings

Having the following example – web application with drop down which has several options. We are using Page objects pattern to model the page. Page object has a method which accepts the option to be selected. String seems as a natural solution but is wrong. Although string will work enumeration is the only right solution. Drop down has limited and already defined options that can be selected. Exposing just string may cause misinterpretations for consumer of your method. It is much more easy to limit the consumer to several enumeration values. In this way consumer knows what data to provide and this automatically keeps code clean from magic strings. If changes are needed they will be done only in the enumeration making code easier to maintain.

Problem with enumerations in C#

Using enumerations for example given above will not work. Unlike Java enumerations in C# are wrappers for int or other numeric types value. You are not able to use text with enumeration element.

Using string values with enumerations

Only way to use string values in enumerations is by adding it as an attribute to each enumeration’s element. It takes several steps in order to accomplish this.

  1. Create the attribute class that will be applied to enumeration element
  2. Create extension method that is responsible for reading string value from enumeration element
  3. Apply string value attribute to enumeration element
  4. Use in code

Bellow are code samples how to use string values with enumerations in C#. Defining and reading of the attribute is functionality built in NTestsRunner.

Define attribute

First step is to create class that extends System.Attribute. It has only one string property to hold the text in it. Text is passed in constructor. Note that this class is defined in System name space in order to have it by default skipping the need of importing name space you might not be aware of.

namespace System
{
	public class StringValue : Attribute
	{
		public string Value { get; private set; }

		public StringValue(string value)
		{
			Value = value;
		}
	}
}

Read the attribute

C# provides so called extension methods, a great way to add new functionality to existing type without creating new derived type. Reading of string value from enumeration element is done with GetStringValue extension method. With reflection all StringValue custom attributes of element are obtained. If some found text of first is returned. If not then string representation of element is returned.

using System.Reflection;

namespace System
{
	public static class ExtensionMethods
	{
		public static string GetStringValue(this Enum value)
		{
			string stringValue = value.ToString();
			Type type = value.GetType();
			FieldInfo fieldInfo = type.GetField(value.ToString());
			StringValue[] attrs = fieldInfo.
				GetCustomAttributes(typeof(StringValue), false) as StringValue[];
			if (attrs.Length > 0)
			{
				stringValue = attrs[0].Value;
			}
			return stringValue;
		}
	}
}

Apply to enumerations

Once StringValue class is ready it can be applied as attribute to any enumeration.

public enum Messages
{
	[StringValue("Problem occured, try again later")]
	ProblemOccured,
	[StringValue("Successfully done")]
	Success
}

Use in code

In code string value can be obtained from enumeration’s element with GetStringValue method.

Assert.AreEqual(Messages.ProblemOccured.GetStringValue(), App.MessageBox.GetText());

Conclusion

Using enumerations is mandatory to make readable and maintainable automation. Working effectively with enumerations will increase your value as automation specialist.

Read more...

NTestsRunner for functional automated tests

Last Updated on by

Post summary: NTestsRunner implementation details and features.

In previous post I’ve described unit testing frameworks and why they are not suitable for running functional automated tests. I introduced NTestsRunner – very simple runner that can be used for running your automation tests. This topic is dedicated for implementation details of the NTestsRunner.

Verifications

It is important in functional testing to be able to place several verification points in one test. For this purpose abstract class Verification is implemented. It has two properties to store more details about verification and time it was taken. Constructor receives comma separated string values. In case of zero strings are passed then result is empty string. If one string is passed then this is the result. If more than one string is added then first string is taken as formatting string and others are used to build up the result. Logic is similar to string.Format(String, Object[]) method.

public abstract class Verification
{
	public string Result { get; private set; }
	public DateTime ExecutedAt { get; private set; }

	public Verification(params object[] args)
	{
		...
	}
}

Passed or Failed

In automation test may have two conditions – passed or failed. This is why two concrete classes are extending Verification: VerificationPassed and VerificationFailed. They do not add any other functionality. Those classes use parent’s class constructor. This is example how to instantiate object from those classes:

string value = "number";
int number = 1;
Verification result =
	new VerificationFailed("This is formatting string {0} {1}. ",
		value,
		number);

Test case result

Test case is generally a set of conditions to verify whether given scenario works are per user requirements. In automation world test case is test method with several verification points inside. In NTestsRunner TestCaseResult is class representing the idea of a test case. It has properties for name, time to run and list of all verifications with count of passed and failed.

public class TestCaseResult
{
	public List<Verification> Verifications = new List<Verification>();
	public string Name { get; set; }
	public int VerificationsFailed { get; set; }
	public int VerificationsPassed { get; set; }
	public TimeSpan Time { get; set; }
}

Test plan result

TestPlanResult in NTestsRunner has nothing to do with test plan term from QA world. Here this is a representation of a test class with test methods inside. It has properties for name and time to run. Also there is list with all TestCaseResults, i.e. test methods in that class. There are counters for passed and failed test cases and also counters for all passed and failed verifications inside all TestCaseResults.

public class TestPlanResult
{
	public List<TestCaseResult> TestCases = new List<TestCaseResult>();
	public string Name { get; set; }
	public int TestCasesPassed { get; private set; }
	public int TestCasesFailed { get; private set; }
	public int VerificationsPassed { get; private set; }
	public int VerificationsFailed { get; private set; }
	public TimeSpan Time { get; private set; }

	public void Count()
	{
		...
	}
}

Class and method attributes

In order to make one class a test class it should have with [TestClass] attributes. To convert method to a test one it should have [TestMethod] attribute. Just the attribute is not enough though. Method should have special method signature. This is required by NTestsRunner.

Test method signature

In order to run without exception test method needs to conform to two rules:

  1. To have attribute [TestMethod]
  2. Method to receive parameter List verifications in its signature, i.e.
    [TestMethod]
    public void TestMethod1(List<Verification> verifications)
    

Configurations

Configurations can be found on NTestRunner home page.

Execution

Once object from NTestsRunner is instantiated and configured tests with Execute() method. In side this method all classes from calling assembly (the one that holds the tests) are taken. If TestsToExecute is configured then only those with name matching given values are taken. If no TestsToExecute is provided then all classes with attribute [TestClass] are taken. Methods from each class are taken by default in order of appearance in the class. If method has [TestClass] attribute then method is executed by passing List object to it. Inside the method Verifications are collected as list into TestCaseResult object. After method is run TestCaseResult is added to its parent TestPlanResult which is added to list with all results. In the end results are saved as XML and HTML.

Results in jUnit XML

In order to integrate with CI tools such as Jenkins or Bamboo results are exported to XML file after execution has finished. File is named Results.xml and is located in test results folder. XML format is implemented according to junit-4.xsd.

Results in HTML

Tests result are saved as HTML report for better readability. File is named Results.html and is located in test results folder.

Usage

In order to use NTestsRunner a console application project is needed. This project will hold test classes. As the one bellow. Take into consideration that this is very simplified usage pattern. In reality Page objects design pattern will be used. Page objects will make the verifications and return them.

[TestClass]
public class TestClass1
{
	[TestMethod]
	public void TestMethod1(List<Verification> verifications)
	{
		// Do some actions
		verifications.Add(new VerificationFailed("There is error"));
		// Do some actions
		verifications.Add(new VerificationPassed("Everythign is OK"));
	}
}

In its main method new instance of NTestsRunner is created. Configurations are done and test executions is started. It is that simple to use it.

class Program
{
	static void Main(string[] args)
	{
		NTestsRunnerSettings settings = new NTestsRunnerSettings();
		settings.TestResultsDir = @"C:\temp";
		settings.MaxTestCaseRuntimeMinutes = 2;
		settings.TestsToExecute.Add("TestClass1");
		settings.PreventScreenLock = true;

		NTestsRunner runner = new NTestsRunner(settings);
		runner.Execute();
	}
}

Pros and cons

NTestsRunner has its pros and cons.
Pros are:

  • Pretty easy to use
  • Open source and can be customised to you specific needs
  • Gives you ability to make several verifications in one test and in case of fail it doesn’t break current test method
  • Tests are stored into console application that can be easily run
  • Results are saved in jUnit XML for CI integration
  • Results are saved in HTML

Cons are:

  • Test methods should have specific signature
  • It is not easy to migrate existing tests to new format

Conclusions

This is pretty good tool for running functional automated tests. It is very easy to use and is made especially for running functional automated tests. You can definitely give it a try.

Read more...

Running functional automation tests

Last Updated on by

Post summary: Unit testing frameworks are not very suitable for running functional tests. NTestsRunner is an alternative way of running functional automated tests.

Unit testing

Unit testing is focused on testing code on low level. Methods, sets of methods or modules are being tested by writing test code which invokes those methods with specific arguments. In unit testing all external dependencies (database, file system, network, etc.) are removed. Those resources are simulated in unit tests by using so called mock objects. Mock objects are controlled by tests designer and have predictive behaviour. Running piece of code which doesn’t have external dependencies happens almost immediately. So unit tests are executed for very low amount of time. It is considered set of unit tests taking longer than 5 minutes is not well designed. Unit tests are strictly focused. One test tests only one condition. Each test is not related in anyhow to other tests.

Unit testing frameworks

Unit testing frameworks conform to unit tests purpose and design. Tests should not depend on each other. For this reason unit testing frameworks execute tests in random order (xUnit.net or MS Unit Testing Framework), other like NUnit in alphabetic order of tests method names. Checking for given conditions are done with assertions. If one assert fails current test execution is stopped and test is marked as failed.

Functional testing

Functional testing is focused on ensuring that software product works as per user requirements. Real life software has external dependencies. Most software products has some kind of user interface. Automation tests are focused on verifying that UI works correctly. Transferring and rendering data to UI takes time, database operations take time, file and network operations also take time, etc. In general functional tests are more complex and take much longer to execute. In order to be efficient checking of several conditions are defined in each test. Functional tests can be manual and automated. In current post when I mention functional tests I mean only automated.

Requirement for running functional test – many verifications

In a perfect situation functional one test should verify only one condition. In reality because of too many external dependencies time for tests execution is large. An time matters. In order to shorten this time we make several checks in one test. For e.g. in e-commerce web site order is placed. We verify order confirmation page that there is order number, that address is same used during checkout, that user’s email is correct. We also can verify inside user’s email box that received mail is correct. We can check in database for some properties of the order. If we have to do one order for each check tests will take significantly long time. To be efficient we do all checks with one order in one test case. We need a framework which allows you to have multiple verifications in one test. Further more if verification is failed test execution should continue.

Requirement for running functional test – controlled sequence

What is really good to be avoided but sometimes cannot is test dependency. Sometimes one tests needs another to have done something before continuing. As I said this should be avoided but in order to be efficient you should make trade off between good tests design and time to execute. For e.g. you may want to cancel and then refund order placed in e-commerce web site. Generally it is best to place new order for this test but if placing of order requires too much time then an option is to reuse already existing order from previous test. We need to be able to control test case execution order.

NTestsRunner

In order to have control over tests and use many verifications I’ve created NTestsRunner. Its code is in GitHub NTestsRunner repository. This is .NET library. You create console application with your tests and use the library within. Tests are annotated is similar fashion as with unit testing frameworks. Tests are executed in sequential order. There could be many verifications. Results are saved as HTML and XML in jUnit format. NTests runner is described in more details in this post NTestsRunner for functional automated tests.

Read more...