Running functional automation tests

Last Updated on by

Post summary: Unit testing frameworks are not very suitable for running functional tests. NTestsRunner is an alternative way of running functional automated tests.

Unit testing

Unit testing is focused on testing code on low level. Methods, sets of methods or modules are being tested by writing test code which invokes those methods with specific arguments. In unit testing all external dependencies (database, file system, network, etc.) are removed. Those resources are simulated in unit tests by using so called mock objects. Mock objects are controlled by tests designer and have predictive behaviour. Running piece of code which doesn’t have external dependencies happens almost immediately. So unit tests are executed for very low amount of time. It is considered set of unit tests taking longer than 5 minutes is not well designed. Unit tests are strictly focused. One test tests only one condition. Each test is not related in anyhow to other tests.

Unit testing frameworks

Unit testing frameworks conform to unit tests purpose and design. Tests should not depend on each other. For this reason unit testing frameworks execute tests in random order (xUnit.net or MS Unit Testing Framework), other like NUnit in alphabetic order of tests method names. Checking for given conditions are done with assertions. If one assert fails current test execution is stopped and test is marked as failed.

Functional testing

Functional testing is focused on ensuring that software product works as per user requirements. Real life software has external dependencies. Most software products has some kind of user interface. Automation tests are focused on verifying that UI works correctly. Transferring and rendering data to UI takes time, database operations take time, file and network operations also take time, etc. In general functional tests are more complex and take much longer to execute. In order to be efficient checking of several conditions are defined in each test. Functional tests can be manual and automated. In current post when I mention functional tests I mean only automated.

Requirement for running functional test – many verifications

In a perfect situation functional one test should verify only one condition. In reality because of too many external dependencies time for tests execution is large. An time matters. In order to shorten this time we make several checks in one test. For e.g. in e-commerce web site order is placed. We verify order confirmation page that there is order number, that address is same used during checkout, that user’s email is correct. We also can verify inside user’s email box that received mail is correct. We can check in database for some properties of the order. If we have to do one order for each check tests will take significantly long time. To be efficient we do all checks with one order in one test case. We need a framework which allows you to have multiple verifications in one test. Further more if verification is failed test execution should continue.

Requirement for running functional test – controlled sequence

What is really good to be avoided but sometimes cannot is test dependency. Sometimes one tests needs another to have done something before continuing. As I said this should be avoided but in order to be efficient you should make trade off between good tests design and time to execute. For e.g. you may want to cancel and then refund order placed in e-commerce web site. Generally it is best to place new order for this test but if placing of order requires too much time then an option is to reuse already existing order from previous test. We need to be able to control test case execution order.

NTestsRunner

In order to have control over tests and use many verifications I’ve created NTestsRunner. Its code is in GitHub NTestsRunner repository. This is .NET library. You create console application with your tests and use the library within. Tests are annotated is similar fashion as with unit testing frameworks. Tests are executed in sequential order. There could be many verifications. Results are saved as HTML and XML in jUnit format. NTests runner is described in more details in this post NTestsRunner for functional automated tests.

If you find this post useful, please share it to reach more people. Sharing is caring!
Share on FacebookShare on LinkedInTweet about this on TwitterShare on Google+Email this to someone