Post summary: Introduction to mutation testing and examples with PITest tool for Java.
Mutation testing
Mutation testing is a form of white-box testing. It is used to design new unit tests and evaluate the quality of the existing ones. Mutation testing involves modifying a program code in small ways, based on well-defined mutation operators that either mimic typical programming errors (such as using the wrong operator or variable name) or force the creation of valuable tests (such as dividing each expression by zero). Each mutated version is called a mutant. Existing unit tests are run against this mutant. If some unit test fails then mutant is killed. If no unit test fails then mutant survived. Test suites are measured by the percentage of mutants that they kill. New tests can be designed to kill additional mutants. The purpose of mutation testing is to help the tester develop effective tests or locate weaknesses in the test data used in the existing tests.
It is not very often when I get surprised by discovering new testing technique I’ve never heard about, so I must give credits to Alexander Todorov since I learned this one from his presentation.
PITest
Mutation testing can be done manually by changing program code and running the tests, but this is not really effective and can lead to serious problems where you commit mutated code by mistake. Most effective and recommended way of doing mutation testing is by using tools. PITest is a tool for mutation testing in Java. It seems to be fast growing and has a big community.
Integrate PITest
Examples given in current post can be found in GitHub sample-dropwizard-rest-stub repository. It is very easy to use PITest. The first step is to add it to pom.xml file:
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.1.10</version>
<configuration>
<targetClasses>
<param>com.automationrhapsody.reststub.persistence*</param>
</targetClasses>
<targetTests>
<param>com.automationrhapsody.reststub.persistence*</param>
</targetTests>
</configuration>
</plugin>
The example given above is the most simple one. PITest provides various configurations very well described into PITest Maven Quick Start page.
Run PITest
Once configured it can be run with: mvn org.pitest:pitest-maven:mutationCoverage or if you want to ensure clean build every time: mvn clean test org.pitest:pitest-maven:mutationCoverage
PITest results
PITest provides very understandable reports. Along with the line code coverage, it measures the mutation coverage. There are statistics on package level.
PersonDB is the only class that has been uni tested. It has 91% line coverage and 1 not killed mutation. Opening the PersonDB class gives more details what is not covered by tests and what the mutation is:
PITest has negated the condition on line 44, making the mutated code to be: PERSONS.get(person.getId()) == null. Unit tests had passed although this mutation. Full reports can be found in PITest report example.
Action PITest results
Current results indicate that unit tests are not good enough because of the survived mutation. They are also insufficient as one line of code is not tested at all. The first improvement is to kill the mutation by change line 37 of PersonDBTest.java from PersonDB.save(person); to assertEquals(“Added Person with id=11”, PersonDB.save(person)); and PITest is run again then results show all mutations are killed.
Still, there is one line of code not covered. This will require adding a new unit test to cover the update person functionality.
PITest performance
Doing a mutation testing requires significant resources to run a large amount of unit tests. Examples given above work very fast, but they are far away from the reality. I was curious how this works on a real project so I run it on one which is relatively small one. It has very good unit tests though with 95% line coverage – 2291 out of 2402 lines. Still, PITest found only 90% mutation coverage (849/940). 51 out of 940 mutations that survived and 40 with no unit test coverage which gives a room for improvement. The total run took 3 mins 11 secs. See results below:
PIT >> INFO : Found 464 tests
PIT >> INFO : Calculated coverage in 22 seconds.
PIT >> INFO : Created 132 mutation test units
- Timings
==================================================================
> scan classpath : < 1 second
> coverage and dependency analysis : 22 seconds
> build mutation tests : < 1 second
> run mutation analysis : 2 minutes and 48 seconds
--------------------------------------------------------------------------------
> Total : 3 minutes and 11 seconds
--------------------------------------------------------------------------------
==================================================================
- Statistics
==================================================================
>> Generated 940 mutations Killed 849 (90%)
>> Ran 2992 tests (3.18 tests per mutation)
==================================================================
- Mutators
==================================================================
> org.pitest.mutationtest.engine.gregor.mutators.ConditionalsBoundaryMutator
>> Generated 18 Killed 9 (50%)
> KILLED 7 SURVIVED 7 TIMED_OUT 2 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 2
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.VoidMethodCallMutator
>> Generated 115 Killed 96 (83%)
> KILLED 81 SURVIVED 12 TIMED_OUT 15 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 7
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator
>> Generated 503 Killed 465 (92%)
> KILLED 432 SURVIVED 16 TIMED_OUT 33 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 22
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.MathMutator
>> Generated 10 Killed 9 (90%)
> KILLED 8 SURVIVED 0 TIMED_OUT 1 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 1
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator
>> Generated 294 Killed 270 (92%)
> KILLED 254 SURVIVED 16 TIMED_OUT 15 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 1
> NO_COVERAGE 8
--------------------------------------------------------------------------------
Case study
I used PITest on production project written on Java 8 extensively using Stream APIs and Lambda expressions. The initial run of 606 existing test cases gave 90% mutation coverage (846/940) and 95% line coverage (2291/2402).
Note that PITest calculates in statistic given above that existing tests are 464. This is because some of them are data driven tests and JUnit calculates the total number of 606 because every data row is counted as a test. Understand how to make JUnit data-driven tests in Data driven testing with JUnit parameterized tests post.
After analysis and adding new tests total test cases number was increased to 654 which is almost 8% increase. PITest run shows 97% mutation coverage (911/939) and 97% line coverage (2332/2403). During the analysis, no bugs in code were found.
Conclusion
Mutation testing is a good additional technique to make your unit tests better. It should not be the primary technique though as tests will be written just to kill the mutations instead of actually testing the functionality. In projects with well-written unit tests mutation testing does not bring much of a value, but still, it is a very good addition to your testing arsenal. PITest is a very good tool to do mutation testing in Java I would say to give it a try.
