What about code coverage

Mar 19, 2015

Code coverage is a useful metric to check what part of the code your tests are exercising. Is really depends on the tools to be used for gathering code coverage metrics but in general, they work in similar fashion.

How it works

One approach is to instrument application under test. Instrumentation is to modify original executables by adding metadata so code coverage tool is able to understand which line of code is being executed. Another option is to run the application through code coverage tool. In both ways, once the application is running tests are being executed.

What tests to run

Tests that can be run to measure code coverage can be unit tests, functional automation tests or even manual tests. It doesn't matter the important part is to see how our tests are impacting application under test.

Results

Once tests are finished code coverage metrics are obtained from code coverage tool. To have detailed information most of the tools take the original source code and generate a visual report with color information which line is executed and which not. There are different levels of coverage, method, line, code block, etc. I personally prefer lines and when speaking further I'll mean lines of code executed.

Benefits

Code coverage information is equally useful for QA and developers. QA analyze code that has not being executed during tests to identify what tests conditions they are missing and improve their tests. Developers analyze the results to identify and remove dead or unused code.

When to do it

Code coverage of unit tests is good to be done on each build. Those can be scheduled in continuous integration jobs and can be run unattended. Code coverage on automated or manual tests is more like a nice to have activity. We can live with or without it. It is useful for big matured products where there are automation test suites. You can also do them against your tests code in order to optimize it. Removing dead code optimizes the product and makes its maintenance easier. I would say doing it too often should be avoided. Everything depends on context but for me best is once or twice a year.

What does code coverage percentage mean

In one word - nothing. You may have 30% of code coverage but to cover most important user functionality and bug rate to be relatively low and you may have 90% with dummy tests made especially to exercise some code without the idea of actually testing it. I was lucky to work on a project where developers keep the code tidy and clean and my tests easily accomplished 81% just by verifying all user requirements. I may say 80-85% is the maximum you may get.

Pitfalls

I would not recommend putting code coverage as an important measurement or key performance indicator (KPI) in your testing strategy. Doing so and making people try to increase the code coverage will in most cases result in dummy tests made especially this percentage to go up. Code coverage is in most cases insignificant aspect of your testing strategy.

How to do it

Practical tutorials on how to do code coverage with Java, C#, and JavaScript can be found on my blog:

Conclusion

Code coverage is an interesting aspect of testing. It might enhance your tests if done wisely or it can ruin them. Remember tests scenarios should be extracted from user requirements and features. Code coverage data should be used to see if you have some blind spots reading the requirements. It can help developers remove dead or unused code.

Tags:

C#Code coverage Java Non-functional