Navigation

Although most modern literature agrees that testing is essential for successful software projects, testing has to be done right. Several bad habits can be identified that not only negatively affect testing performance, but also reduce your development speed. One of the first mistakes people make when they first discover testing is trying to reach 100% code coverage. I was no exception.

Test Balance

The more tests you have for your application, the more difficult it becomes to introduce changes. Everytime you change an existing feature, you have to change at least one test. Everytime you make a refactoring of your project’s architecture (which you should do frequently), you most likely need to adapt multiple tests. As a rule of thumb you can say that the more tests you have, the more difficult it becomes to make changes in your code.

We know that not to write tests is not a solution. Lots of tests, on the other hand, are bad for maintainability. Instead, try to keep a certain test balance by testing only what is important. The following section shows you one possible approach for identifying important tests called risk-based testing.

Risk-Based Testing

The purpose of testing is to avoid regressions in your software resulting of its complexity. These potential regressions can be expressed as risks ranked by their probability and severity. The probability expresses how likely the regression is to occur. The severity expresses how severely the regression would affect the functionality of your application. Both can be expressed in numbers, for instance in a range of 1 to 10 with 10 being the highest severity/probability. Look at the following table for example:

Risk Probability (P) Severity (S) Total (PxS)
getGroup() returns a wrong object 1 3 3
calculatePrice($product) uses a wrong algorithm 7 8 56

The table contains risks for two fictitious methods getGroup() and calculatePrice(). Imagine that the first of these methods only returns an internal property $group. The probability of someone breaking this code by accident is near to non-existent. Assume further that the group is only used for display purposes, so the severity of a broken code is not very high either.

The second method, calculatePrice(), uses an internal algorithm to calculate the price of a product. Because it depends on an external object $product, and because the algorithm is non-trivial, the probability of breaking the code is rather high. Even higher though is the severity of a broken functionality – what could be worse than wrongly calculated prices.

Is it worth to unit-test getGroup()? Probably not. We calculated an overall total of 3, which (in our scale from 1 to 100) is really not that impressive. Is it, on the other hand, worth to test calculatePrice()? Judge for yourself.

Obviously I selected two extreme examples here. Also you will almost never calculate probabilities and severities for testing single methods like I did it here. But you should try to develop a feeling for what is worth to be tested and what is not.

Test Complexity

Now that you have identified which risks you should cover in your test suite, the question is to what extent you should do so. Look at the following to approaches for testing calculatePrice():

Approach 1: The fair-enough approach

$t = new LimeTest();
 
// @Test: calculatePrice() multiplies the price with the amount
 
  $product = new Product();
  $product->price = 100;
  $product->amount = 2;
  $t->is($calculator->calculatePrice($product), 200);

Approach 2: The safe approach

$t = new LimeTest();
 
// @Before
 
  $product = new Product();
  $product->price = 100;
 
// @Test: calculatePrice() multiplies the price with the amount (amount == 1)
 
  $product->amount = 1;
  $t->is($calculator->calculatePrice($product), 100);
 
// @Test: calculatePrice() multiplies the price with the amount (amount == 2)
 
  $product->amount = 2;
  $t->is($calculator->calculatePrice($product), 200);

The first approach assumes that testing whether the result equals price multiplied by amount is enough. The test can obviously not check whether the developer has hard-coded a fixed return value of 200 or uses a fixed amount of 2. It expects the developer to be reasonable.

The second approach takes care of these eventualities by writing another test to make sure that changed amounts result in changed outputs. But even now, the developer could write a simple switch statement in calculatePrice() and return either 100 or 200, based on the amount. The test will not notice.

The conclusion is that you can never test your code for all eventualities. However complicated your test may be, there will always be a way to fake the implementation. But is it likely that developers will fake it? Probably not. This being said, the best approach in my opinion is approach 1 because it keeps effort required to adapt or change the test to a minimum.

Conclusion

Don’t try to achieve 100% test coverage. Identify the most important risks, and test for those. Don’t test for all eventualities either. Always keep in mind that the developers working on your code are somewhat reasonable people. If they aren’t, you are probably better off without them anyway.

Posted Monday, March 15th, 2010 at 09:00
Written by: | Filed Under Category: Best Practices
You can leave a response, or trackback from your own site.

5

Responses to “Why Not To Want 100% Code Coverage”

Social comments and analytics for this post…

This post was mentioned on Twitter by stephenmelrose: RT @webmozart: New blog post: Why Not To Want 100% Code Coverage http://bit.ly/cL40SS #symfony #testing…

One thing I like as a guideline is to always test for the working expected scenario, and also try to test for the error condition, and ensure the code breaks properly. The definition of properly is obviously context dependent. Sometimes an exception is good, sometimes returning null is better.

It’s important to test for potential failures, because eventually a non-expected input will get through and that day you’d rather have your code fail gracefully than explode in the user’s face or worse cause a security breach.

Very interesting read! Thanks a lot for elaborating on this topic.

[…] ein paar Tagen hat Bernhard in seinem Blog einen weiteren sehr anschaulichen Artikel zum Thema Testabdeckung geschrieben. Er liefert darin eine schlüssige Begründung warum eine hundertprozentige […]

Writing tests for test coverage is just the tip of the iceberg.

This happens usually when people write their tests last and not before code is created. There’s no clear expectation and code tends to work fine and in the end people abuse software metrics to meet a certain goal.
I wouldn’t say that TDD or BDD is the way to go but these tend to solve these issues for me.

In my opinion breaking tests are not a bad thing. Of course they require time to fix, but they also show you why something broke.

Could be a feature, bug or generally code which needs to be refactored. All of the above could be either in your application or of course in your test suite. That’s not a problem though because it helps me evolve.

Tests are a tradeoff but you get security in return. If time is an issue in general it comes down to how you “sell” testing to either yourself or your client/project manager etc..

We actually test large portions of our code-base and we also frequently integrate. I run my test suite and push code live.

There’s usually no, “maybe this works”. And if stuff still breaks, then I have to write another test and improve my test suite because it obviously didn’t catch whatever I just introduced.

In the end it’s a matter of finding the right balance and as you mentioned identifying the necessary pieces to test.

E.g. I rarely write controller tests because they (IMHO) primarily test framework code. If I end up finding a bug in the framework I am using, I’ll write one test to make sure it’s covered (and doesn’t come back), but that’s it.

Leave a Reply

 

Additional Resources