Upcoming Training"Mastering Symfony2 Forms" in Barcelona, 22 May 2015
Although most modern literature agrees that testing is essential for successful software projects, testing has to be done right. Several bad habits can be identified that not only negatively affect testing performance, but also reduce your development speed. One of the first mistakes people make when they first discover testing is trying to reach 100% code coverage. I was no exception.
The more tests you have for your application, the more difficult it becomes to introduce changes. Everytime you change an existing feature, you have to change at least one test. Everytime you make a refactoring of your project’s architecture (which you should do frequently), you most likely need to adapt multiple tests. As a rule of thumb you can say that the more tests you have, the more difficult it becomes to make changes in your code.
We know that not to write tests is not a solution. Lots of tests, on the other hand, are bad for maintainability. Instead, try to keep a certain test balance by testing only what is important. The following section shows you one possible approach for identifying important tests called risk-based testing.
The purpose of testing is to avoid regressions in your software resulting of its complexity. These potential regressions can be expressed as risks ranked by their probability and severity. The probability expresses how likely the regression is to occur. The severity expresses how severely the regression would affect the functionality of your application. Both can be expressed in numbers, for instance in a range of 1 to 10 with 10 being the highest severity/probability. Look at the following table for example:
|Risk||Probability (P)||Severity (S)||Total (PxS)|
The table contains risks for two fictitious methods
calculatePrice(). Imagine that the first of these methods only returns an internal property
$group. The probability of someone breaking this code by accident is near to non-existent. Assume further that the group is only used for display purposes, so the severity of a broken code is not very high either.
The second method,
calculatePrice(), uses an internal algorithm to calculate the price of a product. Because it depends on an external object
$product, and because the algorithm is non-trivial, the probability of breaking the code is rather high. Even higher though is the severity of a broken functionality – what could be worse than wrongly calculated prices.
Is it worth to unit-test
getGroup()? Probably not. We calculated an overall total of 3, which (in our scale from 1 to 100) is really not that impressive. Is it, on the other hand, worth to test
calculatePrice()? Judge for yourself.
Obviously I selected two extreme examples here. Also you will almost never calculate probabilities and severities for testing single methods like I did it here. But you should try to develop a feeling for what is worth to be tested and what is not.
Now that you have identified which risks you should cover in your test suite, the question is to what extent you should do so. Look at the following to approaches for testing
Approach 1: The fair-enough approach
$t = new LimeTest(); // @Test: calculatePrice() multiplies the price with the amount $product = new Product(); $product->price = 100; $product->amount = 2; $t->is($calculator->calculatePrice($product), 200);
Approach 2: The safe approach
$t = new LimeTest(); // @Before $product = new Product(); $product->price = 100; // @Test: calculatePrice() multiplies the price with the amount (amount == 1) $product->amount = 1; $t->is($calculator->calculatePrice($product), 100); // @Test: calculatePrice() multiplies the price with the amount (amount == 2) $product->amount = 2; $t->is($calculator->calculatePrice($product), 200);
The first approach assumes that testing whether the result equals price multiplied by amount is enough. The test can obviously not check whether the developer has hard-coded a fixed return value of 200 or uses a fixed amount of 2. It expects the developer to be reasonable.
The second approach takes care of these eventualities by writing another test to make sure that changed amounts result in changed outputs. But even now, the developer could write a simple switch statement in
calculatePrice() and return either 100 or 200, based on the amount. The test will not notice.
The conclusion is that you can never test your code for all eventualities. However complicated your test may be, there will always be a way to fake the implementation. But is it likely that developers will fake it? Probably not. This being said, the best approach in my opinion is approach 1 because it keeps effort required to adapt or change the test to a minimum.
Don’t try to achieve 100% test coverage. Identify the most important risks, and test for those. Don’t test for all eventualities either. Always keep in mind that the developers working on your code are somewhat reasonable people. If they aren’t, you are probably better off without them anyway.