Thursday, March 01, 2007

How good is enough

This is the title of a recent column on JDJ magazine by Nigel Cheshire, and almost in the same month ACM published its own issue of special edition on software quality where number of papers shed their thoughts on the same question “How good is enough”; a month apart, in IEEE Software journal the coauthor of Testing Extreme Programming Lisa Crispin published her article “Driving Software Quality: How Test-Driven Development Impacts Software Quality” related to this topic as well. Why all of sudden all this attention to this seemingly straightforward question? The answer is simple because most of us do not know the answer.

How many times as a manager you have seen project produced higher than normal defect counts in the QA stage after many late night pizza deliveries and weekend team lunches and finally finished implementation on time. Just when you thought your job is done, it actually just got started. The bug fixing and rework stage actually turn out to be even longer and more painful than the implementation. Many projects eventually get bogged down and lead to significant schedule and budget overrun even after they managed to initially achieve the scheduled milestone for construction phase. What went wrong? Why do we keep repeating this self destructive pattern over and over again? Nigel is his article proposed a very prominent viewpoint in which he points out that one of the major contributing factor to this problem is the over-simplified concentration on using project schedule as the dominate metric to measure the project progress. Traditionally due to the relative easiness for measuring project schedule, it is widely adopted in the industry as a, and some times the only, metric for software project. However when you use the schedule alone without the help from other quality related metrics, it does not really tell you anything other than how many days have been passed since you project started. Without the context information from quality-aware metrics when a developer tells you that “I finished the implementation and checked in all the source code on time” what it really means is “I really have no idea if this code will work or not” so this could be well the end of the construction or the start of it. Nigel in his article comes to the conclusion that “What gets measured gets managed”, but he did not get down to the details on how you suppose to measure the true quality, cost, and progress.

In Agile process model, we throw away the pure construction milestones because we understand the pure construction without test case verification base on the user requirement does not reflect the progress at all, that’s why Lisa in her article on IEEE Software journal focused on how Test-Driven Development and Design can give you a hand on this. In her article, she basically comes to the plain ugly truth that “Anything not covered in a test isn’t going to be there” so the only way to guarantee you deliver what your customer has asked for is by creating test cases for every single requirement, and only by satisfying the test cases you move the project forward. This is especially crucial in Agile project, since you have to constantly make “How good is enough” decisions, no only on overall quality alone but for almost everything else such as your just good enough design and just in time implementation. With the help of test driven requirement it makes our life much easier to answer the question, but writing testable or quantifiable requirement is not just about some extra description in our story card or requirement document, however it requires a fundamental mindset shift from the traditional way of presenting and organizing requirement. But if you are interested in this topic Nail Maiden in his back-to-back article on requirement quantification “Improve Your Requirements: Quantify Them” and “My Requirements? Well, That Depends” provided some very interesting idea and material for further reading, and while you exploring all these new and exciting technique and theory in the search of the answer of this million-dollar question just don’t forget this century old wisdom:

Perfection is the enemy of the good.

Gustave Flaubert
French realist novelist (1821 - 1880)