Finally, abundant proof in one book that the traditional waterfall approach is a terrible way to manage software projects, and is therefore slowly being displaced by agile and iterative approaches. In Agile and Iterative Development: a Manager's Guide, renowned consultant and author Craig Larman does a devastatingly thorough job of debunking waterfall once and for all.

Larman cogently and painstakingly explains how several of waterfall's practices have been conclusively linked to project failures, and how, on the other hand, the practices of Agile and iterative methods like Scrum and XP reduce project risk. He summarizes research findings encompassing thousands of projects, and quotes the supporting opinions of standards bodies and industry thought-leaders. The net effect is compelling, to say the least.

As Larman shows, one of the biggest problems with waterfall is that it pushes most of the "high-risk and difficult" elements toward the end of the project. It end-loads the pain.

We've all seen this before. Everything looks great for the first few weeks and months of a project. The PERT and Gantt charts all seem to indicate rosy project health. Then in the project's last few weeks or months, management is shocked to discover that the features are wrong or unusable, or the architecture won't work, or the components cannot be integrated, or the system is full of defects. At that point, options are sad and few: extend the deadline, bust the budget, rework extensively -- or cut losses and cancel. Such scenarios are the norm, not the exception.

Larman quotes the joke about the guy who fell off a cliff:

As he was hurtling down, someone yelled, "How are you doing?" The guy replied, "So far, so good!"

The numbers are overwhelming, and somewhat depressing. A study in the UK shows that of 1,027 projects, only 13% did not fail, and waterfall-style scope management was the "single largest contributing factor for failure, being cited in 82% of the projects as the number one problem." A 1995 study of over $37 billion USD worth of US Defense Department projects concluded that "46% of the systems so egregiously did not meet the real needs (although they met the specifications) that they were never successfully used, and another 20% required extensive rework" to be usable.

In "another study of 6,700 projects, it was found that four out of the five key factors contributing to project failure were associated with and aggravated by the waterfall model, including inability to deal with changing requirements, and problems with late integration." Yet another study of over 400 waterfall projects reported that only 10% of the developed code was actually deployed, and of that, only 20% was used.

Larman also quotes the opinions of Harlan Mills, Fred Brooks, Barry Boehm, James Martin, Tom DeMarco, Ed Yourdon, and others. He quotes recent findings by several standards bodies around the world. The concensus is overwhelming: waterfall doesn't work, and Agile and iterative approaches do.

For our own summary of the specific Agile practices that are most associated with project success, see the following article.

Many large and small organizations worldwide are formally adopting agile and iterative practices. Informally, this experimentation has been going on for decades. Larman points out that iterative process design goes back more than 30 years to Tom Gilb's Evo method. Even the man whose original article gave waterfall its momentum, Winston Royce, apparently regretted later having written it. The late Royce, according to his son, was actually a misunderstood advocate of iterative and incremental development. The research seems to indicate that a shrinking percentage of projects are using waterfall "by the book" anymore.

If you are an Agile skeptic, this book may rattle your conviction. If you are fence-sitter, it may convince you. And if you already have Agile fire in the belly, then certainly this book will stoke that fire. After reading it, I am left wondering how intelligent, experienced software development management can justify the continued use of a process that has wasted so much money and caused so much pain. And I am left with no doubt that agile methods constitute an unstoppable trend in software project management. There is, indeed, a revolution going on.

-- Patrick Wilson-Welsh, Adaption Software     11/18/2003

[More of Adaption's Articles.]