Extreme Programming works by creating rich channels of communication between programmers and customers, by requiring customers to select the most valuable features, by mandating ultra-simple system design, by delivering system increments early and often, and by testing the system exhaustively throughout development.

XP incorporates customer feedback into the system throughout development. It uses changing system requirements to evolve the system into what customers really want.

XP programmers work in tightly-knit teams that practice TDD, program in pairs, share all the source code, keep the design ultra-simple, refactor mercilessly, and integrate continuously. A good XP team can produce an astonishing amount of robust software per week. XP also requires courage, high discipline, and accountability of all teammembers.

So how do the practices work, in general terms? Agile methods like XP were designed by industry veterans who, after suffering project failures time after time, finally devised practices that prevent that failure. We've summarized the key practices below.

Too many projects try to eat the whole cow in one bite, and choke to death. Studies such as the famous Standish CHAOS Report show conclusively that the longer the project, and the more function points it addresses, the likelier it is to fail. The decline in failure rates from 31% in the 1994 CHAOS report to 25% in the 2000 report are largely credited to smaller, shorter projects and smaller teams. Another large case study "showed that defects rise non-linearly as project size grows." Yet another study showed that the percentage of useful code increased as project size decreased.

In short, there is plenty of evidence to support dividing projects into smaller, more manageable iterations.

Several studies failed to correlate project success with waterfall-style predictive planning, in which scope is nailed down up-front. No matter how hard we try to predict detailed project scope at the outset, it is not possible to do. Instead, in XP you plan and build a discrete system increment in each closed-end iteration, letting its scope slip but not the date. At the end of the iteration, you compare metrics on predicted progress vs. actual progress, and use any discrepancy to plan the next iteration. You use the data from the first few iterations to arrive at a concrete estimate for final system scope.

Two studies showed large rates of requirements change during projects. Another study found that iterative, incremental delivery "appears to contribute to both a lower defect rate and higher productivity."

The best practice is to iterate and deliver incrementally, treating each iteration as a closed-end "mini-project," including complete requirements, design, coding, integration, testing, and internal delivery. On the iteration deadline, deliver the (fully-tested, fully-integrated) system thus far to internal stakeholders. Solicit their feedback on that work, and fold that feedback into the plan for the next iteration.

Craig Larman's latest book quotes studies showing that "broadly, defect reduction comes from avoiding defects before they occur and from feedback" such as tests and evaluations. One study showed that as the time lag between coding and testing decreased, defect rates likewise decreased.

Test-Driven Development, together with refactoring, effectively eliminate the sad tradition of open-ended debugging, replacing it with "pre-bugging": finding and eliminating bugs at inception. The practice of continuous integration is also credited by Larman and others with lowering defect rates and increasing productivity.

Several studies correlate project failure with failure to tackle integration soon enough. The best practice is to address high-risk and high-business-value issues as early as possible. For example, program and test the core architecture, and integrate its major components, in early iterations. This way when you encounter big unanticipated problems, you still have time to address them.

In XP, you rank all features by business value. In each iteration, you plan to build the highest-priority remaining features. The finished system consists only of high-ROI features. And should the project be cancelled early, the system is robust and fully integrated, and may contain the enough business-critical functionality to pay back your development investment so far.

Responding quickly to unanticipated requirements change requires keeping the system extensible. In turn, this requires understanding and applying the principles and patterns of object-oriented design. Don't let unhealthy dependencies develop between components. Continually refactor the design, so that it can accommodate arbitrary change at any point in its lifecycle.

Agile practices like XP are well worth the trouble to learn and adopt. If you decide to go Agile, we can help you with training, mentoring, and coaching. For more info, call us at 902.681.1640, or email us.

For even more details on how specific XP practices work, see this writeup.