Large Scale Project Lessons - Long Tails and their Impact on Iterative Development
One of the challenges of effectively implementing a large scale, enterprise iterative development program is that there is little in the way of literature on how to effectively scale iterative projects into a program. Most of the literature focuses on the 'relatively' well defined issue of managing a single software development project. Looking at Scrum, XP, DSDM and even RUP none of these approaches really address the challenges of implementing iterative development techniques across a broad number of software development projects that operate as part of a program.
While there is a good program management plug-in for RUP and Ambler's Enterprise Unified Process extends RUP beyond its project phases, there is nothing readily available that provides a lightweight and highly engaged program management office which applies some of the same principles of iterative development to a Program Office. Of course there is a well established body of knowledge around Program or Programme Management but these bodies of knowledge emphasize a program office focused more on administrative and financial oversight of a number of projects.
Large enterprise deployments to tens of thousands of users have 'long tails'. These tails represent the time and effort that goes into deploying an application into multiple separate environments for testing, training and ultimately transitioning into Production. RUP uses the Transition phsae to capture some of these activities but most large organizations have their own specific processes including 'black-out' periods, SOX and other compliance needs and the need to coordinates across multiple departments. Where many large organizations struggle with adopting iterative development processes is when they begin to examine the practicality of moving a release through the long tail of their deployment process. Since this tail often has an insatiable desire for detailed requirements (to justify purchasing expensive hardware or to satisfy Service Level Agreements with outsourced support organizations) it becomes enticing for the organization to move back to its waterfall approach (or it redefined iterative development as simply applying to the RUP Construction phase).
Tackling this challenge requires a thorough understanding of the organizational dynamics of large scale organizations and a willingness to educate the often independent groups which support the 'tail' of the deployment process. The first step is to adopt a more program centric view of how multiple projects can work together leveraging an approach such as the System of Systems pattern. The next is to develop a communication and awareness plan which helps support groups in the 'tail' better understand how iterative development practices work and what they can do to support this approach. I'll try and write up some specific observations of how to tackle these program level iterative development issues in the weeks ahead.