by Bryan Campbell and Dr. Glenn Ray

In October 2000, the customer, FederalRe, Inc. (a fictitious name to protect the client’s proprietary information), initiated the development of a new software application known as the Property Reinsurance System (PRS).  The application was intended to be an object oriented, web based application, which could provide real-time property re-insurance quotes to FederalRe customers.  As part of the development process, FederalRe required that a specific software development methodology, the Rational Unified Process (RUP), be used throughout the lifecycle of the project.  As the consulting company responsible for delivering the project under a fixed time and price contract, we recommended that RUP be complemented by a lightweight methodology known as eXtreme Programming (XP). 

PRS was intended to replace an existing client-server application that had been initially developed in the early 1980s for a specific customer.  Subsequently, this system was customized and marketed to other clients.  Currently, 130 insurers use the system generating more than $40 million (U.S.) in annual revenue to Federal Re.

The project had several stated objectives and constraints:

1.   Develop a new n-tiered, web services based version of PRS. Web services refers to exposing core application services through an XML interface using an HTTP transport layer.

2.   Develop the system using a Microsoft toolset (Visual Basic 6.0, ADO 2.5 and ASP 3.0), as required by the strategic architectural direction of FederalRe’s parent organization.

3.      Apply an iterative development process to the application based on the Rational Unified Process.

4.     Develop new functionality, not present in the current client-server version of PRS, and examine re-engineering opportunities in the process of developing the system.

Through the Rational Unified Process the project was delivered in a series of iterations over four phases (Inception, Elaboration, Construction and Transition). The project was a $3 Million (U.S.) initiative with a twelve month development period and involved a team of 11 full-time members (one project manager, two system analyst/architects and eight developers). 

The Rational Unified Process  (RUP) undeniably ranks among the leading software development processes available today. RUP evolved from the pioneering work of the ‘Three Amigos’ (Grady Booch, James Rumbaugh and Ivar Jacobson) who aligned their individual efforts to develop an object oriented, visual modeling process into a comprehensive approach with a common language, the Universal Modeling Language (UML) notation.  Currently supported by Rational Software Inc., the methodology is distributed on a CD in HTML format and is tightly coupled with some of the leading object oriented development tools such as Rational Rose and Requisite Pro.  RUP contains a wealth of information, including process roles, activities, recommended artifacts and document templates.

RUP is described as use-case driven, architecture-centric, iterative and incremental. Functional requirements are captured in use cases that drive the development process. Specified use cases support the analysis & design and test workflows. Use cases also help define the architecture, which in turn, influences the selection of use cases.  This interplay between use cases and architecture evolves in the context of an iterative and incremental process.  A project is divided into a series of iterations where the most architecturally significant and/or technically complex use cases should be tackled initially.  During early project iterations, some previous work may be discarded but later iterations add a greater proportion of incremental functionality.

The motivation for a process like RUP is based on the argument that iterative processes reduce project risk.  Requirements that change over time are more easily incorporated into an iterative process, and management gains greater visibility into the current status of the project. Project cost is reevaluated at the end of each iteration. By focusing on essential requirements first, a functional system can be delivered sooner with deferral of lower priority functionality to subsequent releases. 

As seen in Figure 1, RUP is composed of four phases and nine workflows.  The phases are Inception, Elaboration, Construction and Transition.  The workflows are Business Modeling, Requirements, Analysis and Design, Implementation, Test, Deployment, Configuration & Change Management, Project Management, and Environment.  Phases are subdivided into iterations during which the workflows are performed.  The core workflows, Requirements through Test, essentially constitute a mini-waterfall process. So RUP can be viewed as a succession of short waterfall processes during which a system is built incrementally.  Ambler [2] aptly describes this type of process as serial in the large-scale and iterative in the small-scale.

Figure 1 - The Rational Unified Process

During the Inception phase, the project scope is defined and the business case is justified.  High-level use cases are developed for the system’s primary functional requirements.  Project management tasks include conducting an initial risk assessment, and estimating resources, schedule and overall cost.  The completion of the Inception phase often acts as an important control gate for determining whether to proceed with the project.

During Elaboration, the problem domain is subjected to detailed analysis and the architectural foundation is established.  A supplementary specification is developed for non-functional requirements.  In addition, the architecture and design of the system should be validated throughout this phase and reflected in its iterations.   

In the Construction phase, the detailed designed is realized and implemented iteratively.  The objective is to produce a functional component of the system by the end of each iteration.  Supporting documentation is also developed iteratively.

The Transition phase results in the delivery of the system to the user community.  This commonly occurs after a beta release has been tested by end users and debugged.  The final iterations within Transition should focus on more debugging and lower risk development.

It is important to recognize that phases are more conceptual than tangible, particularly between Elaboration, Construction and Transition.  Determining when one phase ends and another begins is less important than ensuring that the fundamental components of each of the phases is addressed throughout the lifecycle of the project. 

RUP’s strengths lie in its foundation as an iterative process that is requirements driven and architecture-centric.  A working component of the system is delivered at the end of each iteration that enables the client to make a go/no-go decision.  The process is extensively documented, model artifacts are based on UML, and supported by a suite of automated tools.  It is a robust process framework that can be tailored to project needs.  

Nevertheless, there are deficiencies in RUP. Ambler notes that it is essentially a single-project development process.  It fails to consider enterprise-level, multi-project issues. As a result, application architectures may fragment across the enterprise and opportunities for software reuse may be lost.  It does not address the post-Transition phase concerns of operational support and maintenance, which over a system lifetime probably consume more total resources than initial development. Since it is based on UML, non-UML model artifacts are not covered and the process documentation can consume expensive resources.  In Ambler’s opinion, the marketing of Rational’s commercial tools dominates the process.  Other weak areas include metrics, testing and the crucial area of human resource management.        

Hesse’s [7] criticisms of RUP are more fundamental.  He argues that phases, which are a holdover from the classic waterfall process, are not an appropriate structuring concept for OO projects.  Instead, process activities and iterations should be linked to architecturally significant milestones.  Workflow is an overloaded term and its relationship to phases is overly complex.  RUP lacks high-level support for structuring complex processes such as hierarchy, recursion and orthogonality.  RUP should be decomposable into sub-processes.  There is inadequate support for project management and roles/interactions of project groups, especially in the area of user feedback.

In addition to these criticisms, we found that RUP lacks detailed advice on the Implementation workflow.  Since coding is arguably the most important project activity, a software development team needs comprehensive guidance on how to most efficiently manage the actual coding of the system.

Kent Beck [4] started XP about six years ago, and it is probably the best known of the lightweight methodologies. Others include Peter Coad’s Feature-Driven Development (FDD), Alistair Cockburn’s Crystal family of methodologies [12], Jim Highsmith’s Adaptive Software Development [13], SCRUM [14], Scott Ambler’s Agile Modeling [1] and others.  More recently, the leading proponents of lightweight methodologies have formed the Agile Alliance, an umbrella organization that promotes the principles of lightweight processes.   A growing number of websites, magazine articles, books and conferences devoted to agile methods attest to their current popularity among programmers.    A complete discussion of agile methods is beyond the scope of this article and the reader is referred to [6] for further information.

XP promises a high level of customer satisfaction based on four process principles: communication, simplicity, feedback and courage.  To gather requirements, it supports communication within a development team that includes client staff, and it encourages the use of the simplest design and implementation that meets those requirements.   Feedback is obtained through extensive unit testing during each iteration.  In fact, unit tests should be written prior to actual coding. With this foundation of teamwork, simplicity, and automated testing, XP claims that developers have the courage to respond to unexpected requirements changes.

Its probably more accurate to classify XP as a collection of practical  programming techniques than a complete software development process.   Figure 2 depicts the process flow for an XP project.  Requirements are captured in user stories, which are essentially high-level use case descriptions without detailed, step-by-step functional details.   The XP website provides the following description of user stories:

User stories serve the same purpose as use cases but are not the same. They are used to create time estimates for the release planning meeting.  They are also used instead of a large requirements document. User Stories are written by the customers as things that the system needs to do for them. They are similar to usage scenarios, except that they are not limited to describing a user interface. They are in the format of about three sentences of text written by the customer in the customer’s terminology without techno-syntax.

In XP (Figure 2), each iteration is preceded by a Release Planning activity that describes the work to be done based on the following inputs: requirements (User Stories), high-level architectural analysis (Architectural Spike) and throw-away programs that explore particularly difficult areas (Spikes).   The iteration proceeds after confident estimates are obtained and the code is debugged until it passes all acceptance tests.  The result of an iteration is the release of a functional sub-system.   Eventually, the complete system is developed through this iterative process.

XP is an iterative process designed for small (2-12) teams co-located at the client’s site.  Developers, who work in pairs, write unit tests prior to writing the actual code and they review each other’s code.  The code should be the simplest possible implementation of the requirements and it should be refactored repeatedly.

In reaction to heavyweight processes that have elaborate management structure and produce extensive documentation artifacts, XP deliberately minimizes these process activities.  Project management is viewed as inefficient bureaucracy, and code is regarded to be the only documentation that is inherently reliable. These policies are justified with claims that modern software development, especially internet applications, can not be developed with heavyweight processes when requirements are unpredictable and delivery timeframes are short.   

XP’s strength is its unflinching focus on getting the code written so that each iteration yields a functional product. It is a hands-on approach driven by developers with direct client involvement to foster communication, much of which is informal.  Product quality is achieved through pair programming and automated unit testing.

XP does have limitations.  Unlike RUP it does not have as much focus on establishing environment or architecture requirements for the system.  In fact, its focus is really in RUP’s construction phase where code is being planned and implemented.  The claim that pair programming is more productive is debatable. It does not provide clear guidance on what models or documentation artifacts should be developed, and a distributed development environment inhibits informal communication.  With minimal management and documentation activities, the ability of XP to scale to enterprise level projects is questionable.  This often makes it difficult for larger corporations to support application development driven by XP due to concerns that the software will be difficult to support and maintain.

As mentioned previously, the client had already adopted RUP as its official process throughout the enterprise and this project was the first to use it. From our critical analyses of RUP and XP discussed above, we decided the best course of action was to selectively blend the strongest features of RUP and XP.  This is straightforward since the two processes are complementary.  Most importantly, RUP and XP are both iterative processes.  Although neither are enterprise-level processes, that limitation was not significant for this project.  RUP was used as the base process with XP primarily filling in the gaps of the implementation workflow. 

Per RUP, the requirements process was use case driven.  Model artifacts followed UML notation.  For the implementation workflow, we used pair programming and unit testing, both recommended XP techniques.  Programming estimates were developed by the developers based on design artifacts ‘handed off’ to them at the start of each iteration.  Based on management considerations, iterations were four weeks in duration and a formal demonstration of the system’s functionality was held at the end of each iteration.   Architectural considerations were the primary factor in the development of a project plan that mapped use cases to iterations and this plan was updated at the start of each iteration. 

The following section describes the application of this blended RUP/XP  process to the project.

The Inception phase of the PRS project was a fixed price, fixed time contract that identified a series of key artifacts needed to identify the success criteria for the project.  The Inception phase followed the workflows of the Rational Unified Process and utilized the resources, as defined within RUP, of a Project Manager, Architect, Designer, Use Case Specifier, User Interface Designer and an Implementer.

The PRS project was initiated through a series of workshops intended to both solicit requirements while at the same introducing key stakeholders to the development process and project scope.  The workshops were half-day sessions with notes from each session captured and compiled for review in the following session.  A total of 15 workshops were held over four weeks, the results of which guided the development of a series of artifacts that would be used to develop a project plan and estimate.

The workshop sessions were complemented by analysis of business documents and interviews with stakeholders in order to develop most of the artifacts identified for this phase.  The artifacts produced as deliverables included the Software Development Plan, a Class Diagram, Project Glossary, Vision Document, an Exploratory Prototype, Candidate Architecture, Software Requirements Specifications, Business Rules, Business Use Cases Model and Use Cases, Collaboration and Sequence diagrams and Status Reports[1].  In addition, several other non-RUP artifacts were developed which included project artifacts (GANTT charts, a RUP Worker Matrix and a Mentoring Plan).

While a large number of formal artifacts were developed as part of the Inception Phase, the primary focus was on the Business Use Case Model and the Class diagram.  The exploratory prototype helped business stakeholders visualize the parameters of the project and was later used as the preliminary user interface during the Elaboration and Construction phases.  The remaining artifacts either served as direction setting documents (e.g., Vision and Business Case), or did not add sufficient value for use in subsequent phases (e.g., Business Rules and Software Requirements Specifications artifacts).

As discussed above, the key deliverables from the Inception phase was a set of high-level use cases, and a business domain class model.  The first objective of the Elaboration phase was to develop a more detailed plan for the completion of the project in nine, one-month iterations.  This timeframe was a hard customer requirement based on competitive factors. 

The initial task was to construct a development plan for the entire project that could be easily revised at the end of each iteration.  This plan, the Project Development Routemap (PDR), was maintained in an Excel spreadsheet.

The first step was to develop a list of all known use cases to be developed.  This included the Inception phase use cases, including the decomposition of some into more detailed use cases, e.g., CRUD (Create, Read, Update, and Delete) use cases, system administration use cases, etc.  Secondly, a list of classes was prepared.  This included the Inception phase classes supplemented with additional classes such as DB access. 

Desired functionality was ranked on two criteria: from highest to lowest risk, and from highest to lowest priority.  The preferred sequence of development is from use cases of highest risk/highest priority to use cases of lowest risk/lowest priority.

However, we agree with Hesse that in a truly architecture-centric process, iterations should be defined on meaningful architectural milestones.  This means the development sequence of use cases must be modified so that use cases composing an architecturally significant part of the system are developed in the same iteration. However, this requirement must be relaxed when the amount of work is too great for an iteration, the estimates of which are created by the development team themselves.  In that case, the remaining work is deferred to the following iteration. At the end of each iteration, the entire routemap is revised based on the amount of work completed and any new requirements discovered.

The strategy for segmenting the project into iterations during the Elaboration and Construction phases is governed by three principles in the following order of importance.  First, project management considerations dictate that iterations be a fixed duration that provides adequate time to develop significant functionality while short enough that the client can maintain visibility into the project’s status.  We believe that one-month iterations are a good compromise.  Deadlines are predictable and easy to schedule.  The development team can implement significant functionality in a month, and the client knows the project’s status.   Preparing for an end-of-iteration demo is time-consuming both for the project team and the many stakeholders who must participate.  If iterations are too short, you lose efficiency.  If iterations are too long, the project may flounder or stray off course.

Secondly, the actual work performed during an iteration is governed by architectural concerns with the most critical architectural issues addressed first.  The key architectural concern for this application was the rating engine.  It was also deemed the highest priority and the most complex of all the use cases.  The rating engine incorporated several independent rating algorithms, and the user could optionally specify the particular algorithm to use for a group of policies. Since practically all other functionality depended on the rating engine, it was tackled first. 

Third, core architectural components like the rating engine should be developed in parallel with related functionality.  For example, to realistically use the services of the rating engine, it was necessary to also develop lightweight implementations of related use cases and classes even though they were not necessarily the highest priority or most complex.    In fact, the necessity of incorporating this supplemental functionality required that only half of the rating algorithms could be implemented in the first iteration with the remaining algorithms deferred to the second iteration.

As we continued to discover new requirements during the project, we revised the routemap at the end of each iteration.  Figure 3 shows the final breakdown of work by iteration.

Figure 3 - Iteration Schedule of the PRS Project

For an iterative project to be successful, a key skill is the ability to estimate the productivity of developer resources.  Overestimation is a bigger concern because developers may complete their work prior to the end of the iteration and have no work to do until additional use cases are specified.  Underestimation of work is also undesirable due to developer frustration and the impression that the project is not making adequate progress.  Of course, use cases not implemented are simply deferred to the next iteration.

Figure 4 - PRS Project Development Route Map

Figure 4 shows the Project Development Routemap for the initial iteration. The use cases and class to be implemented are listed and assigned priority and complexity rankings (high/medium/low).  The level of effort in perfect engineering days (PED) is based on the complexity attribute.  In this project, use cases were assigned a PED of 5 days for high complexity, 3 days for medium complexity and 1 day for low complexity.  The respective values for classes were 3, 2 and 1 days respectively.  A perfect engineering day assumes that a developer works at a high level of efficiency in a highly productive development environment for eight hours without any interruptions or lost time.  This is an unrealistic assumption.  Developers typically must ramp up on a new development environment, gain understanding of the problem domain and learn to work as a team.  Of course, there are always interruptions and downtime. 

Therefore, the level of work must be converted from PED to actual work days.  The total work in PED is multiplied by an adjustment factor (5) which reflects the productivity assessment of the team.  Initially, the adjustment factor is higher reflecting the development team getting introduced to the problem domain, developing a required team camraderie and accounting for other unforeseen productivity factors (such as installing new software, configuring computers etc.).  These estimates are used only as a tool to determine how much analysis and design is required for each iteration.  The developers themselves provide the detailed estimate of time required to realize the functionality of each iteration.  Over two or three iterations, however, the adjustment factor is refined to more accurately reflect true development productivity.

Then, the total days required (100 days) is divided by the number of developers to obtain the level of effort in team days.  Team days can be easily converted into team weeks and team months.  However, as the project progresses, the development team should become more productive and the PED adjustment factor should accordingly decrease.  In this project, the adjustment factor gradually declined to a stable value of 2.2 by the start of the fourth iteration.

The usefulness of this approach is dependent on assessing the complexity of use cases and classes in terms of PED and the selection of suitable PED adjustment factors through the project lifecycle.  XP recommends the use of a range between 2 and 5 with 2 being optimistic and 5 being used for unfamiliar technology.  Since the correct adjustment factor can only be determined after several iterations, our approach has been to apply a conservative estimate that can be adjusted downwards over time.  This semi-quantitative technique did a reasonably accurate job of estimating iteration workloads and overall the Project Development Routemap was a useful tool.

Figure 5 - Iterative Process Activity Diagram

Based on our experience, we agree with proponents of both RUP and XP that iterative processes offer a superior approach to software development.  Nevertheless, the successful execution of an iterative process is a challenging exercise requiring astute project management skills.   In our opinion, iterative processes succeed not because they are easier to execute. On the contrary, they place very high demands on project staff to meet frequently recurring deadlines.

This high performance level requires a clearly defined sequence of activities that explicitly maps roles to artifacts. Although RUP and XP offer much useful advice on this topic, we found that neither offer a sufficiently detailed iterative process plan.  Figure 5 is an activity diagram that depicts the process employed in each iteration of our project.  The diagram includes a swimlane for the client, analyst/architect and developer.  A role has lead responsibility (not necessarily sole responsibility) for all activities (rounded ellipses) within the role’s swimlane.  In cases where responsibility does not neatly map to a single role, the activity is shown straddling two swimlanes (e.g., Elicit requirements activity).  To our knowledge, this is a non-standard notation for UML activity diagrams but we believe it conveys useful information.  The obvious shortcoming of this notational compromise is that an activity can only straddle adjacent swimlanes.  

The central complexity of an iterative process is shown between the fork and join (heavy horizontal lines) bars within the analyst/architect swimlane.  The analyst/architect must always do analysis and design one iteration ahead of the developers.  At the end of an iteration, the developers have completed the implementation of the previous iteration’s analysis and design, while the analyst/architect has completed analysis and design tasks to be implemented during the next iteration.  Throughout the development process, however,

In addition, the analyst/architect has important responsibilities for support of the developers.  The analyst/architect must be available to answer questions from developers and to possibly modify the design of the system based on its actual implementation.  This role must prepare an iteration DB for developers, write test cases based on use cases, and perform integration testing after the developers finish their iteration build. 

Balancing these two sets of parallel activities requires careful time management by the analyst/architect.  In fact, the iterative process is a very delicate time management exercise.  The client must be able to provide requirements feedback early on during the current iteration. The analyst/architect must quickly prepare models for requirements validation and guidance to developers.  Finally, the developers must finish their implementation with sufficient time remaining for integration testing prior to the client’s demo at the end of each iteration.

This tight integration among the various roles makes the iterative process highly susceptible to disruption by factors such as vacations, sickness, and routine business schedule conflicts.  In our opinion, this high sensitivity to everyday events is one of the most serious weaknesses of an iterative process.  In particular, we found that having an adequate amount of time for integration testing was a constant challenge.  That problem is discussed in more detail below.

Although RUP was the client’s officially adopted process, every attempt was made to keep its implementation as lightweight as possible.  There were two rationales for this decision.  First, we expected an iterative process to be highly disruptive internally to the client as it had previously only followed waterfall based software development projects and we wanted to minimize the organizational impacts.  Secondly, we wanted to assess the potential usefulness of XP techniques within a RUP framework.

In keeping with an agile RUP process, we produced a compact suite of artifacts in order to minimize the resources needed to create and maintain these artifacts.  Functional requirements were specified in use cases maintained in a word processing document.  Test cases were taken directly from the use cases and were also maintained in a word processing document.  A class model was maintained in Rational Rose.  Screen designs were prepared informally by hand.  Occasionally, sequence diagrams were prepared by hand and usually discarded shortly after preparation. Only the use cases, test cases and class model were maintained throughout the entire project.  In summary, the artifacts were non-graphical with the class model being the notable exception.  The use case and test case documents grew to be quite lengthy, approximately 150 and 100 pages, respectively. 

According to RUP, the objective of the Elaboration phase is to baseline the architecture to provide a stable basis for the subsequent development of the system.  The most significant requirements are implemented during this phase, which should consume about 20% of project resources and 30% of schedule time.  The Construction phase  (65% of resources and 50% of schedule) is devoted to discovery and implementation of the remaining requirements.  In our experience, the boundary between Elaboration and Construction is somewhat arbitrary.

For this project, we defined the Elaboration phase to consist of three iterations.  This is based on the demonstrated ability to reliably calculate reinsurance premiums for a policy using a variety of rating algorithms.  The remaining six iterations are assigned to the Construction phase since they dealt with the more routine aspects of the business domain.  In terms of project resources (and schedule time), this results in a ratio of 1:2 between the Elaboration and Construction phases, which is within the guidance provided by RUP.

The transition phase of the PRS application involved the final iterations of the project and introduced the deployment of the application to an offsite hosting environment.  While the implementation complexity of this final  iteration was intended to be less than the iterations covered in Elaboration and Construction phases, the environment variables introduced more risk than had originally been forecast. 

FederalRe had recently established a contract with an offsite hosting company to provision 24x7 support services for its web based applications.  The rationale for the decision was based on the significant resource requirements for true 24x7 support.  However, the infrastructure implemented for the hosting environment contained some differences from the environment where the development and integration work had been completed.  The synchronization between these different environments also proved to be demanding particularly in ensuring the configuration of the production environment.  This also impacted the timelines required for Quality Assurance certification.

The transition period also provide an opportunity to address any outstanding defects that were identified in the final construction iteration as well as any minor functionality requirements (mostly centered around screen navigation) that could not be addressed in the previous iteration.

Using the blended RUP/XP process described above, this successful project was completed on time and within budget.  We believe there are several reasons for the project’s success.  First, it benefited from strong client commitment, both to the project and the process.  The consulting team was highly experienced in both the business domain and technology.  Project management, from both client and vendor, was intimately involved throughout the engagement. The entire team worked at the client’s site, enjoyed open access to stakeholders, and worked in a dedicated development lab. 

We believe this project benefited greatly from an iterative process.  Iterative development imposes a high level of discipline on all parties involved since deadlines occur frequently, and on a predictable schedule.  Since the most difficult functionality was tackled first, the development team was confident they could complete the project after the second iteration was finished.  The client was comfortable since they could see tangible evidence of progress on a monthly basis.  In our opinion, iterative development is a major risk reducer for two reasons.  First, the probability of failure is much less since risk is proactively managed throughout the project lifecycle.  Secondly, the system can begin generating revenue sooner because a functional system is delivered at the earliest possible date. In short, a successful project using an iterative process results in a higher rate of return on investment with lower assumed risk. This is a ‘best of both worlds’ scenario.

The inherent benefit of an iterative process is even more striking when one considers the partial process implementation used this project.  Supporting documentation was fairly minimal.  The only automated tool used for analysis and design was Rational Rose.  There was no automated tool for managing requirements or test cases.  The only maintained model was a UML class diagram. 

Nevertheless, we are convinced that iterative development processes like RUP and XP have substantial room for improvement.  In reality, both are process frameworks that must be tailored to specific projects.  There is only general guidance on how to customize these processes for a particular project, especially for organizations transitioning to an iterative process.

In particular, we think practitioners would benefit from detailed process activity diagrams like the activity diagram presented here.  We can envision a suite of such diagrams depicting best practices for various types of projects based on key project variables such as business domain, project size, performance requirements (e.g., real time, high reliability, etc.), project roles, and application architecture, just to name a few.   At the very least, the availability of detailed process diagrams would provide a basis for critical analysis, and help the industry evolve past its current state, which is all too often dominated by marketing hype.

There are some particularly challenging aspects of iterative processes that should be noted. A commonly touted advantage is that iterative processes avoid the unpredictably long testing period at project end.  Admittedly, it is less expensive to cure a defect early in the project.  Nevertheless, testing remains a major resource commitment that continually threatens the project schedule.  Even with automated unit testing, it was very difficult for the developers to finish the iteration build in time for adequate integration testing particularly with regression testing.

In this project, developers were scheduled to deliver a build early in the final week of the iteration.  The analyst/architect then performed integration testing.  Bugs were logged in a Lotus Notes database and the developers fixed bugs generating a new build at least daily. This cycle was repeated until Thursday afternoon at which time the system was frozen for the Friday demo to the client.  Unfortunately due to the ambitious workload, the initial build was often delivered later in the week.  This prevented the analyst/architect from conducting thorough testing, and the developers did not have enough time to fix all the bugs.  The unstable build was delivered to the client’s QA department on the following Monday as the project team kicked off the next iteration.

As a result, it was not possible to achieve the level of stability required and some known bugs were carried over to the next iteration.  This ongoing deferment of bug fixes to later iterations became a major impediment and resulted in a longer than planned stabilization period after iteration nine.  To some degree this necessitated an iteration ten, although this was also a reflection of a significant architectural change associated with an off-site hosing service.  We feel this problem can be partially addressed by automated integration testing.  To successfully use automated testing, it is crucial that the GUI be stabilized as early as possible.  Unfortunately, this was not possible because the usability expert joined the project very late resulting in extensive revisions to the GUI.

Another mitigating technique is to schedule a buffer period for extra testing periodically throughout the project, a practice employed by Microsoft.  The idea is to designate certain iterations as major release points which are allotted an extra period of time to stabilize the application with extra testing.  We believe that inserting a buffer period after every third iteration, for example, is merited given the scheduling challenges.  For a one-month iteration, we recommend a buffer period of at least one week (see Figure 6).

Figure 6- Iterative Process With Buffer Period for Testing

Another process area in need of reevaluation is determining the optimum suite of artifacts needed for communication with the client and the developers.  This tends to be controversial, particularly with adherents of agile methods who argue that the only reliable documentation is the code.   While certainly true, our experience indicates few business stakeholders know how to read code and even fewer are interested in learning how.  Therefore, this ‘lightweight’ argument largely misses the point.  In our quest to minimize documentation, we omitted routine UML models like activity and state diagrams that can be effective communication devices.  Instead, we generated documents over 100 pages in length containing dozens of use cases and test cases.  We did maintain a detailed class model that was indispensable for developers but largely ineffective for business users.

Not surprisingly, business users became more confused about the functional relationships of the use cases as the document grew in complexity.  A few carefully drawn activity diagrams could have helped immensely. 

It was also evident that developers can benefit from more artifacts than use cases and a class model.  Developers commonly spent precious time trying to decipher a use case when a well-designed activity or sequence diagram would have sufficed.  For complex use cases, we suggest a simple activity diagram to illustrate the alternative flows, and we advocate sequence diagrams for the major execution scenarios.  Sequence diagrams are especially valuable at linking the use cases to the class model.

Another important factor in determining appropriate artifacts is the client’s concern about ongoing support of the application.  Our client was extremely concerned about receiving mentoring throughout the project and substantial time was devoted to addressing that need.  The client wanted the ability to maintain both the iterative process and the application after the engagement was finished.  We believe that the breadth and depth of required documentation is a function of the client’s legitimate concern over their ability to protect a crucial investment after the consultant has left.  In other words, the  ‘lightweight’ vs. ‘heavyweight’ debate over documentation is largely irrelevant when not considered within the client’s context.

The client’s long-term goal in adopting RUP is to achieve a repeatable process for software development throughout the enterprise. This requires deliberate actions designed to acquire and maintain process knowledge within the organization.  From this perspective, documentation and mentoring assume an importance beyond the boundary of a single project.  We suggest clients carefully assess their long-term, enterprise-level knowledge management requirements when deciding how lightweight the documentation and mentoring should be for a specific project.  We are concerned that the overzealous adoption of lightweight methodologies may leave a client with insufficient in-house expertise.

We also believe these concerns are applicable to the consulting organization, too.  With a lightweight methodology, it is possible that junior members may have less opportunity for professional growth. 

This project gave us the valuable opportunity to assess some of XP’s implementation techniques, namely unit testing and pair programming.  We think unit testing is very useful and recommend its adoption where feasible.  This was the first time our development staff had done unit testing and it was partially implemented, mostly in methods that accessed data.  We believe that a full unit testing program would have significantly improved the quality of the code resulting in a shortened testing cycle.

We are more cautious in the use of pair programming even though its merits have been touted in published studies [11].  For example, it is very easy for the programmer not at the keyboard to become less engaged.   Personality conflicts can also occur although we did not encounter this problem but have witnessed it on other projects.  Our overall impression is that pair programming did not result in dramatically higher productivity or quality of code.  We do think that pair programming is useful as a mentoring technique between senior and junior staff and encourage its use in the early iterations of a project while developers are still learning the application domain.  However, one positive benefit provided by paired programming was that it provided a much more robust scheduling mechanism for vacations, sick days and other human factor elements of development.  This resource redundancy helped provided significant flexibility to the project schedule.

We are intrigued by the variation on pair programming employed at Microsoft [5] where the pair consists of a programmer and a tester.  We think that integrating a pair of programmer and tester roles with unit testing has potential, and in a future project we intend to evaluate this approach.  In our technique, a pair will be assigned a related set of use cases, and each programmer will write their own unit tests prior to writing their own code. Each member of the pair would then review his partner’s unit tests and code prior to jointly integrating their work.  We feel this arrangement will result in higher productivity and independence while preserving the comfort zone of a partner’s help with difficult issues.

Overall, we believe there is much to be learned about optimizing iterative processes such as RUP and XP.  Our experience with XP is generally consistent with its use on a large financial services project [8]. We hope other workers will publish critical reviews of actual projects supported with metrics, where possible.

An iterative process blending the best practices of the Rational Unified Process (RUP) and eXtreme Programming has been used to deliver FederalRe’s Property Reinsurance System (PRS) on time and within budget.  This web services application is deployed and currently in use by FederalRe’s clients. 

By any objective criteria, this project was successful. We attribute that success to the use of an iterative process supported by strong project management and competent technical staff.  Although iterative processes are difficult to implement, we are convinced they significantly reduce project risk. A committed project management, on both the client and consultant sides, is a precondition for the effective execution of an iterative process.  Finally, the technical staff must be able to work efficiently under a tight schedule to meet project milestones. 

Nevertheless, we are also convinced that the iterative process employed in PRS can be substantially improved.   The process needs to be specified in greater detail, especially in mapping roles to activities with resultant artifacts.  To maintain the project schedule, it is critical that additional automated tools be used, especially for requirements management and testing.  We recommend that buffer time for testing be inserted into the project schedule approximately every third iteration so that the application can be stabilized.  Otherwise, defects are carried into forward iterations.

Some process activities require further evaluation to assess their applicability.  We found unit testing helpful and plan to expand its use in future projects.  We are more cautious toward pair programming.  Although we think it has a niche, we are unconvinced that it is a more efficient implementation activity.  We do plan to explore variations on pair programming to determine if greater programming efficiency can be gained. 

Overall, the industry is making significant progress toward the goal of developing reliable software in a cost-effective manner, and the iterative development process is an important evolutionary step.  We encourage practitioners to implement and follow an iterative process, carefully monitor process efficiency, and report what works and what does not work.  In the meantime, beware of simple processes that claim happy programmers guarantee happy customers. In the long run, we expect the industry will evolve toward an iterative process framework that can be easily