Blending A Test Strategy Into an Iterative Development Project

by Bryan Campbell

March 18, 2002

Testing has always been the Achilles heal of many software development projects but if you choose to follow an iterative development approach you’ll need to pay special attention to your testing strategy.  The challenge with testing has often been ensuring that adequate time, expertise and resources can be applied to uncover any defects within an application prior to its production release.  Application testing is driven by two irrefutable facts: one, even the best developers make mistakes and two, it is much cheaper to correct a defect in development than in production.  Recently, a relatively new development approach known as ‘iterative development’ has become popular in both commercial and internal development environments.  While iterative development heavily emphasizes testing in its development process, it is still necessary to develop a comprehensive testing strategy to ensure a high quality software product.

Iterative Development

Iterative software development has become increasingly popular in the software development community for a variety of reasons.  One of the most important is its flexible nature in dealing with requirements changes throughout a project’s lifecycle.  While the definition of iterative software development is open to some interpretation, most practitioners would claim it includes frequent releases of working software delivered in short, consistent intervals, it addresses the most complex, or high risk, items early in the development cycle and relies heavily on the expertise and estimates of the developers actually doing the work.  One of the most popular forms of iterative development has been documented by Kent Beck in his eXtreme Programming (XP) methodology.  Beck’s influential work has also been complemented by a variety of other leaders in this area such as Jim Highsmith (Adaptive Software Development), Peter Coad (Feature Driven Development), Scott Ambler (Agile Modeling) and others.  Together these iterative development approaches are often grouped under the term Agile Development, reflecting their lightweight, and therefore ‘agile’, approach to software development. 

Another popular iterative development methodology has been the Rational Unified Process (RUP) offered by the Rational Software Corporation.  RUP is a more formal process than that advocated by the agile development group but it adheres to essentially the same iterative characteristics defined above.  One area that RUP differs significantly from the agile theorists is its focus on documentation and its use of a standardized notation, the Unified Modeling Language (UML).

Both the agile development methodologies and RUP recognize the value and importance of testing.  However, both approaches require an active and comprehensive approach to testing in order to ensure that the benefits of their respective iterative approaches (rapid development with the agile methodologies and thorough, standardized documentation with RUP) are not devalued by unmanaged defects in code.

Testing in the Iterative Development Approach

Both RUP and XP include a heavy focus on testing throughout their methodology framework.  Within RUP, a test ‘workflow’ exists that illustrates how to develop and support a test management strategy for all elements within the project.  XP focuses on testing more at the unit level and is based on the assumption that no code can be released until it has passed its basic unit tests.

Within RUP, the testing workflow mirrors the development workflow with increasing amounts of testing activity as the project moves through RUP defined ‘phases’.  Interestingly, the amount of testing effort is seen to decrease in the final ‘Transition’ phase even though the RUP process supports regression testing throughout the development lifecycle.  RUP provides an excellent overview of the testing process including key concepts such as coverage measures (both code and requirements) and quality measures (such as defect tracking and trend analysis).

Both XP and RUP emphasize the need for automating testing particularly for regression testing of the application (testing all test cases with the release of a new iteration).  RUP uses the concept of a test case to validate functional testing of the system, which is developed from the use cases documenting system functionality. XP bases its testing structure on acceptance tests (formerly known as functional tests) which capture ‘black box’ test results (validating external outputs not internal component testing of code).  XP advocates creating acceptance tests from user stories that are similar to RUP’s use cases but are shorter (typically “three sentences of text written by the customer”). However, XP does not provide any guidelines on managing the testing sequence beyond the code level and its primary emphasis on managing and correcting defects is with the developers.  Developers are tasked with the responsibility of scheduling time each iteration to correct defects, which are based on acceptance test scores.  XP lacks the holistic approach to testing that the RUP methodology provides and does not go into the detail on the organizational alignment required for the project. 

The one most significant deficiency of both RUP and XP regard their defect correction from a previous iteration within a current iteration.  As we’ll see, managing defects within an iterative process become one of the most complex elements of a project.  The challenge lies in ensuring that defects identified outside of the tests executed by developers (during User Acceptance testing for example) can be adequately factored into the development cycle and corrected,.

In addition, an appropriate testing framework needs to be overlaid on both RUP and  the agile methodologies to ensure that an application is adequately tested.  There is a significant risk in the case of agile development projects that there will be insufficient testing of the application particularly outside of a pure code coverage perspective.  It is important to ensure that tests are applied to overall performance of the system (stress, load and performance tests), that the security of the system has been tested (of which the greatest security risks might lie in router and operating system configuration) and that functions such as data integrity and user interface testing are also captured.  RUP is also potentially guilty of not applying an appropriate testing strategy within its methodology (even though Rational sells a suite of testing tools). RUP does not provide enough detail on how to blend its testing approach into the rapid development environment of an iterative development project.  Without an appropriate framework all rapid development gets you is an unstable product faster.

Blending a Testing Framework into an Iterative Development Project

Regardless of whether you are following an agile development or RUP project, testing is important.  If you’ve read this far or if you’re starting from this section, here are some proven techniques that can help ensure your iterative development project creates a reliable and stable application.

Define a Test Strategy:  You need to understand what you want to test and more importantly who is going to do the testing.  The Unit tests advocated in XP provide good white box (structural) testing at the code level, however, these still need to be complemented by black box (functional tests) and appropriate component level testing.  Don’t lose sight of the forest for the trees.  Develop a test plan (RUP provides a good illustration of one) that shows how you’re going to address testing across the entire application.  For projects that I manage, I develop test plans for Functional Tests (driven by use cases), User Interface Tests (since these are often not defined in Use Cases), Performance Tests (load, stress and performance related), Data Integrity Tests and Security Tests.  I assign different individuals to create and maintain these test plans and have peer review sessions to ensure their completeness.  Developers are still required to create unit tests and these tests must be successfully passed before being integrated.  While developing a test plan of this nature might feel like a lot of work, the payoff is knowing that your testing the application from a variety of perspectives.

Test in Layers:  Both RUP and XP place a great deal of emphasis on the developers to test the application.  This makes sense from the perspective that they know the code and they should be responsible for fixing their own mistakes.  However, it is important to have different groups with different motivations testing the application.  On larger projects, I’ll ensure that a Quality Assurance group separate from the development group tests the application, particularly the ‘system components’ (data integrity, security, performance tests).  I’ll also have a User Acceptance group test the application primarily from a functional and user interface perspective.  Having developers focus on code level testing, QA focusing on system testing and users focusing on whether the functionality meets their needs should help identify most defects before the system moves into production.

Insert a Testing Buffer period between iterations:  Iterative development projects focus on delivering functional pieces of code in short and consistent periods.  While this provides high visibility into a software development project and allows end-users to provide immediate feedback on the functionality developed for them, it can be very difficult to coordinate defect correction particularly as iterations build upon one another.  Oftentimes defects in a build might not be discovered for several weeks until user acceptance testing or quality assurance has had a chance to regress through their own tests.  After awhile tracking which defects have been corrected from which iteration can be overwhelming, particularly as your development team is preparing a build for an end of iteration demo.  To avoid this I insert a testing buffer every three iterations.  I use this one to two week period to allow the developers to catch up on all of the outstanding severity one and two defects (three’s too if time permits) and also let them catch their breath.

Assign a Testing Coordinator:  Someone needs to keep on top of what has been tested and what defects are outstanding.  Depending on the size of the project this might or might not be a full time job but someone needs to ensure that defects are properly captured and corrected.  This role is usually a good one to advocate testing automation and the person in the role should be both a strong developer and well organized.  The testing coordinator also needs to help the project manager understand the real state of the application (architects and lead developers are often too close to the application to offer accurate assessments).  The testing coordinator should be responsible for reviewing and assessing all of the defects in the defect tracker used each day.  The coordinator should work closely with the architect to determine if a defect really exists or if it reflects functionality that will be built in an upcoming iteration (remember iterative development means only partial functionality is available since everything is incrementally developed over the lifecycle of the project).  The coordinator needs to also work closely with those who sourced a defect to ensure they re-test it prior to closing it.  It’s not uncommon to have an individual enter a number of defects but not close them after they’ve been corrected, the coordinator needs to ensure that defects are well managed otherwise the defect log can become overwhelming.  Finally, the testing coordinator needs to actively ensure that tests are regressed frequently throughout the lifecycle of the project.  Many projects start short changing full regression testing in order to save time near the end of a project.  While there might be some ‘balancing’ required between full regression testing and meeting project deadlines, the test coordinator should ensure that tests are at least alternated between iterations, that a full regression test is applied every three iterations and that test cases are prioritized so more complex functionality is tested frequently.

Conclusion

For a more detailed perspective on the elements of this article, read Iterative Development Testing Approaches – Managing Iterative Testing in an Agile Development Project by Bryan Campbell and Dr. Glenn Ray. For more information please contact Bryan Campbell at bryanc@digitalesp.com or 919-608-0659