Agile Rational Unified Process: RUP experiences from the trenches

Introduction Scott W. Ambler
Authors Mark Lines, Joshua Barnes, and Julian Holmes

Introduction by Scott W. Ambler

This article, actually a compilation of three articles, provides proven advice for applying agile strategies on IBM® Rational® Unified Process®, or RUP®, teams. The articles are written by Mark Lines, Joshua Barnes, and Julian Holmes respectively, co-founders of Unified Process Mentors (www.upmentors.com). These three have mentored literally thousands of practitioners around the world on all aspects of software development, conducted dozens of conference presentations, authored many publications, sat on advisory boards, and chaired user groups. They work with organizations around the world to make process theory practical, driving success through structured change and leading by example.

My experience is that RUP, done right, is agile ¹ and that RUP encapsulates much of the advice needed to scale agile techniques successfully. The first article, Bringing Discipline to the Agile Lifecycle by Mark Lines, shows how RUP requirements management techniques and risk-driven lifecycle bring the level of discipline required in many organizations without losing the flexibility that is the hallmark of agile methods. He argues that you really don’t want the requirements to change radically later in the lifecycle, and that a bit of up-front investment can radically reduce your cost, schedule, and overall project risk. The second article, Strategies for Bringing Agility to RUP by Joshua Barnes, approaches the software process challenge from the opposite direction. He suggests ways to improve your RUP-based process “in flight” — many projects are started with fixed goals in mind and with rigid regulatory constraints placed upon them, yet once the team is partway through the project they realize that they can loosen up because the fixed goals aren’t so fixed and the regulations aren’t so rigid. The third article, Geographically Distributed Agile Teams: Enabling Individuals and Interactions with Processes and Tools by Julian Holmes, overviews strategies for improving collaboration within a distributed agile team. Achieving effective collaboration within a project team can be a difficult challenge for a co-located team, let alone a geographically distributed one. Julian provides advice for developing a collaborative team culture while still maintaining consistency of approach, delivery, and management of shared work products.

Bringing Discipline to the Agile Lifecycle

By Mark Lines

Agile projects can iterate endlessly, with the project only ending when it is cancelled due to an exhausted budget. Gold plating of features implemented in early iterations and unreasonable requirements churn are often the culprits. RUP adds structure to an agile approach by focusing on driving out requirements uncertainty early and tightening process control naturally as the project progresses toward completion.

One unfortunate trend that I have seen in Agile projects is a tendency for the projects to go from iteration to iteration, with seemingly no end in sight. Some project managers (PMs) seem to forget traditional PM rigors in favor of the Agile principle of “the customer is always right.” This leads to continuous requirements change from iteration to iteration. Often, as work items such as requirements come off the stack for implementation in the current iteration, an equal or greater number of work items are added back to the stack. Time and budget are quickly exhausted, with a large backlog of requirements yet to implement.

The two stages of a RUP project

Where RUP can really help these situations is by helping teams to recognize that all iterations are not the same. Walker Royce has described a project lifecycle as consisting of two stages. ² The first stage, corresponding to approximately the first 20-40% of the lifecycle, he calls the Engineering stage. This stage is made up of the Inception and Elaboration phases of the Unified Process (UP), which may in turn be carved up into multiple Iterations. The Engineering stage is characterized by much churn regarding all aspects of the project, such as plans, requirements, architecture, and code. This is natural and expected, as both the business and technical stakeholders struggle to understand what the system solution is and how to implement it.

Royce describes the second stage of the lifecycle as the Production stage, encompassing the Construction and Transition phases of the UP lifecycle. It is characterized by implementing the remaining requirements (approximately 60-80%) using proven techniques established in the earlier iterations of the Engineering stage.

Adapting change control rigor during the project

During the Engineering stage we try to avoid burdening the users with change control procedures. In turn, they should not expect firm commitments until the delivery team has implemented some of the requirements, thereby obtaining a baseline of metrics from which to extrapolate some firm commitments. With this behavior in mind, it is acceptable to add, change, and reprioritize requirements on the stack in the early iterations as the users see fit (within reason).

By building increments of functionality in Elaboration related to risky aspects of the software we can remove much of the uncertainty related to the project, such as understanding and establishing the requirements. Risk related to requirements uncertainty is mitigated with techniques such as prototyping, storyboards, visual modeling, and regular demonstrations of working software increments to the project stakeholders. Commitments regarding scope and schedule are usually expected at the end of this stage. The good news about this approach is that users are asked to commit to requirements much later than in a conventional waterfall approach. They have time to start to see their software evolve before we subject them to strict change control procedures, or as Scott Ambler likes to call them, “change prevention” procedures.

The bad news is that we still need to tighten up change control rigors at some point. It is unreasonable to expect the IT project delivery team to commit to schedule and budget if the requirements scope is continually changing underneath them.

I coach teams to consider the Elaboration phase of a project as their “sandbox” time. It is the time for IT to figure out what is possible and how to do it. As well, it is the users’ time to figure out exactly what they want.

Once the Elaboration “sandbox” phase is over, we enter the Production stage of the project. At this point, the PM needs to tighten up the change control or risk failing to deliver on the team’s commitments. Figure 1 illustrates this concept of increasing process control. Note that the Change Control Rigor curve does not necessarily increase as rapidly in the Construction phase as the diagram depicts. It depends on the unique aspects of your organization and project. For contractual relationships with users, it may be necessary to institute strict change control procedures early in Construction. For more collaborative and trusting relationships with your users, you may delay tightening process controls until later in Construction and the Change Control Rigor curve would be pushed to the right.

Figure 1: Change control rigor increases as the project progresses.

Educating our stakeholders is key

Unfortunately, many “agile RUP” PMs don’t change their behavior, or set appropriate expectations, as they move from iteration to iteration throughout the lifecycle.

Users often haven’t been trained to understand the changing emphasis across the UP phases, which as Gary Evans likes to say are basically the seasons of a project. They see that they have considerable change freedom early in the project, and naturally assume that this will be acceptable throughout the life of the project. I have witnessed end-of-iteration demos to stakeholders late in the project, meant to show progress before moving on to implement new requirements, turning into requirements elicitation sessions! PMs sometimes leave the meeting with more requirements added to the backlog than they implemented in the current iteration! This can be a frustrating experience.

It seems that many PMs who have agile training that is not UP-based do not fully appreciate the need for increased change control rigor as the project progresses. They manage each iteration in the same way, and don’t set expectations properly with the users.

Properly educated users and analysts understand that it is their job to drive out requirements uncertainty in the first half of the project, as requirements changes in the latter half of the lifecycle may need to be subjected to more rigorous change control procedures.

I am often asked at what point should we lock down the requirements and subject our users to formal change request procedures. Well, consistent with RUP’s flexibility, the answer is “it depends.” Ideally, in more collaborative environments, the answer is “never.” For more contractual arrangements, we need to have a more formal approach. Despite our best efforts, requirements will not be perfectly detailed at the end of Elaboration and will change in Construction. What I suggest to PMs is to allocate a contingency bucket of relative functionality “points” to be drawn from when new scope is added to the requirements stack in Construction. This approach is consistent with the way that I have depicted the gradual increase in Change Control Rigor in Figure 1. In the first couple of iterations of Construction, we can ease the users into more strict change control by allowing them to introduce new scope until they have exhausted their contingency points.

In summary, I have seen projects waste literally millions of dollars before eventually coming to a realization that they are still far away from having the solution that they initially envisioned. This is the result of both poor process control late in the lifecycle, as well as taking the collective eye off the initial vision and rather concentrating only on the needs of the current iteration. Remember that unlike pure Agile methodologies, with RUP each iteration is not treated the same. We need to use an adaptive project management style as we move through the lifecycle if we are to be successful.

Strategies for bringing agility to RUP in large organizations

By Joshua Barnes

Larger organizations that adopt RUP typically focus initially on their most common project type (i.e., larger projects in terms of duration, resource level, budget commitment, criticality to the business, etc.). These “larger” types of projects often fall under the scrutiny of control frameworks such as Sarbanes-Oxley (SOX) or Control Objectives for Information and related Technology (CobiT 4.0). This process rigor is often part of the project’s lifecycle whether it adds value or is just overhead that burns time and budget. In the early stages of adopting RUP, these larger projects are not allowed to optimize the process to be more agile due to the robust constraints place upon them. The question I am often asked is “how can we bring more process agility to our projects?” in later stages.

In my experience, larger organizations have areas such as Internal Audit, External Audit, and Governance/Compliance (to name a just a few) that have to buy into a smaller version of the process that has less ceremony. Agreement to create a more agile version of the “standard” process often does not come easily and places constraints on what types of work efforts would be permitted to use it. A threshold of some sort is typically used, such as less than ‘X’ hours of work effort or ‘Y’ dollars of budget.

It is a good day when agreement to commence on an agile version of the standard process is gained. These smaller work efforts can pilot the agility you have desired to introduce across the organization. You anticipate that once the value of more agility is proven, you can incorporate the lessons learned into the “larger” projects using the “standard” process. Below are three of the most common problems I encounter once the go-ahead has been given:

1. Not knowing where to start
2. Working group too large
3. Working group members lack recent experience

Problem #1: Not knowing where to start

You do not have to recreate the wheel, nor should you just put a new title onto an existing process. Figure 2 depicts process as a continuum, with a robust and traditional version of RUP tailored for an organization on one end and almost no process (ad hoc) on the other.

Figure 2: Balancing agility with discipline

Start with your existing tailored version of RUP for larger projects as your most robust and structured version. Next, look at a smaller version of RUP, such as OpenUP, as your minimal complete software development process for small teams. The goal is to eliminate waste and unnecessary process overhead in your robust version, optimizing the process for smaller teams and work efforts. Strive to move from the right side of the process continuum depicted in Figure 2 to the left, towards “pure agile.” This is a balancing act — agility against discipline as you move as far left as possible on the continuum without increasing project risk to an unacceptable level.

The first steps I have clients take look similar to the strategy for the initial implementation of RUP: analyze what will work out of the box, what will require customization, and what will not add any value and can be removed ³ . Analyze existing roles, tasks and products and remove what you can. Remember that these process components can be added back in by teams that may occasionally need them. Review the remaining tasks and products to identify how to make lighter-weight versions of them. Note that this should not be a laborious and lengthy process; the goal is to get to a smaller and lighter initial process framework and validate the decisions with real work efforts. Get out of the conference room and refine and revise as you learn lessons in actual practice.

Problem #2: Working group too large

I often encounter the scenario where a team to do process definition work is formed, meetings scheduled, milestones identified — with the goal of gaining complete consensus across the organization. This is not an agile approach to say the least. Although this kind of approach may typify large company cultures, it is far from optimum and you should strive to avoid this anti-pattern. Worse yet, the team often grows too large because every area that could potentially mount a work effort using the new, smaller process framework must be represented on it. This leads to a bloated work group that rarely comes to agreement on anything. Everyone declares “we are different” and that whatever is being discussed “will not work for us” without various changes.

An approach that has repeatedly worked effectively with clients is to look around the project for an area that has a large percentage of its work efforts in this “smaller” category. One client site had a development area that focused exclusively on an externally facing web application that frequently required small updates in very short timelines. They were screaming to have a lightweight process and gladly signed up to collaborate on defining a minimum process they felt they could use without introducing an unacceptable level of risk into their projects. Working with them, we were able to tailor a smaller, more agile version of the standard process in a few weeks, which they then piloted. This first pass was not what everyone in the entire organization would ultimately need, but it provided a starting point for us based on actual results that proved valuable. This approach cut down on trying to solve world peace all at once, which tends to just end up being a continuous series of meetings and negotiations, producing little more than meeting minutes.

Problem #3: Working group members lack recent experience

The representatives that are typically assigned to such efforts usually have titles like “Project Manager” or “Lead.” These are senior people with broad experience, which is great. However, most often they have not been a practitioner developing software in a really long time. Often their role is to move from meeting to meeting, representing their functional area and reporting back with the progress, risks, and issues.

The result is what should take a few weeks to complete now takes months and months. Since they do not have current experience using the existing process, every single little detail must discussed ad nauseam before any decision can be reached. The end result is typically a slightly smaller version of the standard process with customizations that don’t offer much relief to project teams that apply it in practice. I have seen this approach cost companies lots of money with very little return on the investment for the people that have to use it.

The team you need to make a smaller, more agile version of RUP must be comprised of project practitioners, the ones that have learned lessons using their organization’s standard version of RUP on real projects. The ones that have felt pain due to the lack of process optimization that they could leverage to produce high-quality, workable applications for their stakeholders. Working with a recent client, we were able to select a few project team members who had participated in the initial RUP adoption projects to be our subject matter experts from the “field.” They provided insight into how the process had overhead in actual application that would absolutely stifle a small work effort. Some of the initial barriers that we were getting hung up on with the internal audit and compliance representatives could then be broken down. It is one thing to discuss issues in the abstract, but when you have people who are truly passionate as a result of experiencing real-life pains day in and day out, it can be really hard to make a theoretical case against their requests.

These are just three of some of the most common issues I see with my larger clients trying to bring agility to their “standard” version of RUP. I will expand on the details of what I have seen work in solving/avoiding these three problems, as well as others, in follow-up articles.

Geographically distributed agile teams: Enabling individuals and interactions with processes and tools

By Julian Holmes

Many of the techniques that agile and iterative methods bring to software development require that the project team collaborate across all disciplines of expertise. This collaboration is particularly important when we consider the practice of iterative planning and development, and when the agile team minimizes the use of documentation as the approach to communication amongst team members.

It’s because of their effective collaboration that small co-located teams, using daily stand-up meetings and the continuous integration of their efforts, are typically more successful in delivering continual value to the customer than are geographically distributed teams.

However, different business strategies and pressures can lead to an organization being unable to have their people co-located for every project. Examples of these business “realities” include:

• Office space restrictions for the project
• Required expertise not available at the project location
• Customer located away from the project
• Outsourcing strategies utilizing third-party suppliers for specific skill sets
• Offshoring strategies that split the project roles between near and far locations
• Plus many more…

So how can an organization that has realized the potential of collaborative, agile techniques maintain those benefits when the team becomes distributed? The question is particularly pertinent because the 2007 Agile Adoption Survey by Dr. Dobb’s Journal ⁴ found that some organizations are trying to be Agile with distributed teams; and while some are succeeding, their success rate is lower when compared with co-located teams.

Agile practices provide excellent structured approaches to the work that needs to be done in a software development project. However, they almost all depend upon the experience and collaboration of the team to perform the actual software development work in a consistent and high-quality manner. As soon as the team is distributed, it becomes a far greater challenge for the work of the team to remain consistent in its approach, its delivery, or even the communication of its activity and the deliverables.

Having established that most of the challenges these teams face are related to the difficulty of maintaining collaboration when distributing a team, three of the key challenges are:

• How to develop a collaborative team culture?
• How to maintain consistency of approach and delivery?
• How to manage the development of shared work products?

Developing a collaborative team culture

Providing a project team with co-location certainly helps foster a collaborative nature to their work, but with a distributed team, it can be difficult for them to feel like a team at all.

Using tools can help. The most obvious are communications aids such as instant messaging, phone and video conferencing, and virtual workspaces. However, even the availability of these tools can provide limited value, unless the people at each end of the channel feel comfortable in communicating with the other people.

Everyone knows how much easier it is to speak to someone on the telephone, if you have met them before and established some kind of previous connection. It is likewise with many of the other communications tools just mentioned, which suggests the use of a rather less obvious “communication tool” — a mode of transport that enables team members to meet face-to-face.

Although traveling great distances can be very costly to a project in terms of both time and budget, the benefits of establishing improved communications, trust, and knowledge sharing can be great. However, commuting to and from a distant location would not be the right solution, so a balance does needs to be found.

Maintaining consistency of approach and delivery

With a collaborative team, and a culture of effective communication in place, the work required to deliver the project becomes the next problem to overcome. While the value of “individuals and interactions over processes and tools” is extremely important for agile teams, a defined process and a consistent framework of process language can really help enable individuals and interactions in distributed team situations.

RUP provides a process framework of software development language, a standard notation with the Unified Modeling Language (UML), and an approach to communicating the chosen process practices to a complete project team. Without a common language of software development, local variations in the use of terms or descriptions of activities, deliverables, or roles can lead to great confusion over the responsibilities and the expectations of team members at each location upon the others. It’s for these reasons that many an organization decides to invest in the education of its people to provide them with this common language. This can be achieved in many ways, with mixed results.

Simple training can provide people with basic language and notation awareness, but not necessarily the skills to use and communicate with them effectively. Additional standards and templates to be followed and used by all can provide consistency, but don’t encourage the interaction.

It is much better to have a mentor work with a project to encourage the interaction of all individuals, using the common language that can easily be interpreted and understood by project stakeholders and observers. This mentor will also help tailor the process framework to the needs of the project in alignment with the rest of the organization.

So, if we are to use a process framework like RUP, it also needs to be tailored to the environment. This can be achieved by documenting a consistent approach in a RUP Development Case. But through the use of IBM Rational Method Composer, this process tailoring can be formally described and published in HTML for any project team member to refer to as part of their role.

Managing the development of shared work products

Although a defined process, with the common language and approach that it brings, can assist in determining the activities of the distributed team, the volume and complexity of the work to be performed can still be overwhelming. This can be especially the case when a distributed team has to share and work upon many of the same deliverables concurrently.

The inability of a team to review and collaborate in real-time on the deliverables of the project can also hinder quality and slow progress. The team has to find ways to ensure that their outputs are accessible by any other member of the team at all times. For some it can be unnerving to share their work as they develop it, but if the project team is to collaborate, maintain quality, and drive progress quickly, it’s the only way.

One of the simplest ways to achieve this is to create a common central file repository, where everyone regularly uploads their work, and shares it with the team. Of course this repository needs to be well structured and managed, but it does get people sharing nonetheless. However, having to perform a separate activity to publish your work, after having spent time developing it, is an overhead that people don’t naturally want to accommodate. A smarter tooling approach is required. This is one scenario where IBM Rational solutions provide excellent support for distributed teams. Examples include:

• IBM Rational ClearCase® and IBM Rational ClearQuest® — Configuration Management and Change Management repositories that can be distributed across multiple sites, or accessed though HTTP, to provide everyone on the team access to all work products and the reason for their creation or revision.
• IBM Rational Software Modeler and IBM Rational Software Architect — Capturing UML diagrams in a single model, fully integrated with ClearCase to enable concurrent distributed work on the same model set.
• IBM Rational RequisitePro® — Managing the project requirements in a common repository, with traceability to the models, and accessed through the Eclipse IDE or through an HTTP client.

These solutions have existed for some time, but the Jazz project, a collaboration between IBM Research and IBM Rational, has also been developing new solutions for distributed teams. The first deliverable from Jazz, IBM Rational Team Concert, provides the mechanism for the management of project work products, through the provision of a distributed change and configuration management solution. However, Rational Team Concert also integrates that functionality with communications tools, integrated process support tools, and the process enactment of the approach captured in Rational Method Composer.

Agile, distributed, and still RUP

Having established the need for both process and tools in the support of the distributed but agile project team, a significant question remains: Can the cost of the investment in these solutions, the potential project overhead in their use, and the risks to project collaboration and hence successful delivery, be justified against the business reasons for not co-locating the team?

Often the answer has to be “Yes,” because the costs and issues of co-location are too great. (If it is a “No,” then some serious business change may be required!) So, for most organizations the combined process and tools solutions provided above will certainly help.

Parting thoughts

Many organizations around the world are taking an agile approach to RUP, and these three articles provide good insights for doing exactly that. Although many in the Agile community are loathed to admit it, RUP popularized many of the concepts which they take for granted, such as iterative development, delivering working software each iteration, testing throughout the lifecycle, and many more. Furthermore, RUP addresses the full spectrum of development issues, including the not-so-sexy ones that you must solve to develop systems at scale. The RUP done right is agile, and Agile done right is…. RUP?

References

1. Royce, W. (1998). Software Project Management: A Unified Framework. Addison Wesley.
2. Ambler, S.W. (2007). Dr. Dobb’s Journal 2007 Agile Adoption Rate Survey. www.ambysoft.com/surveys/agileMarch2007.html.
3. Barnes, J. (2007). Implementing the IBM Rational Unified Process and Solutions: A Guide to Improving Your Software Development Capability and Maturity. IBM Press.

Notes

1. http://www.ibm.com/developerworks/blogs/page/ambler?entry=agile_and_rational_unified_process
2. Royce, Walker. Software Project Management: A Unified Framework. Addison Wesley, 1998.
3. Barnes, J. Implementing the IBM Rational Unified Process and Solutions: A Guide to Improving Your Software Development Capability and Maturity. IBM Press, 2007.
4. Ambler, S.W. “2007 Agile Adoption Rate Survey” from Dr. Dobb’s Journal. www.ambysoft.com/surveys/agileMarch2007.html.

Resources

• Participate in the discussion forum.

• A new forum has been created specifically for Rational Edge articles, so now you can share your thoughts about this or other articles in the current issue or our archives. Read what your colleagues the world over have to say, generate your own discussion, or join discussions in progress. Begin by clicking HERE.
• Global Rational User Group Community