Iteration and Requirements Management in TOGAF

Setting the Scene

The recently released TOGAF 10 standard has expanded on many areas of the TOGAF standard with greater discussion and context around each of the important cornerstones of this framework. As a practitioner and trainer of TOGAF for over a decade now I have found the standard to be clear and useful in some areas and frustratingly obscure or underdeveloped in other areas, especially when the rubber hits the road and I have attempted to put the framework into practice!

An area I have found lacking in the past is the use of iteration within the ADM (Architecture Development Method) and more specifically, its relationship to requirements management. It should be noted I come at this from an Agile mindset having adopted the principles of Agile development in the 90s. It is with this perspective (or viewpoint) that I examine the cycles that exist within the ADM and how they are related to the incorporation of business requirements into the overall architecture development process.

In the core topic on Applying the ADM, Chapter 2: Applying Iteration to the ADM, iteration is discussed qualitatively in some depth, however, the content remains abstract and qualified by the organisational context which might never be fully known or provide any useful direction. There is little prescriptive or quantitative guidance provided to answer questions such as:

• How long should an iteration last?
• How many transition architectures should be used within one pass through the ADM? And should any be used at all given we could iterate through the ADM as many times as needed as an alternative?
• How are iterations affected by changing business requirements? And should they be?
• How do we determine the number and scope of deliverables that should be an output of each iteration? Too fine-grained and we risk a large number of iterations, too coarse-grained and we may end up with a BDUF approach with long timeframes between outputs.
• How many times do we iterate? And does the nature of the iteration (e.g., phases included, flow through the phases) affect this?

It is good to see some expansion and clarification on this topic in TOGAF Series Guides; in particular the TOGAF Series Guide: A Practitioners’ Approach to Developing Enterprise Architecture Following the TOGAF ADM. Section 5.2 discusses iteration at length with Section 5.2.3 providing some useful insights around the non-linear nature of the ADM, a principle that must be internalized by every practitioner if they are to fully benefit from the ADM. Yet, the above questions remain largely unanswered, although we do get some hints in Section 5.2.4 which talks about a 3-to-10-year outlook for developing the architecture to support strategy. There are some useful example project Plans including timelines hinting at the order of magnitude of these cycles through the ADM. Notably, the Sample Project Plan to Develop Architecture to Support Project shows a 5-month duration for this traversal through the ADM.

So, what do we make of all of this? Before answering this question, I believe we need to consider an intersecting aspect within the ADM that has a major impact on iteration - requirements management (see Architecture Development Method, Chapter 13: ADM Architecture Requirements Management).

Requirements Management in TOGAF

I always found this topic the most confusing and convoluted within TOGAF. The first thing to note is the following quote from the TOGAF 10 specification:

The ability to deal with changes in requirements is crucial. Architecture is an activity that by its very nature deals with uncertainty and change — the "grey area" between what stakeholders aspire to and what can be specified and engineered as a solution. Architecture requirements are therefore invariably subject to change in practice.

Also from the specification:

It is important to note that the Requirements Management circle denotes not a static set of requirements, but a dynamic process whereby requirements for Enterprise Architecture and subsequent changes to those requirements are identified, stored, and fed into and out of the relevant ADM phases, and also between cycles of the ADM.

My interpretation of the above, which may be different from other TOGAF practitioners, is that architecture requirements are subject to frequent change throughout any iteration through the ADM and that in fact can change within a single phase of the ADM and between cycles through the ADM. In effect, they can change frequently.

This, in my view, does not align well with Agile practice. Time-boxing (for sprints or iterations) is an important concept in Agile thinking as is the important restriction that (from Scrum):

Once the team has committed to a sprint backlog, the task work begins. During this time in the sprint, the team is protected from interruptions and allowed to focus on meeting the sprint goal. No changes to the sprint backlog are allowed; however, the product backlog can be changed in preparation for the next sprint.

This constraint on change applies to requirements too, and there is an important reason for this. Too much churn in requirements leads to chaos; more specifically, it can stop the delivery of a "potentially shippable product increment", in Scrum terms. In other words, for the duration of the time-box (typically 1 to 4 weeks) no change to requirements is allowed.

This begs the question - where is this constraint in TOGAF? As far as I can tell, there is none. Requirements are allowed to change throughout the ADM, within all phases, without constraint.

I have written before on the quicksand that is requirements management; unless the term "requirement" is very narrowly defined it will be interpreted as it seems to be in TOGAF and in the wider community as any need whatsoever. Everything that is needed becomes a requirement. This model is broken on many levels.

The first, is that there only one type of requirement that matters and that is the business requirements which are linked directly to the business objectives and goals of the TOGAF transformation program. TOGAF instead refers to many types of “requirements”:

Deliverables in later ADM phases also contain mappings to the design requirements and may also generate new types of requirements (for example, conformance requirements, time windows for implementation).

It may be semantics, but these, to me, are not requirements at all but design constraints. Conformance requirements, for example, define criteria for the architecture to conform to, and can be derived from business requirements but equally could arise from policies, procedures, or legislation. They are a design constraint, for sure, but not a requirement in the sense that they reflect anything that matters to the wider business audience.

To say that these must be managed as requirements is not only misleading but dangerous; it implies that once created the target state architecture must meet these requirements to be considered compliant. This constrains the final solution to an artificial requirement which over time may diverge from the business requirement that spawned it in the first place (or changes in policies, procedures or legislation). It may also contradict other design decisions.

In practice, it is difficult to impossible to manage these fine-grained requirements over the long-term (having seen this fail in several organisations where I have worked has convinced me of this) and becomes a burden that cannot be sustained in the end. Instead, what we really need is design artefacts produced by each phase, which capture what is traditionally referred to as functional and non-functional requirements. This allows the architect to keep the architecture aligned with business requirements, rather than fighting to keep requirements aligned with each other, a potentially enormous drain on time and resources.

Let me try to explain this with an example. Let's say we have a business requirement generated in Phase A of a digital transformation program that states:

The target enterprise architecture must ensure all business processes that deal with client management within the organisation are automated to the greatest extent possible, are secure according to corporate security policies and fully integrated with the new CRM system being introduced into the organisation (the new CRM is replacing a legacy technology, that has been in use at the organisation for over a decade).

This business requirement will spawn many functional and non-functional "requirements"; for example, a non-functional requirement could be: “The new CRM (Customer Relationship Management) system implements SSO for all workflows using a user's existing network login credentials”. The question then becomes should we manage these as TOGAF recommends, within the scope of the Requirements Management phase? I believe not - this treatment is reserved only for business requirements aligned to the goals and objectives of the transformation program. These non-business artefacts are design artefacts (namely, constraints, assumptions or pre-requisites) and should be in the ADD (Architecture Definition Document) not the ARS (Architecture Requirements Specification) or Architecture Repository.

The distinction is subtle but important; as a design artefact it simply becomes part of the target architecture specification. There is no need to update it or manage it in the Requirements Management phase of the ADM. Because it is designed to address a business requirement it is updated in line with any changes to the business requirements, as any architectural deliverable is.

But I digress somewhat.

Back to the original issue - allowing requirements to change at all times within the ADM. If we restrict this only to business requirements as described above, then we can better interpret the following statement from Section 13.5.2 of the TOGAF standard:

During the phase execution, new requirements generated for future architecture work within the scope of the current Statement of Architecture Work need to be documented within the Architecture Requirements Specification, and new requirements which are outside of the scope of the current Statement of Architecture Work must be input to the Architecture Requirements Repository for management through the Requirements Management process.

This works if we reinterpret this to read:

During the phase execution, new business requirements generated for future architecture work within the scope of the current Statement of Architecture Work (SAW) need to be documented within the Architecture Requirements Specification, and new business requirements which are outside of the scope of the current Statement of Architecture Work must be input to the Architecture Requirements Repository for management through the Requirements Management process. Regardless, business requirements outside the scope of the current SAW will not be included in the current ADM iteration.

In effect this locks in the business requirements for this cycle through the ADM and its associated SAW. Any new out of scope business requirements will be allocated to subsequent iterations. This constraint prevents excessive churn that could cripple the whole architectural effort, similar to how out of scope backlog items in scrum are not allowed to be worked on until the next sprint begins. It time-boxes the ADM cycle.

Of course, the above constraint does not stop us from cancelling the iteration and starting a new one through the ADM which does include these new out of scope requirements, but this has serious implications in terms of wasted time and effort. And it would certainly need to go through Phase H for handling major changes in an orderly and disciplined way.

The Iteration Game

Let's revisit the iteration questions from above. How long should an ADM iteration last? A business transformation program typically lasts from 3 to 10 years so aligns well with the TOGAF Architecture to Support Strategy ADM cycle. Within this program we conduct several iterations through the ADM. The question then becomes what is the minimum time and scope for one iteration through the ADM?

Given the number of activities and phases within the ADM, a team would struggle to conduct all of these in less than 6 months. Consider Phase E alone; COTS (Commercial Off the Shelf) selection and options analysis can take months in itself. I am basing this on experience, but I would say anything under 6 months would be very ambitious. Maybe we should specify 2 years as the minimum? The problem with this is funding and the fiscal year. Most organisations fund projects and programs on a yearly basis. Anything that takes longer than that is difficult to fund and even worse, easy to cut if nothing is delivered within that timeframe. We want to deliver a working and deployed architecture at the end of every ADM iteration. It follows then that 6 months would be the perfect timeframe, since we can have 6 iterations in a 3-year program. This aligns well with the Architecture to Support Project timeframe from TOGAF.

What about transition architectures? This seems to be a legacy of waterfall thinking; it assumes that we have a fixed immutable project or program plan that we divide into shorter timeframes to see some results sooner or implement quick wins (or both). Each transition architecture is determined up front, and each is executed until completion of the project or program without any retrospectives. Personally, why not just cycle through the ADM again instead? This would guarantee that any new business requirements or changes in the solution context are included in the scope before the next cycle beings. Transition architectures make sense if our time to initial delivery is long; but we have guaranteed they will not be if we stick to our 6-month iterations. So why not just avoid these completely? Open to debate on this one.

How iterations are affected by requirements has been covered above. Business requirements are fixed for the during of an iteration. Derived technical (functional or non-functional) "technical requirements" will of course emerge and those in scope should be treated as part of the target state design.

By defining timeframes and fixing scope over an iteration (sprint) through the ADM we can now size the deliverables for each sprint to whatever can be realistically achieved in that timeframe, given available resources.

Finally, how many times do we iterate? As long as we stick to a 6-month iteration over the ADM as a whole, iterations between phases and within phases would last in the order of days or weeks.

Conclusion

In a nutshell, TOGAF lacks important guidance in the use of iteration within the ADM. In this article I have tried to extrapolate what is in the specification, combined this with personal experience and common sense to arrive at this missing piece. Granted, some of the assertions I have made here are controversial, however, I have found them useful in the practical application of TOGAF. I would be interested in hearing from other perspectives or other people's experience with iteration and requirements management in TOGAF. For me, ensuring Agile Principles are baked into TOGAF and the ADM is critical to its success in practice.