Agile Software Development Methods (Part 1)

What has been is what will be, and what has been done is what will be done; and there is nothing new under the sun. – Ecclesiastes 1:9

Most of the activities found in Agile processes can be divided into two groups – Individual practices and Group practices. Deploying the individual practices is relatively simple in most organizations. It is more difficult to deploy group practices, especially those practices that involve teams beyond the development team. The Individual practices, largely geared towards developers, are well–known best practices that are performed by any mature organization.

The Group practices require customers, stakeholders, end-user, external system interfaces, technical and business managers to do things they don’t normally do, are not proven, or that results in a perceived loss of control.

The Problem to be Addressed

The nation's prosperity depends on software, the hardware infrastructure that supports this software, the networks to connect this software, and the security and safeguards of the software and its data. The software industry depends on the timely delivery of high-quality, secure, and safe products to customers. This industry is slipping further behind in quality and timely delivery every year, along with security issues. The gap continues to grow. One question that needs to be answered is – which approaches – in the presence of risks to success – of managing software projects or developing software intensive systems can be applied to this problem and others encountered by organizations dependent on software for their success?

Methods impose discipline in software development to make it more predictable, effective, and efficient. Founded on engineering disciplines, software disciplines have precisely defined processes by which software is developed, acquired, installed, and operated. As new methods are developed, agility, lightness, and adaptability are considered more important than the historical engineering-based methods.

Which of these agile methods should be applied to which problems are a question asked by IT Managers. The answer is not obvious (although the agile advocates would want us to think it's their specific approach - SAFe, Disciplined Agile, and a plethora of others - are confusing and sometimes misleading.

Two Definitions of Agility

The use of the term Agility can be confusing, but two different interpretations have become clear. The first is a popular phrase describing the use of practices and technologies to produce software. Aside from some claims of better quality and shorter delivery schedules, the external appearance of the software products has no distinguishable markings of having been produced by an Agile process.

The second definition of agility is more focused and deals with the problems and needs that the software industry experiences and therefore part of the notion of a next-generation development organization. This definition is concerned with the fragility of current development practices – for example, the inability of anyone method to satisfactorily deal with change, uncertainty, and unpredictability arising from disruptive factors in the business environment.

The Meaning of Agility

Businesses, services, manufacturing, and infrastructure are experiencing a common phenomenon – continuously increasing rates of change, uncertainty, and unpredictability in the business environment. Large uncertainties and unpredictability are fast becoming the norms of everyday life.

The problem is then identified as the business's inability to cope with large uncertainties and unpredictability. Traditional business practices are rooted in the assumption of certainty and predictability. If things change at all they do so only slowly.

In an emerging business environment, future requirements are unknown and will only become known in the future. These requirements cannot be known in advance or specifically planned for. The organization must deal with them as they arrive. Today's businesses don't operate on this basis. People want to know what tomorrow will bring so that they can be ready with a response. Businesses don't like surprises because surprises can be dangerous. Unexpected changes can harm a business because enterprises are fragile – they are easily damaged and broken by unexpected and unpredictable changes and events.

So what is an agile software process? The simple answer is – an agile process is one that is not easily damaged and broken by unexpected and unpredictable changes and events. How is this achieved? By allowing the participants to rapidly adapt to these unexpected and unpredictable events. Agility means organizations and processes that rapidly adapt to tomorrow's surprises. Conceptually this is what an agile process needs to do – but this is not easy since it requires the development of a new set of engineering and business management practices as well as changes across the enterprise.

At this point you should now see that agility is not just about the speed of response – it is about rapid adaptation. In other words, enterprise elements adapting in response to unexpected and unpredictable changes and events, new opportunities and customer requirements, new technologies, and changes to industry structures, regulations, and technologies. By enterprise elements, it is meant goals and objectives, technology, organization, etc. This leads us to the definition:

An agile organization is fast-moving, adaptable, and robust. It is capable of rapid adaptation in response to unexpected and unpredicted changes and events, market opportunities, and requirements. Such an organization is based on processes and structures that facilitate speed, adaptation, and robustness and that delivers a product or project capable of achieving competitive performance in a highly dynamic and unpredictable business, government, of technical environment.

Numerous reports have analyzed why software projects fail, agile and traditional. [2] Here's a summary of those findings, ranked by importance:

The majority of the items associated with both the success and failure of projects are related to the management of the software process. This includes stakeholder participation, clear requirements from the stakeholders, stabilizing these requirements, tracking these requirements, and verifying that the requirements have been fulfilled in some way. One of the tenants of the Agile process, especially eXtreme Programming is that the requirements will become clear as the project evolves and that changes in these requirements can be absorbed by the development team since they are continually redacting the code anyway.

This in the challenge of discussing Agile processes — are they appropriate for the general population of software development projects and products, or are they suited for specialized environments in which the requirements can be discovered during the development process?

Are they appropriate in environments where safety and safeguard assure is mandatory? How about embedded systems, where updating, correctly, changing the behaviour of the systems can only be done when the system is shutdown and restarted?

The Taxonomy of Methods

The definition of a software process is embodied in how a software organization does business, that is, how management and engineering practices are implemented to support software development and its maintenance. This view of software process assumes that an organization has a set of building blocks that defines the general way it does business and that some subset of those building blocks is implemented for each software project.

- Software Technology Support Center, Hill Air Force Base, Ogden Utah

This business-centric definition of the software process assumes there are two basic project categories – project-centric and product-centric. They are both based on the delivery of some form of value for the stakeholder. However, there are different forces at work in each category. Understanding the difference between them is an important factor in choosing a method for the delivery of this value.

As an aside, Products are developed in companies through the use of projects when then is the need to account for the cost of that product in the General Ledger (FASB-86)

When You Anything About Any Agile Approach

No matter what the method is named, what the method claims to contribute to the field, and especially no matter what special powers are attributed to the method, there are a small number of essential attributes of any software development method must possess in order to deliver value to the stakeholders.

• An application domain – in which the problem exists. The generalization of a development process must take into account the business domain. Applying a specific method to the wrong domain can have the same effect as applying the wrong method to the right domain. Both result in some type of failure for the project.
• A conceptual model – that references a formalism in the domain of interest. These models are descriptive in nature and provide guidelines for making decisions about the problems encountered in the application domain.
• A formal model of some kind – establishes the rules for the acceptance and correctness of the method. These models are prescriptive in nature and describe the steps, actions, outcomes, and relationships between each of them.
• An implementation domain – which provides the means of solving the problems encountered in the application domain using a conceptual or formal model of a method.

All this formality is meant to bring light to the fact that the method's discussion is complex and fraught with confusing and sometimes contradicting terms, concepts, and claims.

Nine Best Practices of Software Development, No Matter the Method

We can and do, argue endlessly about which software development method, framework, toolkit, or whatever the current fully buzzword-compliant market pitch wants us to listen to.

But there are Nine Best Practices that must be in place for any software project to be successful that must be answered, starting with:

What are the attributes of a well–run project, independent of any specific method?

The first answer to this is to look at a higher level than any software development process, by looking through the eyes of a program manager, a project manager, or a business stakeholder.

The Nine Best Practices constitute a management control system that provides a disciplined set of processes for containing and directing the complexity of a software development project — independent of the development method. [1]

The figure below describes the relationship between the Nine Best Practices and their outcomes. The best practice fit into four major categories:

1. Identify and correct defects and potential problems as early as possible
2. Plan and estimate all project activities and deliverables
3. Minimize rework caused by an uncontrolled change
4. Make effective use of the resources

Your first reaction to these statements is likely but aren’t you restating the obvious? The Nine Best Practices are in fact obvious. The problem comes when it is time to deploy them into the software development environment and attempt to reap the rewards. What seems obvious is actually difficult. It is this framework that will allow us to discuss Agile processes and how they fulfill the higher-level needs of a project and its sponsoring organization.

What's Next?

With this starting point, today's agile development and enterprise-class solutions will be examined from the point of view of the problem domain rather than the solution looking for a problem to solve.

[1] The Software Program Managers Network, established in 1992 by the Assistant Secretary of the Navy, is to identify proven industry and government software best practices and convey these practices to managers of large-scale system acquisition programs. These 16 Critical Software PracticesTM specifically address underlying cost and schedule drivers that have caused many software intensive systems to be delivered over budget, behind schedule, and with significant performance shortfalls. 

[2] The most spectacular Agile project to fail of recent times was the Affordable Care Action website.