Day 87 : Feedback Loops in DevOps #90DaysofDevOps

Feedback loops are a central part of any DevOps team and can greatly affect software delivery times. While many software development teams know that feedback loops are important, it’s easy to misunderstand these loops. An unwanted or long feedback loop can delay software delivery and compromise the relationship between development and IT operations. 

Read on to learn about what DevOps feedback loops are, their role in continuous integration and delivery, common misunderstandings about feedback loops, and how to optimize feedback loops in software development. 

What are DevOps feedback loops? 

A popular definition of feedback loops is that they are, as Ant Weiss writes, “sets of relationships between entities whereas a change in one entity causes a change in another entity and that change eventually leads to a change in the first entity.” In short, there are two or more things that are in a relationship with each other where each entity inputs feedback for the other. They are in a loop because the changes that occur as a result of the feedback between entities is circular. 

Feedback loops are important in DevOps teams, but the DevOps process is the perfect example of a feedback loop. 

DevOps, short for development (Dev) and IT operations (Ops) is a methodology for integrating development and IT operations in software development. Traditionally, the development team is responsible for writing software, and the operations team for deploying it. In any software delivery lifecycle there’s a feedback loop between development and operations, and DevOps teams aim to optimize and shorten that loop in order to avoid bottlenecks and improve communication.

In looking at the DevOps team as an example of a feedback loop, the development team produces a piece of software and delivers it to the operations team. The feedback is the software. The operations team then tests the software and finds some errors in the code that the development team must fix in order to improve functionality. This feedback loops back as the development team implements changes and delivers these updates to the operations team.

Systems thinking is a way of thinking about a whole rather than discrete parts, and feedback loops are a part of that thinking. There are two different types of feedback loops within the software lifecycle: amplifying and balancing. 

Amplifying feedback loop 

An amplifying, or reinforcing loop, is a feedback loop of accelerating change. With this loop, change in the first part of the loop creates the same direction of change in the other entities, which in turn causes even more change in the first entity. 

It’s important to note that this kind of feedback loop can accelerate either positive or negative change. For positive change, more leads to more — which leads to even more! In a negative amplifying loop, less leads to less, which leads to even less. 

This feedback loop is called amplifying or reinforcing because the feedback loop results in incremental advancements as the loop continues. 

A good example of an amplifying feedback loop is climate change. Rising temperatures melt the ice caps which in turn exacerbates the effects of climate change and causes temperatures to rise, which further amplifies ice melting. 

Balancing feedback loop 

Unlike an amplifying feedback loop, a balancing feedback loop causes the system to stabilize to a point where, ultimately, there’s no change. A thermostat is a perfect example of a balancing feedback loop. If your thermostat is set to 70 degrees, a change in temperature below 70 will provide feedback to the system to turn on the heat. The heat will continue until it reaches 70 degrees, at which point, the heat will shut off. 

The role of continuous integration and delivery in feedback loops

Continuous integration and delivery is crucial for healthy feedback loops in DevOps teams. Continuous integration (CI) is a way to consistently build, package, and test software. Development teams work on small batches of code and continuously upload them to the central repository for deployment. These small batches of code provide continuous feedback for the DevOps team to then conduct unit testing. Developing in this way allows for operations to quickly identify bugs and integrate new code for continuous improvement. Development teams use artificial intelligence and machine learning automation tools to detect and correct errors in the code.

Continuous delivery (CD) is a way to automatically push code changes to different environments. It’s an easy way to consistently deliver changes and updates to both developers and users. 

This kind of piecemeal approach to development and delivery is perfect for microservices development that enables the rapid delivery of complex applications. Microservices development allows for the incremental release of complex applications.

Using the right feedback loop technology can speed up your pipeline and reduce inefficiencies. CI and CD pipeline tools are at the core of a DevOps feedback loop. Azure DevOps offers several options for streams of feedback, which is useful for tailoring your pipeline to your organization. Another useful CI tool is CircleCI that automatically creates dashboards when programmers commit code. It does so by integrating with GitHub and BitBucket.

Continuous integration and delivery are great methods that facilitate a shorter and more efficient feedback loop, but they aren’t the only place feedback loops occur in DevOps teams. Understanding where and when feedback loops occur can go a long way in optimizing a long feedback loop and avoiding unwanted feedback loops. 

Common misunderstandings of feedback loops 

Shortening the feedback loop is a common goal of DevOps teams, but some teams fundamentally misunderstand what a feedback loop is, and where it occurs. In an overview of feedback loops published in Medium, Ant Weiss identifies some of the most common misunderstandings of the feedback loop. 

Teams treat notifications as feedback loops

Many DevOps teams automate notifications and alerts for code errors, schedule adherence, and more. The problem arises when teams equate notifications with a feedback loop. An alert is just a piece of information, and someone must act upon it in order for it to be part of a feedback loop. In addition, some of the most important feedback loops occur outside of the team. It’s a mistake to think alerts and notifications are a complete feedback loop. 

Feedback loops are afterthoughts

Teams should consider feedback loops in the beginning stages of a project, and not as an afterthought. Value stream mapping should take into consideration how feedback loops fit into the project. Value stream mapping is a technique that DevOps teams can use to create a visual guide of all the parts necessary to deliver a product. A problem with value stream mapping is that teams often view the flow of feedback as trickling down only, instead of looping around. Team leaders may view bottlenecks as outcomes of specific team issues, instead of understanding that bottlenecks may occur due to an incorrectly applied feedback loop. If teams view  feedback loops as central to the value stream mapping process, it becomes obvious how they have the potential to balance or accelerate the application development process. 

Ignoring the loop

DevOps teams start a project with continuous integration and the best of intentions in place. As the project progresses and things start to inevitably fail, some teams make the mistake of adjusting their continuous integration pipeline, instead of researching the root cause of the failure. They will stop testing, lower the frequency of commits, and even turn off continuous integration. 

This occurs as a result of focusing on the feedback and ignoring the loop. Acting on feedback without considering the source creates a balancing feedback loop where teams are busy putting out fires. By understanding where the mistakes originate in the loop, teams can amplify their feedback loops. 

Accelerating feedback loops to streamline delivery

DevOps teams are under a lot of pressure to deliver products to the end-user as fast as possible. As a result, team leaders can be tempted to amplify feedback loops to achieve faster delivery. The problem is that not all feedback loops are good for the product or the team, and amplifying the wrong feedback loop can slow or stop progress. 

Some feedback loops are unintended and originate outside of the project itself. For example, sometimes stakeholders put a lot of pressure on development teams to meet very ambitious deadlines. This creates feedback that quality metrics aren’t as important as speed. As a result, teams create subpar coding which then provides feedback that it’s possible to deliver a project in a short amount of time. The unintended consequences of this feedback loop is that DevOps teams are under increased pressure to deliver in shorter timelines, which in turn continuously reduces the quality of coding.

Avoiding these mistakes is the first step in creating better DevOps feedback loops, but there are some proactive steps you can take to optimize your feedback loops. 

Optimizing feedback loops in DevOps teams

There are a few steps teams can and should take in order to optimize their feedback loops. 

Uncover all existing feedback loops 

Because feedback loops have such a large impact on DevOps teams, it’s important to uncover all existing feedback loops, both within the team and externally. Team leaders should hold regular value stream mapping sessions and encourage open dialogue. The conversation shouldn’t just revolve around what needs to get done, but rather what teams are already doing and how teams can improve workflows. 

By having these regular sessions, teams can gain awareness about what feedback loops exist, even if they’re unintended. Then, the discussion can move towards the best way to harness those feedback loops.

Do you need stability or amplification? 

Recall that there are two types of feedback loops: amplifying and balancing. Although much of the discussion is about amplifying feedback loops, the reality is that some systems need the stability of balancing feedback loops, and others need amplification. Consider if your system is actually creating positive change or if it’s too busy stabilizing the system by putting out fires and avoiding disaster. Typically, amplification is a great software development feedback loop, because positive change is important for developing and delivering the product. Balancing feedback loops, on the other hand, are more useful during product maintenance when there are no plans to introduce a new feature.  

Act on notifications

One of the best ways to avoid a long feedback loop is to ensure that all automated notifications are actionable and actually acted on. If you constantly have notifications going off but aren’t making changes as a result, then you may be misusing these notifications. Perhaps there are too many notifications for trivial reasons, or your notifications may not have enough information in them to be actionable. In addition, look at who’s in charge of acting on which notifications. It could be that the wrong people are addressing them.

Acting on notifications is essential in reducing your work-in-progress and technological debt. Technological debt is when a team makes a conscious decision not to optimize a specific piece of software for the sake of faster delivery, while planning to address the shortcomings after release. Acting on notifications as they come in reduces the backlog and prevents small problems from snowballing into large problems that add to technical debt. Avoiding technical debt helps to speed up the feedback loop because feedback can only occur on already developed parts of the software. In addition, keeping your work-in-progress list short is essential to receiving quick feedback. 

Focus on human-based feedback loops

DevOps teams often ignore human-based feedback loops — feedback loops where the feedback comes from a human source — because they aren’t as easy to control, identify, or automate. Human-based feedback loops are an integral part of a software development feedback loop and should be addressed. In addition, human feedback tends to be more targeted and quicker than even automated processes. 

Ask yourself if automation is enough to optimize feedback loops, or if there can be a human element as well. For example, linters are a great tool for automatically detecting errors in code, but teams are seeing great success in using pair programming as well. Pair programming is when two coders sit in a room and code in unison. One coder writes the code, and the other checks it for errors and provides suggestions. Pair programming can really amplify the coding feedback loop because it promotes creativity. In addition, while a linter can easily catch small errors in the code, humans are better able to catch larger errors and suggest different solutions. 

Another human-based feedback loop that can benefit your DevOps team is applying cross-functional teams. These are teams of people from different disciplines working together. Members share ideas and feedback directly with each other, instead of holding onto their ideas in silos, making cross-functional collaboration a highly effective tool for a software development feedback loop. As more teams are virtual, tools like Slack help cross-functional teams keep in touch.