What are you doing about technical debt in your portfolio?

Technical Debt is Inevitable. But, how do you manage it?

The debt starts accumulating once a technology starts getting used, configured, deployed etc. With every piece of code, there is debt, no matter the programming language, the software stack, coding style, development methodology etc. What are the fashionable trends today in technology will one day be out of favour. Talent moves on. Old coders and maintainers become obsolete or retire. This is not as grim as it appears to sound. The first step is the acceptance of this reality. Without this foundation, we can't future-proof the IT landscape. More than 30% of the IT budgets focus on addressing technical debt. Here are some key causes of technical debt and methods to prevent accumulation based on my experience and discussion with CIOs & CTOs of organs primarily dealing in financial sector.

What is technical debt?

There are many ways to visualize the technical debt in any IT system. At its core, it is an invisible aspect that creates negative value to the overall IT and business landscape i.e. it brings down its value to be resilient and optimal to changes. Some pay attention to quality of code, people, overall architecture etc. In my experience, the best way to understand holistically is to visualize it as the cost differential between optimal cost of change and actual cost of change.

Note as systems mature, evolve and change, the cost of change increases disproportionately. But, is all the technical debt to be addressed and removed? Not necessarily. Evaluate them using a cost-benefit analysis. In this instance, compare the business value versus remediation of technical debt. There is a threshold level beyond which remediation of debt doesn't add any value. The key is to identify that, lock it and maintain the portfolio around it. But this needs to be recalibrated annually or as and when the IT landscape undergoes major transformation.

Causes of Technical Debt

Overall, the following chart captures all the dimensions of its causes.

The most common causes are described as follows:

1. Outdated software stack
2. Unmaintained software
3. Soon to be obsolete stack
4. Hanging packages - this is a broad category that includes unsupported packages from vendors or heavily customized ones. The vendors may go out of business or disappear after a merger. The version, currently run, may not be supported and the product vendor may not be in a position to provide an upgrade path without too much retrofitting. Or worse still, as I have seen more often, the package is so customized that it ceases to resemble the product (this happens when the organizations buy the product and fit their current processes onto the product thus perpetuating status quo)
5. Trends in programming languages And frameworks. Python replaces Perl, Go replaces Python, Rust replaces Go and so on. The same is true for frameworks. Ruby on Rails used to be in, but now Vue.js is a more hip framework.This is again inevitable. But the problem is that most technology geeks want to ride on the current bandwagon without bothering to see if it is indeed required.
6. Aging cloud infrastructure. Cloud migration simply transports AS IS set up to cloud. IT doesn’t optimise by itself. So, just because you’re in the cloud doesn’t mean you don’t have technical debt. When early cloud adoption began, many core services we take for granted today, like authentication and authorization, weren’t offered by the cloud service providers. Some of the first-ever cloud migrations were stitched together with homegrown tooling, which may not be aging well. Or, organizations may have “lifted and shifted” legacy services into the cloud. Operators overseeing these older cloud-based environments will likely feel the pressure to refactor into more modern, cloud-native styles.
7. The proliferation of microservices and APIs. Furthermore, cloud services retire features all the time. Thus, it can be challenging to keep up with these minute changes to retain backward compatibility for things like API versioning shifts. In the DevOps world, web APIs are the glue that connects our tools. But in an increasingly complex landscape, it can become easy to produce shadow IT and lose track of dependencies.
8. Legacy physical infrastructure: It’s not just the software that becomes debt; the systems and hardware it runs on might also become obsolete. For example, the network itself might be full of old edge switches and wireless access points. An enterprise may hold onto aging on-premises servers. (This is often due to the depreciation benefits of using a capital expense accounting model as opposed to an operational expense model.)
9. Rushed decision-making
10. Skills and resource gaps
11. Failure to implement DevSecOps.

Technical debt gets introduced during development or support of IT landscape either deliberately or inadvertently. Some will involve a decision between choices of delivering on time and living with a certain amount of known debt. Others could have been introduced recklessly. Here is a simple way to look at it using the dimensions of Level of Prudence and Intentionality.

In any case, knowing or estimating the cost of technical debt becomes important.

Impact of technical debt

• Threat to innovation
• Cost of upkeep of the IT landscape increases disproportionately
• Risk of not getting the required support from product vendors
• Risk of not receiving automatically upgrades especially the ones that support changing regulatory requirements
• Loss of overall KSA - Knowledge, Skills, Abilities
• Heavy dependence on key aging resources with niche skills

What are the ways to address / mitigate Technical Debt?

• Treat technical debt like a financial debt. In other words, this needs to be assessed, converted to a $ figure, budgeted, forecast, monitored and end of year stance known to everyone. Note the difference between both.

• Refactor legacy code. Try to use AI tools for this. E.g. PEARL code to JavaScript can be done easily. Do not blindly convert to trendier languages and stack that may not last. Today’s AI models may drive you towards it. Keep a balanced check
• Remember that DevOps and Cloud are not a panacea to all the ills of IT landscape. Do it over rely on them. Do not assume adopting such practices will lessen technical debt automatically. DevOps helps you to delivery multiple releases and features quickly but it will not eliminate technology changes.
• Constantly maintain an updated IT register that includes all hard aspects (Software stack, Hardware, Operating System, Product version, DBMS, Front end, Middle, Back end, Languages etc.) and soft aspects (SDLC methodology, architecture and programming guidelines, product customisation guidelines, knowledge matrix required for the core jobs, knowledge vs personnel etc.). This should be current not lagging by more than 3 months at any point in time. This can act as a single source of the causes of technical debt.
• Do not proliferate the technology across the organisations / locations / departments / functions. This, in no way, will stifle creativity. You should remain lean whenever possible. No unnecessary technology. There could always be a lean team that experiments with technologies, fails fast and recommends. That is fine but the entire stack of the organisations should be a very short list of technologies to the extent possible. This is where the CTO orgnaization's guidelines will have to be enforced strictly.
• Monitor the ratio of customised/bespoke software vs COTS (Commercially Off The Shelf) products. A healthy balance could vary from 30% bespoke to 50% bespoke. Where COTS product is used, strong guidelines should ensure their usage and compliance. For e.g. the level of customisation cannot exceed a certain %, the version should be no less than n-2 (I.e 2 versions behind the latest by the vendor), monitoring of the product vendor/knowledge etc.
• Carefully vet introduction of any new vendors/partners/providers that have the capability to alter the landscape.
• Perform a strong portfolio cost-benefit analysis for introduction of any product/package/tool etc. This rigour is no less than assessing a new financial venture.
• Continually maintain updates of dependencies. Today’s software infrastructure is largely composed, at its foundation, of open source software. And if these packages aren’t mature and well-supported, you may be sitting on an unstable house of cards. It’s a good idea to continually audit your OSS use and respond to updates in a timely manner.

To sum up:

• Technical debt Is inevitable. It starts as soon as any IT activity begins.
• The code becomes older and some of the choices you made in implementing may be outdated. This means systems need to be updated, segments of code to be rewritten. It is no different from financial debt in a way. Mature organizations are running their technical debt as they treat financial debt - with the same rigour.
• Amongst the 5Rs of modernization, RETIRE is also to be used effectively. Typically 20-30% of work of any IT landscape can be shut down with very minimal impact. One can do agile devt to get to a feature or product faster but with a more traditional monolithic architecture, you spend time focusing on the outcome and the scale.

• You don't have micro service or containerize everything. Not all clients need to be mature in line with the pace of technologies.
• Tech debt is like credit card debt—easy to get into and tough to get out of. And just like its financial equivalent, you shouldn’t go too far into technical debt.
• Ask how much stuff are you using and why? It’s become a trope, but the principle of KISS (keep it simple, stupid) bears repeating in software architecture dialogues. Because if you reduce the surface area, it will make ongoing maintenance easier and reduce the potential of cyberattacks.
• All 3 dimensions - People, Process and Platform sides should be considered.