How do you tune the performance of a legacy or a outdated system or application?

A systematic approach is needed: Start by documenting performance metrics to understand the system's state, using tools like VisualVM or New Relic to identify bottlenecks and set benchmarks. Refactor algorithms, optimize database queries, and update deprecated functions. Evaluate hardware, consider server upgrades or cloud migration, and optimize network configurations. Optimize the database schema, implement indexing, and consider partitioning large datasets. Use application-level caching and in-memory solutions like Redis. Address memory leaks, optimize memory allocation, and tune garbage collection. Enhance concurrency with parallel processing and multi-threading. Update libraries and frameworks to leverage the latest features.

The first question to ask is what has changed? I generally get called in after an application that has been fine for years suffers degraded performance. It could be a recent OS upgrade etc etc has lead to the degradation. Once you find that out, it might not be possible to revert (security concerns, support etc) but at least you get a better idea of what your options are.

For legacy systems such as CICS and DB2 because of the fantastic built in metrics it’s often not necessary to run under load it’s much more important to ensure that all functions are executed and have a CICS /DB2 performance engineer who can interpret performance output of the results that should now be used to provide a performance baseline and help establish which features/functions should be optimized

This is the most important step in my opinion, as most legacy (Heritage) mainframe system & applications, are likely to be running well against their business or technical requirements. You really need to discover if there really is an issue with the systems or applications, I.E. CICS/DB2 transactions normally still run at 99% within x milliseconds and it is network & app/screen render that takes most of the time. So just because a system or application is getting on a bit in years, is it really an issue? And is improving it by a few milliseconds or even a large percentage, really going to make much difference to customer/business experience? I suggest that in most cases, ‘older’ system & applications could just be left alone.

I would use an 80/20 rule here. 20% of queries probably generate 80% of the load. Start with the ones generating the most waits (avg wait in seconds * # of executions) or better yet use a monitoring tool to find the biggest culprits. For a newer system I would do a load test to simulate load and extract results to compare against baseline.

In my experience, modern observability tools will not give you all the data you need. Often the tech stack of a legacy application is incompatible with modern observability tools. You will need to go old skool and use the native OS tools, logs and even network sniffers to find out what really is going on.

As mentioned earlier it helps to have a monitoring tool. Sometimes changes have unintended consequences and you might find out that one change creates a ripple effect and slows or blocks another process. Usually a faster query is less likely to create blocking as the resources are released quicker. Goal is to eliminate problematic waits on IO, network, page and memory.

Sometimes when tuning legacy systems less is more! These are systems built when resources were smaller than today. It may seem counterintuitive but in some cases adding more resource to the app may have a negative effect. The classic is adding more CPU to a server when the application has single-threaded components. The better option might be to scale horizontally. So, it is important to understand how the application has been built so you can make tailored recommendations.