How can you design a low-latency software architecture?

Systems that are latency sensitive design can vary significantly based on the latency requirements. The latencies can range from ~100 uS e.g., Stock Exchange execution to ~100 mS e.g., real time fraud detection in payments to ~1 second which is typically high performing web systems. The latency requirements should be defined in SLA measured over percentile e.g. < 100 ms for 90th percentile & < 500 ms for 99th percentile. By having concrete latency requirements we can design the software architecture that will meet these requirements.

Apart from the requirement, we need to be aware of the near ideal latency, realistic latency achievable. But realistic latency may not be defined in isolation. There will be many parameters affecting it. How much processing power, and memory you can throw at your service to run this software, how efficient you want your system to be, which can also be influenced by its cost and the availability of the resources not just to your service and team but also in the industry and market. Remember that we can't be providing the 100% perfect SLAs, we need to find a balance between how fast we can ship something, which is useful to the client/users at this moment, without much compromise to the user experience and cost.

A common technique in low latency architectures is to pre-compute results wherever possible and retain it in primary memory so that there is no I/O or computation required at request time. Results may be retained in the serving process's memory or across a process boundary in a service such as Memcached or Redis. An example of the former approach is indexing in single instance database systems; this approach provides the maximum latency benefit. The latter approach of using an off process cache, exhausts some latency budget for the benefit of being reusable across multiple service processes; this approach is commonly used in web serving clusters.

Based on the requirement the right architecture is to be selected e.g., for very low latency requirements most systems are event driven and written grounds up with C, C++ or GoLang that gives control over how operating systems handles memory, IO etc. rather than a garbage collected language like JVM languages. For latency sensitive web systems any of the web technology can be chosen however it is software architecture that defines how it will perform.

Network & communication is critical to latency management. For low latency application the network IO should be eliminated or limited. For web interactions WebSockets is a good option that provides open socket channel and can enable low latency experience on web browsers e.g. Stock Trading Terminal. Another key approach is to manage the HTTP protocol e.g. by setting NODELAY to ensure the latency is not introduced by optimisation done by Nagle algorithm.

Concurrent design must be accompanied by also splitting computation into parallelizable units where possible. Long running transactions by their very nature will be high on latency. In order to improve latency, transactions may be broken down into workflows where tight consistency requirements may be relaxed in favour of an eventually consistent data model. Of course this may not be applicable in all domains e.g. financial transactions.

Microservices have become default for many organizations.. but quite often they try to implement strong consistency model. with distributed services and designing for strong consistency compromises on availability, latency & scalability.

So many projects I evaluated as a consultant were very tightly coupled. Everything had to happen in order before any results were published. Moving towards an architecture that is multithreaded and loosely coupled is a great way to scale. One example I can think of is Google BigQuery. You can query data that is still enqueued to be placed in the table, so you don't have to wait for it to be fully processed before you query it. Very cool!

Observability is key to measuring & monitoring the latency and ensuring the system requirements are met. Some steps that can help - Log each system latency inner wall to wall - Log latency of each network separated component in the stack - Use correlation keys and distributed tracing to track the performance end 2 end in a complex system - Beware of heisenberg principle

An oft-overlooked possibility is to offload computation to clients so that more compute cores and memory are employed overall. For example, in AI/ML based applications, inference may be offloaded to clients which have local access to the latest version of the model. This will cut latency by eliminating round-trips to the server. Cutting down network round-trips almost always provides significant latency benefits especially for end-user facing applications.

If min. latency is required among services, consider the use of Webflux( Springboot). If min. latency is required among client and server then think upon using sse( server sent events), in case consumer is browser based. It will minimize the latency.

Architecture is more important than tools. Define specialized processing units. Define the communication protocol and interface. Define the on demand scale up. Define the on hotspot relocation. Identify the critical process. Introduce competitions in implementation. Implement isolation to prevent communication interference.