How do you choose the right message queue for your system architecture?

There are many tools and cloud providers for the queue implementation. Instead of focusing on performance, first verify and validate the compatibility of queue providers with the proposed solution. Once shortlisted providers are identified, it should be cross checked with functional requirements of the system which includes throughput and latency.

Queue is a channel between source and target, if target is down or due to network issue not able to connect it should persist the data. System design will need different types of integration patterns and it's not possible to have different types of queue providers for each solution within a system. For one integration you will need guaranteed delivery but for acknowledgement it might not require it. So it's essential to understand the types of channels you are going to use in your system architecture.

When selecting a message queue for your system architecture based on message durability, prioritize queues that offer persistent message storage and delivery guarantees, ensuring messages are not lost even in case of system failures. Assess durability features such as message replication, acknowledgments, and fault tolerance to maintain data integrity and reliability.

Consider the scalability requirements of your application, both in terms of the number of producers/consumers and the message volume. Look for message queues that can scale horizontally by adding more nodes to the cluster. Distributed message queues like Apache Kafka and Apache Pulsar are designed to scale well. Evaluate the ability to scale consumers independently of producers to handle varying workloads.

The message queue should be scalable with evolving architecture and technology. It should provide vertical (max element in queue) and horizontal (adding more nodes) scaling. Assessment of changes should be done, how will it impact existing systems by scaling? What configuration changes will be required? If required, can we replace it? So scaling is a crucial part for selection of a queue.

Assess the specific features required by your application, such as message routing, message filtering, dead-letter queues, and message replay. Consider whether you need built-in stream processing capabilities (e.g., Apache Kafka Streams, Apache Pulsar Functions) for real-time data processing. Evaluate the supported protocols (e.g., AMQP, MQTT, JMS) and client libraries for different programming languages.

Queues are playing a significant role in distributed systems and micro services. They provide loosely coupled systems along with parallel processing with ease. There are many more features to be explored before choosing a specific message queue. As it's challenging and cost intensive to replace or redesign products once bought for a system design purpose. Try to explore and take leverage of open source products like Solace and Kafka for PoC. Once you find a match as per your functional and non functional requirements; proceed with it

Consider the feature set of message queues, focusing on capabilities like message prioritization, message expiration, message filtering, and support for different message formats. Evaluate how these features align with your system requirements, scalability needs, and integration with other components to ensure seamless communication and efficient message handling.

It all depends on the system requirements. Consider all the mandatory and optional features you needed to build it. Verify and validate requirements with the Message Queue providers using documents or contact product suppliers. It's a time consuming and tedious approach. Most organisations are re-using existing message providers instead of upgrading or utilising the latest solutions.

Determine the level of reliability and fault tolerance needed for your system. Consider factors like message delivery guarantees, automatic failover, and data replication. Look for message queues that offer high availability through clustering, replication, and self-healing capabilities. Evaluate the ability to recover from failures and resume message processing without data loss.

Consider the cost of running and maintaining the message queue infrastructure, including hardware, networking, and storage costs. Evaluate the pricing model of managed message queue services (e.g., Amazon SQS, Azure Service Bus) and compare it with the cost of self-hosting open-source alternatives. Factor in the operational costs, such as monitoring, administration, and support, when making the decision.

One of the major constraint in system design is budget. According to budget and operational cost optimal messaging solution should be selected. Latest and mostly advertise solution are not the best, which is fulfilling your requirement are the best. Choosing niche technology/products add additional cost to the project, so choose wisely.

First thing to take into account is to understand the requirements you really have. Usually, people believe that all queues do the same and all have the same use cases, but that is far away from the reality. Each queue has its own use case and each one is the best for a specific use case. Understand requirements, understand possibilities and then, select the one you need to cover requirements

Choosing the right message queue involves assessing requirements, considering performance, supported protocols, scalability, reliability, ease of use, community and support, and cost. Evaluate these factors to ensure alignment with your system architecture's needs and goals.

La elección de la cola de mensajes adecuada implica evaluar los requisitos, teniendo en cuenta el rendimiento, los protocolos compatibles, la escalabilidad, la fiabilidad, la facilidad de uso, la comunidad y el soporte, y el coste. Evalúe estos factores para asegurarse de que se alinean con las necesidades y los objetivos de la arquitectura de su sistema.

Do you need decentralization in your architecture? Centralized brokers may not fit use cases like wireless ad hoc networks. You also want to understand encryption and compression options. For example Kafka offers both in addition to controlling how many messages are sent with a Kafka packet. This can significantly affect message overhead.