RabbitMQ

Giriş
2007 yılında geliştirilmeye başlandı ve Erlang ile geliştirildi. Bu yüzden maalesef JVM içinde gömülü olarak çalıştırılamıyor. Açıklaması şöyle.

One of the oldest open source message brokers can be used with various protocols. Many web developers like this software, because of its useful features, libraries, development tools, and instructions.

In 2007, Rabbit Technologies Ltd. had developed the system, which originally implemented AMQP. It's an open wire protocol for messaging with complex routing features. AMQP ensured cross-language flexibility of using message broking solutions outside the Java ecosystem. In fact, RabbitMQ perfectly works with Java, Spring, .NET, PHP, Python, Ruby, JavaScript, Go, Elixir, Objective-C, Swift, and many other technologies. The numerous plugins and libraries are the main advantage of the software.

2013 yılında Pivotal Software'e katıldı. Açıklaması şöyle

RabbitMQ was originally created by Rabbit Technologies Ltd. The project became part of Pivotal Software in May 2013.

Protokoller

RabbitMQ bir sürü protokolü gerçekleştirir - AMQP, MQTT, STOMP. Ama en önemlisi AMQP protokolünü gerçekleştirir. AMQP protokolünün JMS'e göre avantajı şöyle

While Java has messaging standards like JMS, it’s not helpful for non-Java applications that need distributed messaging which is severely limiting to any integration scenario, microservice or monolithic. With the advent of AMQP, cross-language flexibility became real for open source message brokers.

Çünkü AMQP aynı Kafka gibi sadece byte[] taşıyor. Gerisi ile ilgilenmiyor. Açıklaması şöyle

In AMQP the messages are opaque entities; AMQP does not provide any standard way to encode/decode them.

However, web applications very often use JSON as an application layer format, that is, the JavaScript serialization format that has become a de-facto standard; in this way, the RabbitMQ client Java library can include some utility functions for this task.

On the other side, this is not the only protocol; any application can choose its own protocol (XML, Google Protocol Buffers, ASN.1, or proprietary).

Kurulum

RabbitMQ Kurulum yazısına taşıdım

Push Model İle Message Order Kaybedilir

Açıklaması şöyle

In RabbitMQ, messages go through an exchange, land on a queue then gets distributed to consumers in a way that a message can go to only one consumer. It is super easy to increase the replica count of consumers, but in this topology, you loose message ordering. Because the second message can be processed before the first one is completed.

Şeklen şöyle

Push Model + Acknowledgement

RabbitMQ Broker mesajları dinleyenlere kendisi push'lar. Mesajı alan istemcinin acknowledgement göndermesi gerekir. Acknowledgement şöyledir. 

Positive Acknowledgement

Spring terminolojisinde buna auto ve manual acknowledgement deniliyor.

In RabbitMQ, messages are stored until a receiving application connects and receives a message off the queue. The client can either ack (acknowledge) the message when it receives it or when the client has completely processed the message. In either situation, once the message is acked, it’s removed from the queue.

Prefetch Limit

Prefetch Limit kullanılabilir. Açıklaması şöyle. Prefetch count istemciye kaç tane mesajın push'lanacağı anlamına gelir.

It’s important to configure a prefetch limit in order to prevent overwhelming the consumer (if messages arrive at the queue faster than the consumers can process them). Consumers can also pull messages from RabbitMQ, but it’s not recommended. 

Negative Acknowledgement

Gerekiyorsa Negative Acknowledgement verilebilir. Açıklaması şöyle

A RabbitMQ client can also nack (negative acknowledgement) a message when it fails to handle the message. The message will be returned to the queue it came from as if it were a new message; this is useful in case of a temporary failure on the consumer side.

Şeklen şöyle. Tüketen taraf çöktüğü için bir başkası mesajı işliyor.

Negative Acknowledgement + Retry

Açıklaması şöyle. Bazı eksikler var. 1. Retry için delay verilemiyor. 2. Retry sayısı verilemiyor. dolayısıyla kutudan gelen Nack kullanımı yetersiz.

RabbitMQ provides some kind of support for retries, out of the box, but not with delay. When consuming a message the client declares whether to use auto acknowledgement or not. Auto acknowledgement means the broker can discard messages once delivered to the consumer (but before processing). Manual acknowledgement means the broker should wait for the client (consumer) to explicitly acknowledge when the message processing completed (successfully or not) and can be discarded. Note that there are no timeouts. The message will not be redelivered to a consumer, unless a negative acknowledgement is received or the connection is closed. When using automatic acknowledgements, retries due to application errors are practically not applicable. When using manual acknowledgements, a message can be redelivered by RabbitMQ if it is rejected by the consumer (meaning a negative acknowledgement is sent) and the flag requeue is set to true.

Observability

Açıklaması şöyle. Eğer tüm kuyrukları dinlemek istersek binding key olarak "#" kullanılır

Events history

In AMQP once a message is delivered (and optionally acknowledged), it is removed from a queue, and you will never see it again. What if you need to investigate what happened at that stage?

I can think of several options, for example:

- Enabling debug level of logging in message bus.

- RabbitMQ Firehose.

- Subscribing to all messages and storing/redirecting them to separate storage or message bus (might be done via custom subscriber or via tools like Apache Flink, Kafka RabbitMQ connector, etc.).

We have chosen the last option, which works the following way:

- There is a queue, which is subscribed to all messages in topic exchange via “#” binding key (which means matching any routing key);

- There are consumers, which constantly read messages from that queue and store it in the appropriate format to persistent storage.

Durable Queue

Açıklaması şöyle. Durable kuyruklar broker yeniden başlatılsa bile tekrar yaratılırlar. Eğer kuyruktaki mesajlar da persistent ise onlar da tekrar eklenirler.

Durable queues are persisted to disk and thus survive broker restarts. Queues that are not durable are called transient. Not all scenarios and use cases mandate queues to be durable.

Durability of a queue does not make messages that are routed to that queue durable. If broker is taken down and then brought back up, durable queue will be re-declared during broker startup, however, only persistent messages will be recovered.

Örnek

Şöyle yaparız. Queue durable, aynı zamanda auto_delete özelliği false, Böylece en son consumer kapansa dahi kuyruk silinmez.

curl --user guest:guest \
 -X PUT -H 'content-type: application/json' \
 --data-binary '{"vhost":"/","name":"test-queue-01","durable":"true","auto_delete":"false",
  "arguments":{"x-queue-type":"classic"}}' \
'http://localhost:15672/api/queues/%2F/test-queue-01'

AMQP Frame Tipleri

AMQP Frame Tipler yazısına taşıdım.

High Availability

Açıklaması şöyle

RabbitMQ has provided clustering and highly available queues for several major versions. Version 3.8.0 shipped with “Quorum Queues” which use the Raft consensus algorithm to provide data replication with higher performance than “classic” HA queues.

Subscription Çeşitleri Nedir

İki çeşit subscription var.

1. Durable Subscription

Consumer çalışmasa bile elle silininceye kadar abonelik yaşar

2. Ephemeral subscription

Consumer kapatılınca subscription ve beklemekte olan mesajları silinir.

Exchange Çeşitleri

Exchange Çeşitleri yazısına taşıdım

Lazy Queue

Açıklaması şöyle

People that are trying out RabbitMQ are probably not aware of the the feature lazy queues. Lazy queues are queues where the messages are automatically stored to disk, thereby minimizing the RAM usage, but extending the throughput time. In our experience, lazy queues create a more stable cluster with better predictive performance. If you are sending a lot of messages at once (e.g. processing batch jobs), or if you think that your consumers will not consistently keep up with the speed of the publishers, we recommend that you enable lazy queues.

RabbitMQ Single Active Consumer (SAC)

Açıklaması şöyle. { "x-single-active-consumer", true } parametresi ile sağlıyoruz.

The Single Active Consumer mode allows to have only one consumer at a time consuming from a queue and to fail over to another registered consumer in case the active one is cancelled or dies.