Message Queues receive, hold and pass on messages to other parts of the system. They are useful when your system is slow or in a particular workflow it is doing a heavy computation. It is very useful for jobs that need to be done asynchronously and we do not want users to wait for a long time for any feedback. The usual flow looks as follows:
- A user performs an action that takes a long time to process, so it is pushed to a queue by a server. Server then notifies a user about the status
- Another service processes it and sends a signal that the job is complete
This is very useful when you want to ensure that your system is eventually consistent and it is ok for your system to take a few seconds to show data to other users later.
Transclude of Message-Queue-Example.excalidraw
Use cases
Microservices
Message Queues are useful for building Microservice architecture where you can decouple your system into different services and they can subscribe to events.
Example
Service Orders can create an event “order created” that is listened by Inventory Service and Notification Service, and they can react independently without having to call each other directly
Async processes
They are also useful for asynchronous and long processes such as creating a file or reports, file processing (YouTube or Soundcloud).
Example
Let’s take Soundcloud as an example. A Web Server accepts the request and responds quickly telling the user that the system is processing the requests. It gives a user feedback straight away about what is happening. In a meantime, a service can push it to a queue where it is being processed.
Tools
- Amazon SQS - Managed and hosted, but may have some issues with duplication
- Redis - Useful for simple message broker but if data is not persisted, it can be lost
- Rabbit MQ - uses AMQP protocol so you need to adapt.