Time to design a product!! Monolithic vs Microservices? Monolithic vs microservices architecture battle has been going for a while; the scalability, robustness and efficiency of the product is determined by the architecture leveraged.

Monolithic architecture basically represents a single unified unit and is usually one code base whereas microservices are small, autonomous, and self-contained services, owning their data.

What are Microservices?

Microservices are small, autonomous, and self-contained services, owning their data. Question here is how microservices communicate if they are not wired?

Microservice based applications are basically a distributed system comprising of multiple processes or services and, each service representing a particular feature or module of the application.

Since each microservice functions as individual entity, there should be a communication protocol for the services to interact. Depending on the nature of the services, protocols like HTTP, AMQP, or a binary protocol like TCP can be leveraged for inter communication.

Event-based communication between Microservices

1. Depending on the type of protocol there are two ways for microservices to communicate:
2. Synchronous Communication: A request-response type of communication where client sends a request and waits for the response from the service. This mainly uses synchronous protocols like HTTP/HTTPs.
3. Asynchronous Communication: In this process, the client or the message sender doesn’t wait for a response and it just sends message to a broker.

• Depending on the number of receivers, the communication can be divided as:
• Single: Each request will be processed by a single receiver or service.
• Multiple: Event driven microservice architecture where data propagation between multiple services take place through events.

One of the very common design patterns to wire microservices is Event-driven architecture, wherein events are triggered, detected, consumed, and are reacted to. Event driven architecture is very common in real time applications, that respond to events as they occur in real time.

Let’s consider a scenario where an application is comprised of multiple microservices, and the microservices need to communicate asynchronously. Most importantly, the microservices still need to be maintained individually. As an example, when quantity of items in a warehouse is changed, the corresponding microservice (let’s say “Inventory microservice”) updates the data. Additionally, these changes are to be subscribed by other microservices in real-time so that corresponding opportunities/leads can be created accordingly. So, in this example, Inventory microservice needs to generate event for change in inventory, which will be consumed by other microservice (let’s say “Opportunity microservice”) to create the required opportunity/lead in real-time. This is where Event-driven microservices architecture come into play. A microservice in an event-driven architecture publishes an event when some action is performed.

The main components of event-driven architecture are event producer, event consumer, and broker. Upon trigger of events, the producer sends stream of events to the broker service, which then connects it to the consumer.

Event-driven Architecture

A Producer/Publisher posts an event to a broker. Event indicates a piece of data to be transmitted. It can also be a JSON object indicating the data transacted in the Producer service. When another service (consumer) needs event/data sent to a broker, they subscribe to the required topic in the broker and consume the data.

Now, to handle the processing of stream of events, additional technologies need to be plugged in. One of the most popular tools to deal with streaming data and events is Apache Kafka. Apache Kafka is an open-sourced distributed event streaming platform.

Distributed nature of Kafka promotes to build real-time streaming applications, publish, and consume events in real-time on a large scale. Events that are published to Kafka are not deleted upon consumption and are deleted after a certain retention time. However, the same event/message can be consumed by multiple consumers as part of different microservices.

Kafka Component Overview

The main Kafka components are Event, Topics, Producers, Consumers, Consumer Group, Partitions, Clusters, Brokers

A Kafka event is mainly composed of a key and a value. Since events are written in stream, they are termed as “Event Streaming”. Each new event is added at the end of the stream. For each change in state from the producer service, there will be event created.

For each operation/action, events will be triggered. To differentiate the events and allow consumers to consume the ones they require, the events are divided into Topics.

To allow consumption of a topic by several instances of a consumer, topics are split into Partitions, which helps to handle events at a large scale.

Kafka Producer publishes messages/events to one or mor Kafka topics whereas Kafka Consumer subscribes to one or more topics, and they read messages from the topics they subscribe to.

Consumer Group is a multi-threaded or multi-machine consumption from Kafka topics. To increase the processing capacities, several instances of the same consumer can be added, which will then be registered in the same consumer group. Each consumer in a Consumer Group consumes a subset of the topic’s partitions, equally distributed among the consumers.

Kafka Cluster is a system that consists of several brokers, topics, and partitions for both.

The Kafka Server is known as Broker, which takes care of the Topic’s Message Storage. Each of the Kafka clusters comprises of more than one Broker to maintain load balance.

Below diagram depicts the overview of Kafka components:

Conclusion

Event-driven architecture aims at building robust decoupled real-time system based on trigger of real-time events, which can be consumed by various downstream systems.

Kafka allows events to be exposed in a structured way and to be easily subscribed to maintain scalability, without affecting the producing and consuming applications and thus promotes event-driven architecture in an optimized fashion.