Introduction to Cloud-native Streaming & Messaging Services

Cloud-native is revolutionizing the way applications and services are delivered, and the Cloud Native Computing Foundation (CNCF) is doing a great job of driving standardization and vendor-neutral solutions within the industry. In a recent post, we explored the CNCF’s cloud-native tool landscape from top to bottom. The tools and technology we discussed there will shape the face of cloud-native over the next few years.

Cloud-native streaming & messaging services are one of the most important aspects of the landscape. These services act as the middleware of cloud-native. Acting as “the middleware of cloud-native”, they enable microservices to communicate while remaining loosely coupled and lightweight.

In this piece, we’ll define cloud-native messaging and streaming, explore their use cases, and discuss some of the popular streaming & messaging applications available today.

#What is cloud-native messaging?

Before we build up to cloud-native streaming, let’s start by explaining cloud-native messaging. Cloud-native messaging is a communications model that enables asynchronous push-based communications between microservices. For example, an automated message for business can be triggered by specific events, facilitating smoother workflows and faster responses.

So, how does it work?

As you might expect, once you dive into the nuts and bolts, messaging can become quite complex, but let’s start with the basics.

At a high-level, sending information using a messaging service consists of 3 key components:

• A producer- Creates messages and sends them to the broker.
• A broker- Receives messages from consumers and distributes them to consumers. The broker takes care of queueing, routing messages, and translating between protocols.
• A consumer- Receives messages from a broker.

That seems straightforward, but how does the broker determine what messages should be routed to specific consumers? Implementations vary, and this is where much of the complexity is layered in. The publish/subscribe approach to message delivery is used widely in cloud-native to conceptualize the process. First, messages are sorted by “topics”. Producers (publishers) then send messages to the specific topic, and the broker routes them on to the consumers subscribed to that topic (subscribers).

#Benefits of cloud-native messaging

A common question those new to cloud-native messaging ask is “with the ubiquity of HTTP and RESTful APIs, why should I use a messaging service?” The short answer is that messaging enables lightweight, scalable, event-driven communication between microservices. To understand why, let’s take a look at some of the specific benefits.

• Enabling 1 to many & many to 1 communication- With asynchronous communication, a single message or event can be broadcasted to many consumers. Similarly, messages from many producers can be sent to a single consumer, say, a database. Messaging service provides a simpler alternative to pull-based synchronous architectures like RESTful API. It also reduces workloads on message producers.
• Security- With a broker between consumers and producers, there is no need to expose an API to enable communications between services. Producers and consumers can be agnostic of one another.
• Loose coupling and scalability- Loose coupling is the core principle of sound microservice architecture. With a RESTful API, the client application must “know” how the server-side API works and what endpoints to query. With messaging, services don’t need any information about one another; they just need to connect to the broker.

Of course, this isn’t to say we should do away with RESTful APIs. The takeaway here is to understand the right tool for the job. For more on the deciding when and where to use a RESTful API vs a messaging service, check out James Higginbotham’s Choosing Between Web APIs and Message Streaming article.

#Common cloud-native messaging protocols

Unlike the API space, where HTTP(S) wins the day, there isn’t a single “messaging protocol” for cloud-native apps. Knowing which protocol makes sense for your applications requires understanding the different options, so let’s take a look at the three most common messaging protocols and explore their use cases.

#AMQP

AMQP (Advanced Message Queuing Protocol) is the most popular messaging protocol for cloud-native today and has almost become the de-facto standard. AMQP is highly scalable as demonstrated by several real-world use cases, including NASA’s Nebula Cloud Computing services and JP Morgan Chase where AMQP is reported as sending 1 billion messages a day. In addition to being scalable, AMQP enables granular controls over message routing and has a robust feature set. If you need support for complex routing & messaging or hyper-scalability, AMQP is likely the right choice.

#MQTT

MQTT (Message Queue Telemetry Transport) is a lightweight messaging protocol designed to reduce bandwidth consumption and resource utilization. As a result, MQTT is a popular choice in the world of IoT and M2M (machine-to-machine) communications. If you’re designing applications where resource consumption and simplicity are a priority, MQTT is an excellent choice, and MQTT topics along with wildcards make configuring device communication a breeze.

#STOMP

STOMP (Simple/Streaming Text Oriented Messaging Protocol) is a bit of a shift from AMQP and MQTT. STOMP was designed in the spirit of HTTP: a text-based protocol (both AMQP and MQTT are binary) that is easy to implement on the client-side and supports interoperability across languages and platforms. Apart from just being a simple, text-based alternative to other messaging protocols, STOMP has many novel use cases as well.

#What is cloud-native streaming?

Cloud-native streaming is the processing of continuous data streams and is commonly used for large amounts of data. As a rough analogy, streaming is to messaging what NoSQL is to relational databases.

Cloud-native streaming shares many similarities with cloud-native messaging, and many of the operational aspects are the same. Streaming is asynchronous, has consumers and producers, and uses a publish/subscribe model with “topics”.

So, what’s the difference between cloud-native streaming and messaging?

• A single “log” is used-With streaming as opposed to discrete messages with discrete message IDs, data within a given topic is appended to a log. Events are simply identified by their position within the log.
• Consumers are “smarter” than brokers- A streaming service offloads much of the work to consumers as opposed to having brokers handle the bulk of the business logic. Consumers are responsible for tracking what streams they have consumed and topics they are subscribed to.
• Historical data is retained- Streaming enables consumers to read historical data. With traditional messaging, there is no rewinding through old messages. With streaming services, data is retained for a set amount of time, and consumers can pull historical information.

#Benefits of cloud-native streaming

With that definition, some of the benefits of streaming may become clear. Benefits of streaming compared to messaging include:

• Speed & performance- Streaming services like Apache Kafka can process millions of messages per second with fewer compute resources than messaging services.
• Data persistence- With streaming, data is retained so that consumers can “replay” previous events.
• Processing of “infinite” inputs- Streaming is designed to work with the continuous data flow. It is suitable for high-volume data flows like GPS or temperature sensor data.

Given these benefits, it is easy to understand why streaming has seen significant adoption across in the fields of Big Data Analytics and High-Performance Computing.

#Popular cloud-native streaming & messaging services

With an understanding of what cloud-native streaming and messaging are, let’s take a look at some of the software being used to implement them in the real world.

#NATS

NATS is a CNCF-incubated open-source message broker that is surging in popularity. As Kevin Hoffman described in his Capital One Tech Medium post, NATS fills a unique niche in the cloud-native messaging world. It provides developers with a lightweight, simple, and text-based approach to messaging. It uses a traditional publish/subscribe approach to messaging, providing a mechanism for request/reply.

In addition to enabling simple deployment models, NATS boasts higher brokered throughput than many of the big names in messaging & streaming, including Kafka and Redis.

#RabittMQ

RabbitMQ is an open-source message broker that supports AMQP, MQTT, STOMP, and HTTP & WebSockets (while HTTP isn’t really a messaging protocol, there are use cases for browser integrations and low volume messaging). RabbitMQ is one of the most widely deployed and trusted brokers in the industry being adopted at small and large enterprises alike.

In addition to being a mature, scalable, and reliable platform, RabittMQ has many features that make it attractive for cloud-native use cases. It is possible to deploy RabittMQ with orchestration tools like Chef and Puppet. Additionally, it supports a variety of languages including Java, Python, Go, and .NET, and clusters and federations enable distributed, high-availability (HA) deployments across multiple regions.

Check How To Install and Start Using RabbitMQ on Ubuntu.

#Apache Kafka

Apache Kafka has surged in popularity in recent years. While it started as a messaging service (and still supports “traditional” messaging today), it has evolved into a robust cloud-native streaming service capable of processing millions of events per second. Common use cases for Kafka in the realm of cloud-native include log aggregation (think Hadoop and other highly distributed file systems) and stream processing.

#Conclusion

From containerization and microservices to distributed databases, cloud-native is changing the way software engineering gets done. To remain competitive, application developers must understand how to design robust, resilient, and distributed services. By enabling microservices to communicate in a loosely coupled and lightweight fashion, streaming and messaging are an important part of doing just that.