Developing Message-Based Solutions with Azure Service Bus and Azure Queue Storage in .NET

12 Sep Developing Message-Based Solutions with Azure Service Bus and Azure Queue Storage in .NET

Developing Message-Based Solutions with Azure Service Bus and Azure Queue Storage in .NET

In today’s fast-paced world, efficient communication and data exchange between different parts of your application is crucial. Azure, Microsoft’s cloud platform, offers two powerful messaging services – Azure Service Bus and Azure Queue Storage – to help you build robust, scalable, and reliable message-based solutions. In this blog post, we’ll explore how to integrate these services into your application using .NET, along with use cases, best practices, and a conclusion to guide you on your messaging journey.

Comparing Cloud Messaging Options

Each of the messaging services that are used in the cloud has distinct use cases.

Storage Queues and Event Grid are serverless services ideal for simple and scaled-out workloads that don’t have severe ordering or consistency requirements. The Service Bus service offers a more expensive and complex alternative if a workload has strict consistency and order requirements.

The Event Hubs service is unique because it’s designed for streaming data that must remain consistent across multiple consumers.

Figure 1, Comparing cloud messaging options (Microsoft documentation)

Azure Service Bus

Azure Service Bus is a message broker that facilitates enterprise integration in a fully managed way. It is commonly used to create a decoupled architecture between applications and services. Service Bus provides a secure and reliable asynchronous data and state transfer platform. Messages are used to transfer data between different applications and services. These messages can be in binary format and contain JSON, XML, or plain text.

A namespace is a container that holds all messaging components. Multiple queues and topics can reside within a single namespace, making namespaces ideal application containers.

Some common messaging scenarios are:

• Messaging: transfer business data, such as sales or purchase orders, journals, or inventory movements.
• Decouple applications: improve reliability and scalability of applications and services (client and service do not have to be online simultaneously).
• Topics and subscriptions: enable 1:n relationships between publishers and subscribers.
• Message sessions: implement workflows that require message ordering or message deferral.

 

Events vs. messaging services

 

Event

Events are notifications of a change in condition or state. The publisher of the event doesn’t expect any particular handling of the event, while the consumer decides what to do with the notification. Events can be either discrete or part of a series.

Discrete events are actionable and report a state change. Upon receiving notification, the consumer only needs to know that something happened and can take the next step. The information provided by the event data is about what happened rather than what triggered the event. For example, an event notifies consumers that a file was created, providing general information about the file but not the file itself. Discrete events are ideal for serverless solutions that need to scale.

Series events report a condition and are analyzable. These events are time-ordered and interrelated. The consumer needs a sequenced series of events to analyze what happened.

Message

A message is a piece of data a service generates intended to be used or stored elsewhere. It contains the information that triggered the message pipeline. The publisher of the message expects the consumer to handle the message in a specific way, and there is a contract between the two parties.

For example, the publisher sends a message with raw data and expects the consumer to use that data to create a file. Once the file is created, the consumer is expected to respond to the publisher, indicating that the task is complete. This way, both parties know what to expect and how to proceed.Figure 2, Events vs. messaging services (Microsoft documentation)

Discover Service Bus queues, topics, and subscriptions

The messaging entities that form the core of the messaging capabilities in Service Bus are queues, topics and subscriptions, and rules/actions. 

Queues

Messaging systems use queues to deliver messages in a first-in, first-out (FIFO) manner to one or more competing consumers. This ensures that messages are processed in the order they were added to the queue, and only one consumer consumes each message. Queues offer a significant advantage by temporally decoupling application components. This means that producers and consumers can work independently since the messages are stored persistently in the queue, eliminating the need for producers to wait for immediate responses from consumers. Similarly, consumers can process messages at their own pace. This architecture allows for asynchronous communication, reducing dependencies and enhancing flexibility and scalability in distributed systems.

In message-based systems, the “competing consumer pattern” provides significant benefits such as load leveling. This pattern enables message producers and consumers to operate at different rates, which is particularly advantageous in applications with varying system loads. A queue intermediates between message producers and consumers, ensuring that the consumer application only needs to handle the average load rather than the peak load. The time required to process each unit of work remains constant. The depth of the queue naturally grows and contracts with fluctuating incoming workloads, resulting in cost savings by optimizing the required infrastructure for the application load.

As the system load increases, additional worker processes can be added to read from the queue. This helps improve the utilization of resources. Each message is processed by only one of the worker processes, which prevents duplication. This pull-based load-balancing approach enables worker computers of varying processing power to retrieve messages at their maximum rate, further optimizing resource utilization. The “competing consumer pattern” enhances the scalability and cost-efficiency of the system by adapting to changing workloads and ensuring efficient use of available resources.Figure 3, Message queue with messages (Microsoft documentation)

Using queues to intermediate between message producers and consumers provides an inherent loose coupling between the components. Because producers and consumers are not aware of each other, a consumer can be upgraded without having any effect on the producer.

Topics and subscriptions

In contrast to queues, where each message is processed by a single consumer, topics, and subscriptions offer a one-to-many method of communication in a publish/subscribe pattern. This pattern is helpful for scaling to a large number of recipients. Every published message is made available to each subscription registered with the topic. Messages are sent to a topic and delivered to one or more associated subscriptions, depending on the filter rules that can be set per-subscription. The subscriptions can use additional filters to restrict the messages that they want to receive. Messages are sent to a topic in the same way they are sent to a queue, but they are not received from the topic directly. Instead, they are received from subscriptions. A topic subscription is similar to a virtual queue that receives copies of the messages sent to the topic. Messages are received from a subscription in the same way as they are received from a queue.Figure 4, Topic with three subscriptions with messages (Microsoft documentation)

In comparison, the message-sending feature of a queue is similar to a topic, whereas the message-receiving feature is similar to a subscription. This implies that the same patterns that are applicable to queues, such as competing consumers, temporal decoupling, load leveling, and load balancing, are also supported by subscriptions.

Messages consist of a payload and metadata. The metadata is present as properties containing key-value pairs, which describe the payload and provide handling instructions to the Service Bus and applications. Sometimes, the metadata alone can convey the information the sender wants to communicate to receivers, and the payload can be empty. The object model of the official Service Bus clients for .NET and Java reflect the abstract Service Bus message structure, which is mapped to and from the wire protocols Service Bus supports.

A message in Service Bus has three main components: a binary payload section that Service Bus never modifies and two sets of properties – broker properties and user properties. The system predefines broker properties and can control message-level functionality or map to standardized metadata. User properties are key-value pairs that the application can define.

Payload serialization –  A payload is always a binary block when transported or stored in a Service Bus. The ContentType property allows the application to describe the payload, and the suggested format for the property values is a MIME content-type description according to IETF RFC2045. For example, application/json;charset=utf-8 can be used to describe a JSON payload. 

Unlike the Java or .NET Standard variants, the .NET Framework version of the Service Bus API allows the creation of BrokeredMessage instances by passing arbitrary .NET objects into the constructor.

Rules and actions

Many times, messages with specific properties need to be handled differently. You can configure subscriptions to find messages with desired properties and apply specific modifications to achieve this. Although Service Bus subscriptions can access all messages sent to the topic, only a subset can be copied to the virtual subscription queue. This is done by using subscription filters. These modifications are known as filter actions. When you create a subscription, you can provide a filter expression that operates on the message properties, which can be either the system properties (such as Label) or custom application properties (such as StoreName). The SQL filter expression is optional in this case. If there is no SQL filter expression, any filter action defined on a subscription will be applied to all the messages for that subscription.

Integration with .NET:

To integrate Azure Service Bus with a .NET application, you’ll need to use the Azure SDK. Here are the high-level steps:

1. Install the Azure SDK: Use NuGet to add the `Microsoft.Azure.ServiceBus` package to your .NET project.
2. Create a Service Bus Namespace: In the Azure portal, create a Service Bus namespace, and note the connection string.
3. Send and Receive Messages: In your .NET code, use the Service Bus client to send and receive messages. Here’s a simplified example:

Figure 5, Code snippet to integrate Azure Service Bus with a .NET application (Microsoft documentation)

Azure Queue Storage 

Azure Queue storage is a reliable and scalable cloud-based service that enables the storage of many messages that can be accessed securely from anywhere in the world using authenticated HTTP or HTTPS calls. Each queue message can be up to 64 kilobytes (KB) in size, and a queue can hold millions of messages up to the total capacity limit of a storage account.

The Queue service contains the following components:

• URL format:Queues are addressable by using the following URL format:
  https://<storage account>.queue.core.windows.net/<queue>
  The following URL addresses a queue in the diagram:
  https://myaccount.queue.core.windows.net/images-to-download
• Storage account:All access to Azure Storage is done through a storage account.
• Queue:A queue contains a set of messages. All messages must be in a queue. Note that the queue name must be all lowercase.
• Message:A message, in any format, of up to 64 KB. The maximum time that a message can remain in the queue is seven days.

Figure 6, Queue service componenets (Microsoft documentation)

Common use cases include:

1. Decoupling Components: Use queues to decouple parts of your application, ensuring that they can work independently.
2. Background Processing: Queue storage is perfect for processing tasks asynchronously, such as image or video processing.

Integration with .NET

Integrating Azure Queue Storage with a .NET application is straightforward. Follow these steps:

1. Install the Azure SDK: Add the `Microsoft.Azure.Storage.Queue` package to your .NET project.
2. Access Your Storage Account: Retrieve the connection string for your Azure Storage account.
3. Send and Receive Messages: In your .NET code, use the Azure Queue Storage client to send and receive messages. Here’s a simplified example:

Figure 7, Code snippet to integrate Azure Queue Storage with a .NET application (Microsoft documentation)

Best Practices

To ensure success in your message-based solution, consider these best practices:

General Best Practices

• Always handle exceptions and implement retries to ensure message processing resilience.
• Use appropriate authentication and authorization mechanisms to secure your services.
• Monitor message queues to identify performance bottlenecks and potential issues.

Azure Service Bus Best Practices

• Leverage dead-letter queues for handling undeliverable or poison messages.
• Implement batching for efficient message processing.
• Consider enabling AutoForward to route messages from one queue or topic to another.

Azure Queue Storage Best Practices

• Opt for shorter lease times for messages to allow faster reprocessing in case of failures.
• Consider implementing a poison queue for handling problematic messages.

 

Conclusion

Utilizing Azure Service Bus and Azure Queue Storage can be highly beneficial when building message-based solutions in the cloud. These services can be integrated into your .NET applications, resulting in efficient communication, reliability, and scalability. Azure messaging services can provide the necessary support for developing microservices, event-driven architectures, or IoT solutions. Following industry best practices and safeguarding your messaging solution with proper security measures and monitoring is essential to ensure its success.

References:

– Azure Service Bus Documentation (https://docs.microsoft.com/en-us/azure/service-bus-messaging/)
– Azure Queue Storage Documentation (https://docs.microsoft.com/en-us/azure/storage/queues/storage-queues-introduction)