Azure Service Bus tools

Overview

Azure Service Bus is a multi-tenant cloud messaging service that sends information between applications and services. Data Management has three tools for interacting with Azure Service Bus:

• Azure Service Bus Send tool

• Azure Service Bus Receive tool

• Azure Service Bus Acknowledge tool

Batch transactions

The Azure Service Bus Send and Receive tools support batch transactions. Instead of sending each message as an individual request to Azure Service Bus, messages are collected and sent in batches. This can dramatically improve tool throughput. Optimal batch size depends on the message payload size, the message volume, and other factors.

The Azure Service Bus Acknowledge tool does not support batched transactions.

Shared Service Bus Client for Receive and Acknowledge tools

Messages can only be acknowledged by the Service Bus client who received them.

For the Azure Service Bus tools, when a Receive tool is configured in PeekLock mode, account for the following:

• An Acknowledge tool can only acknowledge messages received by an upstream Receive tool if both share the same Service Bus Client. In Data Management, a Service Bus Client is defined by the Namespace, Service name, SAS key, and SAS key name properties.

• Typically, configuring the Azure Service Bus tools via Settings > Tools ensures that Receive and Acknowledge tools share the same Service Bus Client. However, if these shared settings are overridden at the tool level, changing any of those properties results in the creation of a distinct Service Bus Client. If an Acknowledge tool does not share the same Service Bus Client with its upstream Receive tool, message acknowledgment will fail.

Azure Service Bus tool shared settings

Data Management's tools use shared settings, which allows you to define a single set of configuration properties (typically access credentials) to share across multiple tools in your Data Management Site. You can override these settings on a per-tool basis by opening the Shared settings section on the tool's Properties pane, selecting Override shared settings, and specifying values for that specific tool.

To define Azure Service Bus shared tool settings

1. Go to Settings in the repository.

2. Open the Tools folder.

3. Select the Azure Service Bus tab, and then configure the tool properties for your environment.

To configure default shared tool settings from an Azure Service Bus tool's Properties pane, open the Shared settings section, and select Edit default settings.

To override Azure Service Bus tool shared settings

1. Select the desired Azure Service Bus tool.

2. Go to the Configuration tab on the Properties pane.

3. Open the Shared settings section, select Override shared settings, and specify new values for the tool.

Azure Service Bus tool message properties

The Azure Service Bus Send and Azure Service Bus Receive tools support setting message properties when sending and receiving messages. Properties may be defined inline, or via a separate data stream that you connect to the tool.

When sending large volumes of messages with properties, the "inline" strategy is preferred, as it uses memory more efficiently.

Define properties inline

Inline properties are provided as a JSON document embedded in each record delivered via the Azure Service Bus tool's "M" Messages connector. Each record must include a field of TextVar type that contains this JSON document. Properties are specified as name/value pairs like this:

{
  "name" : "Han Solo",
  "occupation" : "nerf herder",
  "balance" : 0
}

When you configure your Receive tool, set Message properties to Inline, and specify the Inline properties field.

Define properties via a connector

Message properties can also be provided via a separate data stream using the Message properties connector.

To connect an Azure Service Bus tool to a message properties data stream:

1. Select the desired Azure Service Bus tool.

2. Go to the Configuration tab on the Properties pane.

3. Select Message properties and choose Connector.

4. Select Commit to display the M1 input or output connector, and then connect the desired input or output tool.

Records provided to the connector must match the following schema.

Azure Service Bus Receive tool will output properties to the M1 output connector, with each record containing a MessageId and field, allowing you to associate properties with their messages.

Azure Service Bus Send tool

The Azure Service Bus Send tool allows you to send messages to an Azure Service Bus queue or topic.

• The M "Message" input connector accepts a stream of records that includes one field to be passed as the message body.

• The S "Sent" output connector emits a stream of records representing successfully sent messages.

• The F "Failed" output connector emits a stream of records representing messages that could not be sent, and error messages describing the failure.

Azure Service Bus Send tool configuration parameters

The Azure Service Bus Send tool has a single set of configuration parameters in addition to the standard execution options.

Configure the Azure Service Bus Send tool

Before configuring an Azure Service Bus Send tool, you should have an Azure Service Bus connection defined in shared settings.

1. Select the Azure Service Bus Send tool.

2. Go to the Configuration tab on the Properties pane.

3. Select Service bus type and choose the communication mechanism for this Azure Service Bus namespace: Queue or Topic.

   • If Service bus type is Topic, select Subscription name and select the desired subscription.

4. Select Message ID field and select the field in the record provided via the M input connector whose contents will be sent as the message ID.

5. Select Message body field and select the field containing the Unicode string to be sent to the queue as the message's body.

6. Select Message properties and specify how message properties are provided:

   • Select None if there are no defined message properties.

   • Select Inline if message properties are provided as a JSON document embedded in each input record.

   • Select Connector if message properties are provided in a separate data stream.

7. If you selected Inline in step 6, select Inline properties field and select the field containing the JSON document describing the record's properties.

8. If you selected Connector in step 6, select Commit to display the M connector, and then connect the desired input tool.

9. Optionally, select Maximum batch size and specify the maximum number of messages to be sent with each batch.

10. Optionally, override shared settings.

11. Optionally, go to the Execution tab, and then set Web service options.

Azure Service Bus Receive tool

The Azure Service Bus Receive tool allows you to receive messages from an Azure Service Bus queue.

• The optional T "Trigger" input connector causes the Azure Service Bus Receive tool tool to call "receive" on the queue or subscription for each record sent to the input connector.

• The M "Message" output connector emits a stream of records representing received messages.

• The F "Failure" output connector emits a stream of records representing receipt errors.

Azure Service Bus Receive tool configuration parameters

The Azure Service Bus Receive tool has a single set of configuration parameters in addition to the standard execution options.

Parameter	Description
Service bus type	Communication mechanism for this Azure Service Bus namespace: Queue: allows one-directional communication. Each queue acts as an intermediary between message producers and consumers, storing sent messages until they are received. Each message is received by a single recipient. Topic: provides one-directional communication using subscriptions. In contrast to queues, in which each message is processed by a single consumer, topics and subscriptions provide a one-to-many form of communication, in a publish/subscribe pattern. A single topic can have multiple subscriptions. Like a queue, a topic acts as an intermediary, but each subscription can optionally use a filter to receive only messages that match specific criteria.
Subscription name	If service bus type is Topic, name of the subscription.
Include metadata	NEW IN 9.6.1 Attaches the following metadata fields to each received message, when enabled: CorrelationId DeadLetterSource DeliveryCount Label LockToken LockedUntilUtc MessageId PartitionKey ReplyTo ReplyToSessionId ScheduledEnqueueTimeUtc SessionId To ViaPartitionKey This option is disabled by default.
Receive mode	Specifies how messages are handled upon receipt: ReceiveAndDelete: destructive reads in which the message is delivered to a consumer and removed from the queue or subscription in a single atomic operation. PeekLock: non-destructive reads. When a message is delivered to a consumer, it is locked. The message is guaranteed not to be delivered to any other consumers during that lock period. During that lock period, the consumer will attempt to process the message. If the consumer completes the message, it will be removed from the queue/subscription. If the consumer fails to process the message and explicitly abandons it, or if the lock period expires, the message will be returned to the queue or subscription where it becomes available to another consumer. Receive mode must be set to PeekLock if the data flow will contain a downstream Acknowledge tool. Also, the Receive and Acknowledge tools must share the same Service Bus Client; refer to Shared Service Bus Client for Receive and Acknowledge tools for details. There are important performance/reliability tradeoff considerations for these options detailed in the Receive mode and performance/reliability tradeoff section below.
Triggered receipt	When selected, the Receive tool will sprout a T "Trigger" input connector. For each record sent to the input connector, the Receive tool will call "receive" on the queue or subscription. When cleared, the Receive tool will continuously call receive until one of the following occurs: Time limit is exceeded. Message limit is exceeded. Receive timeout expires before the queue/subscription returns any messages.
Message properties Inline properties field	See Azure Service Bus tool message properties.
Maximum batch size	The maximum number of messages to be received in each batch. Fewer messages may be returned if the batch size exceeds the number of available messages, or the Receive timeout expires before the batch is filled.
Receive timeout	Duration (in seconds) that the Azure Service Bus Receive tool will wait for messages to be returned from the queue or topic. The interplay between the Receive timeout and the queue or subscription's Lock duration: a long receipt timeout combined with a short lock duration can lead to acknowledgment failures in peek/lock mode.
Limit messages Message limit	Select Limit messages to specify Message limit as the maximum number of messages to read before exiting.
Limit time Time limit	Select Limit time to specify Time limit as the maximum amount of time (in seconds) that may elapse before exiting. This is different from the Receive timeout parameter in that this time limit will be enforced even if more messages are available on the queue. If both Limit messages and Limit time are selected, the tool will exit when the first of these thresholds is exceeded, even if more messages are available on the queue. See also "The effects of batching on limits" below.
Override shared settings	See Azure Service Bus tool shared settings.

The effects of batching on limits

Time and message limits specified with the Limit messages and Limit time options should be considered an approximation. The Receive tool cannot interrupt a call to receive, so it is possible for the number of received messages to exceed the defined message limit, especially when processing larger batches.

Receive mode and performance/reliability tradeoff

The Azure Service Bus Receive tool supports PeekLock and ReceiveAndDelete modes. This section provides additional information about the performance/reliability tradeoff between these modes.

In ReceiveAndDelete mode, a message is considered consumed as soon as the broker sends it to the client. Because the broker doesn’t await settlement, this mode is typically more performant but far less resilient - messages can be dropped. This can be described as “best-effort once” delivery.

In PeekLock mode, each message is locked before it’s sent to a client, and must be settled explicitly by the receiving client. During the message’s lock period, no other client can operate on the message. If the client completes the message, it is removed from the queue. If the client abandons the message or its lock expires, the message becomes eligible for redelivery to this or other clients (or is dead-lettered if its delivery count exceeds the queue's configured maximum). Azure guarantees at-least-once delivery, but that reliability incurs a performance penalty.

In summary, set Receive mode to:

• PeekLock if reliability is more important

• ReceiveAndDelete if performance is more important

Refer to Message transfers, locks, and settlement - Azure Service Bus for additional information.

Under some circumstances, the rate at which messages are received far outstrips the rate at which they can be acknowledged. This is particularly common when intermediary processing between the Receive and Acknowledge tools is time-consuming. In such cases, one or both of the following symptoms can manifest:

• Backpressure from saturation of the internal message backlog results in abandonment of messages

• Message acknowledgment fails due to lock expiry (e.g., the Acknowledge tool emits messages to its Failure connector indicating the expiration of the message’s lock)

In both cases, the message is returned to the queue for redelivery. If you observe excessive message redelivery (e.g., an increasing DeliveryCount on Message records emitted by the Receive tool’s Messages connector) and eventual dead-lettering, consider experimenting with…

• The Receive tool’s Maximum batch size and Receive timeout properties to attenuate the rate at which messages are received from the queue

• Using the Split and Merge tools to parallelize acknowledgment by “ganging” multiple Acknowledge tools

• Adjusting the project’s memory via Settings > JVM Settings > JVM memory to accommodate a deeper message backlog

This is a trial-and-error process, as optimal settings are highly-dependent on your specific solution design.

Configure the Azure Service Bus Receive tool

Before configuring an Azure Service Bus Receive tool, you should have an Azure Service Bus connection defined in shared settings.

1. Select the Azure Service Bus Receive tool.

2. Go to the Configuration tab on the Properties pane.

3. Select Service bus type and select the communication mechanism for this Azure Service Bus namespace: Queue or Topic.

4. If Service bus type is Topic, select Subscription name and select the desired subscription.

5. Select Receive mode and specify how messages are handled on receipt:

   • Select ReceiveAndDelete to read the message and remove it from the queue or subscription in a single atomic operation.

   • Select PeekLock to lock the message while it is being processed. If the message is successfully processed, it will be removed from the queue or subscription. If the message is not successfully processed, the message will be returned to the queue or subscription.

6. If you have a downstream Azure Service Bus Acknowledge tool, Receive mode must be set to PeekLock.

7. Optionally, select Triggered receipt and choose Commit to display the T input connector, and then connect the desired input tool.

8. Select Message properties and specify how message properties are provided:

   • Select None if there are no defined message properties.

   • Select Inline if message properties are provided as a JSON document embedded in each input record.

   • Select Connector if message properties are provided in a separate data stream.

9. If you selected Inline in step 7, choose Inline properties field and select the field containing the JSON document describing the record's properties.

10. If you selected Connector in step 7, select Commit to display the M connector, and then connect the desired output tool.

11. Optionally, configure Limits: Maximum batch size, Receive timeout, Limit messages, and Limit time.

12. Optionally, select override shared settings.

13. Optionally, go to the Execution tab, and then set Web service options.

Azure Service Bus Acknowledge tool

Azure Service Bus Acknowledge tool allows messages received in Peek and lock mode to be completed, abandoned, or sent to the dead letter queue. This is especially useful for only-once processing use cases.

NEW IN 9.6.1

Recent API changes in the Azure SDK client for Service Bus require that when planning an upgrade to 9.6.1 or later, you should plan to first modify your projects if…

• The project was created in Data Management version 9.6.1 or earlier

• The project receives messages in PeekLock mode using the Azure Service Bus Receive tool

• The project completes/abandons those messages using the Azure Service Bus Acknowledge tool

In such cases, the project must be modified such that the Sequence Number field accompanying each record emitted by the Service Bus Receive tool remains in the record stream until it reaches the Service Bus Acknowledge tool intended to complete or abandon the received message.

This change can be made prior to the upgrade in pre-9.6 versions of Data Management, as the presence of this field in the record stream will not impact the Service Bus Acknowledge tool’s operation.

• This tool can only be used on messages provided by an upstream Azure Service Bus Receive tool configured with a Receive mode of PeekLock.

• Receive and Acknowledge tools must share the same Service Bus Client; refer to Shared Service Bus Client for Receive and Acknowledge tools for details.

Azure Service Bus Acknowledge tool configuration parameters

Azure Service Bus Acknowledge tool has a single set of configuration parameters in addition to the standard execution options.

Configure the Azure Service Bus Acknowledge tool

Before configuring an Azure Service Bus tool, you should have an Azure Service Bus connection defined in shared settings.

1. Select the Azure Service Bus Acknowledge tool.

2. Go to the Configuration tab on the Properties pane.

3. Select Service bus type and choose the communication mechanism for this Azure Service Bus namespace: Queue or Topic.

4. If Service bus type is Topic, select Subscription name and choose the desired subscription.

5. Select the Lock token field whose contents represent the message's lock token UUID.

6. Select the Acknowledgment to send:

   • Complete

   • Abandon

   • Dead Letter

7. Optionally, override shared settings.

8. Optionally, go to the Execution tab, and then set Web service options.