Azure Service Bus tools

Overview

Azure Service Bus is a multi-tenant cloud messaging service that sends information between applications and services. Data Management has three tools for interacting with Azure Service Bus: the Azure Service Bus Send tool, Azure Service Bus Receive tool, and the Azure Service Bus Acknowledge tool.

The Azure Service Bus Send and Receive tools support batch transactions. Instead of sending each message as an individual request to Azure Service Bus, messages are collected and sent in batches. This can dramatically improve tool throughput. Optimal batch size depends on the message payload size, the message volume, and other factors.

The Azure Service Bus Acknowledge tool does not support batched transactions.

Azure Service Bus tool shared settings

Data Management's tools use shared settings, which allows you to define a single set of configuration properties (typically access credentials) to share across multiple tools in your Data Management Site. You can override these settings on a per-tool basis by opening the Shared settings section on the tool's Properties pane, selecting Override shared settings, and specifying values for that specific tool.

To define Azure Service Bus shared tool settings

Go to Settings in the repository.
Open the Tools folder.
Select the Azure Service Bus tab, and then configure the tool properties for your environment.

Property	Description
Namespace	Namespace of the Azure Service Bus instance.
SAS key name	Shared access signature (SAS) policy key for Namespace.
SAS key	Shared access signature (SAS) policy key (or Key Vault reference) for Namespace.
Service name	Name of the queue or topic.

To configure default shared tool settings from an Azure Service Bus tool's Properties pane, open the Shared settings section, and select Edit default settings.

To override Azure Service Bus tool shared settings

Select the desired Azure Service Bus tool.
Go to the Configuration tab on the Properties pane.
Open the Shared settings section, select Override shared settings, and specify new values for the tool.

Azure Service Bus tool message properties

The Azure Service Bus Send and Azure Service Bus Receive tools support setting message properties when sending and receiving messages. Properties may be defined inline, or via a separate data stream that you connect to the tool.

When sending large volumes of messages with properties, the "inline" strategy is preferred, as it uses memory more efficiently.

Define properties inline

Inline properties are provided as a JSON document embedded in each record delivered via the Azure Service Bus tool's "M" Messages connector. Each record must include a field of TextVar type that contains this JSON document. Properties are specified as name/value pairs like this:

JSON

{
  "name" : "Han Solo",
  "occupation" : "nerf herder",
  "balance" : 0
}

When you configure your Receive tool, set Message properties to Inline, and specify the Inline properties field.

Define properties via a connector

Message properties can also be provided via a separate data stream using the Message properties connector.

To connect an Azure Service Bus tool to a message properties data stream:

Select the desired Azure Service Bus tool.
Go to the Configuration tab on the Properties pane.
Select Message properties and choose Connector.
Select Commit to display the M1 input or output connector, and then connect the desired input or output tool.

Records provided to the connector must match the following schema.

Field	Type	Description
`MessageId`	TextVar	ID of the message to which this property belongs.
`Name`	TextVar	Key of this property.
`Value`	TextVar	Value of this property.

Azure Service Bus Receive tool will output properties to the M1 output connector, with each record containing a MessageId and field, allowing you to associate properties with their messages.

Azure Service Bus Send tool

The Azure Service Bus Send tool allows you to send messages to an Azure Service Bus queue or topic.

The M "Message" input connector accepts a stream of records that includes one field to be passed as the message body.
The S "Sent" output connector emits a stream of records representing successfully sent messages.
The F "Failed" output connector emits a stream of records representing messages that could not be sent, and error messages describing the failure.

Azure Service Bus Send tool configuration parameters

The Azure Service Bus Send tool has a single set of configuration parameters in addition to the standard execution options.

Parameter	Description
Service bus type	Communication mechanism for this Azure Service Bus namespace: Queue: allows one-directional communication. Each queue acts as an intermediary between message producers and consumers, storing sent messages until they are received. Each message is received by a single recipient. Topic: provides one-directional communication using subscriptions. In contrast to queues, in which each message is processed by a single consumer, topics and subscriptions provide a one-to-many form of communication, in a publish/subscribe pattern. A single topic can have multiple subscriptions. Like a queue, a topic acts as an intermediary, but each subscription can optionally use a filter to receive only messages that match specific criteria.
Subscription name	If service bus type is Topic, name of the subscription.
Message ID field	Specifies the field in the record provided via the M input connector whose contents will be sent as the message ID. If your records do not include field suitable for use as a natural key, you should generate one (for example, by using the Number Records tool).
Message body field	Field in the record provided via the M input connector whose contents will be sent as the message body.
Message properties Inline properties field	See Azure Service Bus tool message properties.
Maximum batch size	The maximum number of messages to be sent in each batch.
Override shared settings	See Azure Service Bus tool shared settings.

Configure the Azure Service Bus Send tool

Before configuring an Azure Service Bus Send tool, you should have an Azure Service Bus connection defined in shared settings.

Select the Azure Service Bus Send tool.
Go to the Configuration tab on the Properties pane.
Select Service bus type and choose the communication mechanism for this Azure Service Bus namespace: Queue or Topic.
- If Service bus type is Topic, select Subscription name and select the desired subscription.
Select Message ID field and select the field in the record provided via the M input connector whose contents will be sent as the message ID.
Select Message body field and select the field containing the Unicode string to be sent to the queue as the message's body.
Select Message properties and specify how message properties are provided:
- Select None if there are no defined message properties.
- Select Inline if message properties are provided as a JSON document embedded in each input record.
- Select Connector if message properties are provided in a separate data stream.
If you selected Inline in step 6, select Inline properties field and select the field containing the JSON document describing the record's properties.
If you selected Connector in step 6, select Commit to display the M connector, and then connect the desired input tool.
Optionally, select Maximum batch size and specify the maximum number of messages to be sent with each batch.
Optionally, override shared settings.
Optionally, go to the Execution tab, and then set Web service options.

Azure Service Bus Receive tool

The Azure Service Bus Receive tool allows you to receive messages from an Azure Service Bus queue.

The optional T "Trigger" input connector causes the Azure Service Bus Receive tool tool to call "receive" on the queue or subscription for each record sent to the input connector.
The M "Message" output connector emits a stream of records representing received messages.
The F "Failure" output connector emits a stream of records representing receipt errors.

Azure Service Bus Receive tool configuration parameters

The Azure Service Bus Receive tool has a single set of configuration parameters in addition to the standard execution options.

Parameter	Description
Service bus type	Communication mechanism for this Azure Service Bus namespace: Queue: allows one-directional communication. Each queue acts as an intermediary between message producers and consumers, storing sent messages until they are received. Each message is received by a single recipient. Topic: provides one-directional communication using subscriptions. In contrast to queues, in which each message is processed by a single consumer, topics and subscriptions provide a one-to-many form of communication, in a publish/subscribe pattern. A single topic can have multiple subscriptions. Like a queue, a topic acts as an intermediary, but each subscription can optionally use a filter to receive only messages that match specific criteria.
Subscription name	If service bus type is Topic, name of the subscription.
Receive mode	Specifies how messages are handled upon receipt: ReceiveAndDelete: destructive reads in which the message is delivered to a consumer and removed from the queue or subscription in a single atomic operation. PeekLock: non-destructive reads. When a message is delivered to a consumer, it is locked. The message is guaranteed not to be delivered to any other consumers during that lock period. During that lock period, the consumer will attempt to process the message. If the consumer completes the message, it will be removed from the queue/subscription. If the consumer fails to process the message and explicitly abandons it, or if the lock period expires, the message will be returned to the queue or subscription where it becomes available to another consumer. Receive mode must be set to PeekLock if the data flow will contain a downstream Acknowledge tool.
Triggered receipt	When selected, the Receive tool will sprout a T "Trigger" input connector. For each record sent to the input connector, the Receive tool will call "receive" on the queue or subscription. When cleared, the Receive tool will continuously call receive until one of the following occurs: Time limit is exceeded. Message limit is exceeded. Receive timeout expires before the queue/subscription returns any messages.
Message properties Inline properties field	See Azure Service Bus tool message properties.
Maximum batch size	The maximum number of messages to be received in each batch. Fewer messages may be returned if the batch size exceeds the number of available messages, or the Receive timeout expires before the batch is filled.
Receive timeout	Duration (in seconds) that the Azure Service Bus Receive tool will wait for messages to be returned from the queue or topic. The interplay between the Receive timeout and the queue or subscription's Lock duration: a long receipt timeout combined with a short lock duration can lead to acknowledgment failures in peek/lock mode.
Limit messages Message limit	Select Limit messages to specify Message limit as the maximum number of messages to read before exiting.
Limit time Time limit	Select Limit time to specify Time limit as the maximum amount of time (in seconds) that may elapse before exiting. This is different from the Receive timeout parameter in that this time limit will be enforced even if more messages are available on the queue. If both Limit messages and Limit time are selected, the tool will exit when the first of these thresholds is exceeded, even if more messages are available on the queue. See also "The effects of batching on limits" below.
Override shared settings	See Azure Service Bus tool shared settings.

The effects of batching on limits

Time and message limits specified with the Limit messages and Limit time options should be considered an approximation. The Receive tool cannot interrupt a call to receive, so it is possible for the number of received messages to exceed the defined message limit, especially when processing larger batches.

Configure the Azure Service Bus Receive tool

Before configuring an Azure Service Bus Receive tool, you should have an Azure Service Bus connection defined in shared settings.

Select the Azure Service Bus Receive tool.
Go to the Configuration tab on the Properties pane.
Select Service bus type and select the communication mechanism for this Azure Service Bus namespace: Queue or Topic.
If Service bus type is Topic, select Subscription name and select the desired subscription.
Select Receive mode and specify how messages are handled on receipt:
- Select ReceiveAndDelete to read the message and remove it from the queue or subscription in a single atomic operation.
- Select PeekLock to lock the message while it is being processed. If the message is successfully processed, it will be removed from the queue or subscription. If the message is not successfully processed, the message will be returned to the queue or subscription.
If you have a downstream Azure Service Bus Acknowledge tool, Receive mode must be set to PeekLock.
Optionally, select Triggered receipt and choose Commit to display the T input connector, and then connect the desired input tool.
Select Message properties and specify how message properties are provided:
- Select None if there are no defined message properties.
- Select Inline if message properties are provided as a JSON document embedded in each input record.
- Select Connector if message properties are provided in a separate data stream.
If you selected Inline in step 7, choose Inline properties field and select the field containing the JSON document describing the record's properties.
If you selected Connector in step 7, select Commit to display the M connector, and then connect the desired output tool.
Optionally, configure Limits: Maximum batch size, Receive timeout, Limit messages, and Limit time.
Optionally, select override shared settings.
Optionally, go to the Execution tab, and then set Web service options.

Azure Service Bus Acknowledge tool

Azure Service Bus Acknowledge tool allows messages received in Peek and lock mode to be completed, abandoned, or sent to the dead letter queue. This is especially useful for only-once processing use cases.

This tool can only be used on messages provided by an upstream Azure Service Bus Receive tool configured with a Receive mode of PeekLock.

Azure Service Bus Acknowledge tool configuration parameters

Azure Service Bus Acknowledge tool has a single set of configuration parameters in addition to the standard execution options.


Service bus type	Communication mechanism for this Azure Service Bus namespace: Queue: Allows one-directional communication. Each queue acts as an intermediary between message producers and consumers, storing sent messages until they are received. Each message is received by a single recipient. Topic: Provides one-directional communication using subscriptions. In contrast to queues, in which each message is processed by a single consumer, topics and subscriptions provide a one-to-many form of communication, in a publish/subscribe pattern. A single topic can have multiple subscriptions. Like a queue, a topic acts as an intermediary, but each subscription can optionally use a filter to receive only messages that match specific criteria.
Subscription name	If service bus type is Topic, name of the subscription.
Lock token field	Field in the record provided via the M "Message" input connector whose contents represent the message's lock token UUID.
Acknowledgment	Acknowledgment to send for this message: Complete: The message was successfully processed and it should be deleted from the queue or topic. Abandon: The receiver doesn’t know how to process the message and it returns the message to the queue or topic to make it available for other receivers. DeadLetter: The receiver has encountered an error processing the message and marks it as a dead-letter. Dead-letters are sent to a sub-queue within the same queue or topic.
Override shared settings	See Azure Service Bus tool shared settings.

Configure the Azure Service Bus Acknowledge tool

Before configuring an Azure Service Bus tool, you should have an Azure Service Bus connection defined in shared settings.

Select the Azure Service Bus Acknowledge tool.
Go to the Configuration tab on the Properties pane.
Select Service bus type and choose the communication mechanism for this Azure Service Bus namespace: Queue or Topic.
If Service bus type is Topic, select Subscription name and choose the desired subscription.
Select the Lock token field whose contents represent the message's lock token UUID.
Select the Acknowledgment to send:
- Complete
- Abandon
- Dead Letter
Optionally, override shared settings.
Optionally, go to the Execution tab, and then set Web service options.