Azure Storage Queue tools
Overview
Azure Queue Storage is Microsoft's cloud-based service for storing large numbers of messages. You can access messages via authenticated calls using HTTP or HTTPS. A queue message can be up to 64 KB in size, and a queue may contain millions of messages, up to the total capacity limit of a storage account. Data Management's Azure Storage Queue tools support sending, receiving, and deleting messages.
Azure Storage Queue tool shared settings
Data Management's tools use shared settings, which allows you to define a single set of configuration properties (typically access credentials) to share across multiple tools in your Data Management Site. You can override these settings on a per-tool basis by opening the Shared settings section on the tool's Properties pane, selecting Override shared settings, and specifying values for that specific tool.
To define Azure Storage Queue shared tool settings
Open the Tools folder under Settings in the repository.
Select the Azure Storage Queue tab, and then configure the tool properties for your environment:
Property | Description |
|---|---|
Authentication method | Method to use for authentication:
|
Connection string | If Authentication method is Connection String, connection string for this Azure Storage account. |
Account name | If Authentication method is SAS Token, Azure Storage account name. |
SAS token | If Authentication method is SAS Token, Azure SAS token conferring the desired permissions for this storage queue. |
To configure default shared tool settings from an Azure Service Bus tool's Properties pane, open the Shared settings section, and select Edit default settings.
To override Azure Storage Queue tool shared settings
Select the desired Azure Storage Queue tool, and then go to the Configuration tab on the Properties pane.
Open the Shared settings section, select Override shared settings, and specify new values for the tool.
Workers and performance considerations
The Workers property specifies the degree to which the Azure Storage Queue Send, Receive, and Delete tools parallelize their interactions with the Azure Storage Queue API. If more workers are configured, the tool makes more requests in parallel. Throughput scales with the number of workers, up to a point. Setting the worker count too high can result in resource starvation on the Data Management server, or quota exhaustion in Azure Storage Queue. The optimal worker count will vary based on many factors, including system resources as well as bandwidth and latency between the Data Management server and the Azure Storage Queue endpoint.
Operation timeouts and performance considerations
The Operation timeout property specifies the period of time each worker will wait for a request to complete.
Longer timeouts result in more "patient" workers, but may limit throughput as workers wait longer for their requests to complete and pending messages back up waiting for idle workers.
Shorter timeouts may yield higher throughput, but may result in a higher proportion of failed requests due to "impatient" workers abandoning long-running requests.
The Azure Storage Queue tools do not retry requests that have timed out; your solution should implement retry logic for any messages abandoned due to operation timeouts.
Azure Storage Queue Send tool
The Azure Storage Queue Send tool adds messages to an Azure storage queue.
Azure Storage Queue Send tool configuration parameters
The Azure Storage Queue Send tool has two sets of configuration parameters in addition to the standard execution options.
Configuration
Parameter | Description |
|---|---|
Queue name | Name of the storage queue. |
Workers | Number of concurrent workers to enlist in sending messages. See Workers and performance considerations. |
Operation timeout (seconds) | Number of seconds to wait before timing out a send operation. Optional; if set to |
Value field | Field containing the message value. The maximum message size for Azure Storage Queue is 64KB. |
Time to live (seconds) | Period of time the message will stay alive in the queue. Defaults to |
Initial visibility delay (seconds) | Period of time when the message is invisible in the queue. Defaults to |
Override shared settings |
Options
Parameter | Description |
|---|---|
Enable trigger output | If selected, the tool will emit a record to the "T" Trigger output when it is finished sending messages. See Trigger input and output. |
Include input fields | If selected, fields sent to the input connector are passed through to the output connector. This simplifies associating input records with messages sent to the queue. |
Configure the Azure Storage Queue Send tool
Before configuring an Azure Storage Queue Send tool, you should have an Azure Storage Queue connection defined in shared settings.
Select the Azure Storage Queue Send tool.
Go to the Configuration tab on the Properties pane.
Enter the Queue name.
Enter the number of concurrent Workers, and optionally specify the Operation timeout.
Select the Value field containing the message value.
Optionally, configure the message handling options.
Optionally, override shared settings.
Optionally, go to the Options tab to enable trigger output or include input fields.
Optionally, go to the Execution tab, and then set Web service options.
Azure Storage Queue Receive tool
The Azure Storage Queue Receive tool receives messages from an Azure storage queue.
Azure Storage Queue Receive tool configuration parameters
The Azure Storage Queue Receive tool has two sets of configuration parameters in addition to the standard execution options.
Configuration
Parameter | Description |
|---|---|
Queue name | Name of the storage queue. |
Workers | Number of concurrent workers to enlist in receiving messages. See Workers and performance considerations. |
Operation timeout (seconds) | Number of seconds to wait before timing out a receive operation. Optional; if set to |
Batch size | Number of messages to receive per API call to the storage queue. |
Visibility timeout (seconds) | Period of time when the message is invisible in the queue, preventing other consumers from receiving and processing the message. If a message’s visibility timeout expires between message receipt and deletion, the message will again become visible in the queue, making it eligible for redelivery. This can result in the delivery of duplicate messages. If the intermediary tools separating your Receive and Delete tools slow the processing of messages, consider increasing Visibility timeout to ensure downstream tools have time to process records before your message lease expires. |
Limit messages | Select Limit messages to specify Message limit as the maximum number of messages to read before exiting. |
Limit time | Select Limit time to specify Time limit as the maximum amount of time (in seconds) that may elapse before exiting. If both Limit messages and Limit time are selected, the tool will exit when the first of these thresholds is exceeded, even if more messages are available on the queue. The Message limit and Time limit values should be considered approximate. Due to interplay between Batch size and Workers, the Azure Storage Queue Receive tool cannot safely interrupt a call to receive messages without risking loss of messages, so it is possible for the number of received messages to exceed Message limit. |
Override shared settings |
Options
Parameter | Description |
|---|---|
Enable trigger input | If selected, the tool will wait until a record is supplied to the "T" Trigger input before reading messages. See Trigger input and output. |
Configure Azure Storage Queue Receive tool
Before configuring an Azure Storage Queue Receive tool, you should have an Azure Storage Queue connection defined in shared settings.
Select the Azure Storage Queue Receive tool.
Go to the Configuration tab on the Properties pane.
Enter the Queue name.
Enter the number of concurrent Workers, and optionally specify the Operation timeout.
Optionally, configure message handling options.
Optionally, override shared settings.
Optionally, go to the Options tab and enable trigger input.
Optionally, go to the Execution tab, and then set Web service options.
Azure Storage Queue Delete tool
Azure Storage Queue Delete tool removes messages from an Azure storage queue.
Azure Storage Queue Delete tool configuration parameters
Azure Storage Queue Delete tool has two sets of configuration parameters in addition to the standard execution options:
Configuration
Parameter | Description |
|---|---|
Queue name | Name of the storage queue. |
Workers | Number of concurrent workers to enlist in deleting messages. See Workers and performance considerations. |
Operation timeout (seconds) | Number of seconds to wait before timing out a delete operation. Optional; if set to 0, Azure Storage Queue’s default timeout will be used. See Operation timeouts and performance considerations. |
Message ID field | Field containing the ID of the message to be deleted. |
Pop receipt field | Field containing the pop receipt of the message to be deleted. |
Override shared settings |
Options
Parameter | Description |
|---|---|
Enable trigger output | If selected, the tool will emit a record to the "T" Trigger output when it is finished sending messages. See Trigger input and output. |
Configure Azure Storage Queue Delete tool
Before configuring an Azure Storage Queue Delete tool, you should have an Azure Service Bus connection defined in shared settings.
Select the Azure Storage Queue Delete tool.
Go to the Configuration tab on the Properties pane.
Enter the Queue name.
Enter the number of concurrent Workers, and optionally specify the Operation timeout.
Select the Message ID field containing the message ID of the message to delete.
Select the Pop receipt field containing the pop receipt of the message to delete.
Optionally, override shared settings.
Optionally, go to the Options tab and enable trigger output.
Optionally, go to the Execution tab, and then set Web service options.