SQS tools

Overview

Amazon Simple Queue Service (SQS) is a distributed message queuing service hosted by Amazon Web Services (AWS) that supports programmatic sending of messages via web service applications.

Data Management has three tools for interacting with Amazon SQS: the SQS Send tool, the SQS Read tool, and the SQS Delete tool.

SQS tool shared settings

Data Management's SQS tools use shared settings, which allows you to define a single set of configuration properties (typically access credentials) to share across multiple tools in your Data Management Site. You can override these settings on a per-tool basis by opening the Shared settings section on the tool's Properties pane, selecting Override shared settings, and specifying values for that specific tool.

To define SQS shared tool settings

Open the Tools folder under Settings in the repository.
Select the SQS tab, and then configure the tool properties for your environment.

Property	Description
Authentication	Method used to authenticate to the SQS service: Explicit: provide an Amazon Web Services (AWS) Access key and Secret (or Key Vault reference). IAM: authenticate using an Identity and Access Management (IAM) role defined on the Amazon Elastic Compute Cloud (EC2) instance. The following conditions must be true: Data Management is running on an EC2 instance. An IAM role is attached to that EC2 instance via its instance profile. The AWS account and IAM role have the following permissions for the queue specified by each tool's Queue name property: `ChangeMessageVisibility` `DeleteMessage` `GetQueueAttributes` `GetQueueUrl` `ListQueues` `ReceiveMessage` `SendMessage`
Access key	If Authentication is Explicit, one half of the AWS credential pair (for example, `AKIAIOSFODNN7EXAMPLE`).
Secret	If Authentication is Explicit, the other half of the AWS credential pair (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
SQS region	AWS region in which the queue is located.

To configure default shared tool settings from an SQS tool's Properties pane, open the Shared settings section, and select Edit default settings.

To override SQS tool shared settings

Select the desired SQS tool.
Go to the Configuration tab on the Properties pane.
Open the Shared settings section, select Override shared settings, and specify new values for the tool.

SQS message attributes

The SQS Send tool and the SQS Read tool support setting message attributes when sending and receiving messages. Message attributes are optional and distinct from both the message body and from simple attributes (the fixed metadata that SQS defines on every message).

Message attributes may be defined inline, or via a separate data stream that you connect to the tool. When sending large volumes of messages with attributes, the Inline strategy is preferred as it uses memory more efficiently.

Define attributes inline

Inline attributes are provided as a JSON document embedded in each record delivered via the SQS tool's "M" Messages connector. Each record must include a field of TextVar type that contains this JSON document. Attributes are specified as name/value pairs like this:

JSON

{
    "name" : "Han Solo",
    "occupation" : "nerf herder",
    "balance" : 0
}

Define attributes via a connector

Message attributes can also be provided via a separate data stream using the "M1" Message attributes connector.

To connect a SQS tool to a message attributes data stream:

Select the desired SQS Send or SQS Read tool.
Go to the Configuration tab on the Properties pane.
Select Message attributes.
Select Connector.
Select Commit to display the M1 input or output connector, and then connect the desired input or output tool.

Records provided to the connector must match the following schema.

Field	Type	Description
`MessageId`	`TextVar`	ID of the message to which this attribute belongs, allowing you to associate attributes with their messages.
`Key`	`TextVar`	Key of this attribute.
`Value`	`TextVar`	Value of this attribute.

SQS Send tool

The SQS Send tool allows you to send messages to an SQS queue.

The "M" Messages input connector accepts a stream of records that includes fields for the message ID and body.
The optional "M1" Messages input connector accepts a data stream containing message attributes.
The "S" Success output connector emits a stream of records representing successfully sent messages. Each record contains the following fields.

Field	Description
`ID`	User-defined record ID.
`MessageId`	The system-generated message ID.
`MD5OfBody`	An MD5 checksum of the message body.

The "F" Failure output connector emits a stream of records representing messages that could not be sent. Each record contains the following fields.

Field	Description
`ID`	User-defined record ID.
`FailureCode`	The system-generated failure code.
`FailureMessage`	The system-generated failure message.
`SenderFault`	Indicates whether the sender is to blame for the failure.

SQS Send tool configuration parameters

The SQS Send tool has a single set of configuration parameters in addition to the standard execution options.

Parameter	Description
Queue name	Name of the queue to which this tool will send messages.
Enable trigger output	If selected, the tool will emit a record to the "T" Trigger output when it is finished sending messages. See Trigger input and output.
Message ID field	Field in the record provided via the SQS tool's "M" Messages connector whose contents will be sent as the message ID. If your records do not include field suitable for use as a natural key, you should generate one (perhaps using the Number Records tool). Additionally, if Message attributes is Connector, the Message ID field specifies the field containing join keys that will be used to associate incoming message attributes with the messages to which they belong.
Message body field	Field in the record provided via the M input connector whose contents will be sent to the queue as the message's body.
Message attributes	Specifies how message attributes are delivered: None: no message properties are defined. Inline: message properties are provided as a JSON document embedded in each input record delivered via the SQS tool's "M" Messages connector. If you are handling a large volume of messages, choose this option. Connector: message attributes are provided in a separate data stream using the M1 connector. This option reads the entire set of message attributes into memory. Do not use if you are handling a large volume of messages.
Inline attributes field	If Message attributes is Inline, the field containing the JSON document describing the record's attributes.
FIFO queue	If selected, the queue specified by Queue name is treated as a FIFO queue.
Message group ID field	If FIFO queue is selected, field in the record whose value designates the message's group. FIFO queues only guarantee ordering of messages in the same group.
Content-based deduplication	If FIFO queue is selected and the queue specified in Queue name is configured for content-based deduplication, select this to enable content-based deduplication for the queue.
Message deduplication ID	If FIFO queue is selected and Content-based deduplication is not enabled, specify the field whose value provides the message's deduplication ID.
Override shared settings	See SQS tool shared settings.

Configure the SQS Send tool

Before configuring an SQS tool, you should have an SQS connection defined in shared settings.

Select the SQS Send tool.
Go to the Configuration tab on the Properties pane.
Enter the Queue name.
Optionally, Enable Trigger output, and select Commit to display the T connector.
Select Message ID field and select the field whose contents will be sent to the queue as the message ID.
Select Message body field and select the field whose contents will be sent to the queue as the message body.
Select Message attributes and specify how message attributes are provided:
- Select None if there are no defined message attributes.
- Select Inline if message attributes are provided as a JSON document embedded in each input record.
- Select Connector if message attributes are provided in a separate data stream.

This option reads the entire set of message attributes into memory. Do not use if you are handling a large volume of messages.

If you selected Inline in step 6, select Inline attributes field and select the field containing the JSON document describing the record's properties.
If you selected Connector in step 6, select Commit to display the M1 connector, and then connect the desired input.
Optionally, select FIFO queue to treat Queue name as a FIFO queue.
- Select Message group ID field and specify the field in the record whose value designates the message's group.
- If the queue specified in Queue name is configured for content-based deduplication, select Content-based deduplication to enable content-based deduplication for the queue.
- If Content-based deduplication is not selected, Select Message group ID field and specify the field in the record whose value provides the message's deduplication ID.
Optionally, override shared settings.
Optionally, go to the Execution tab, and then set Web service options.

SQS Read tool

The SQS Read tool allows you to receive messages from an SQS queue.

The "M" Messages output connector emits a stream of records representing received messages.
The optional "M1" Messages output connector emits a data stream containing message attributes.
The "A" Attributes output connector emits a data stream containing the fixed metadata that AWS defines on every message.
The "F" Failure output connector emits a stream of records representing receipt errors.

SQS Read tool configuration parameters

The SQS Read tool has a single set of configuration parameters in addition to the standard execution options.

Parameter	Description
Queue name	Name of the queue from which this tool will read messages.
Enable trigger input	If selected, the tool will wait until a record is supplied to the "T" Trigger input before reading messages. See Trigger input and output.
Limit messages Message limit	Select Limit messages to specify Message limit as the maximum number of messages to read before exiting.
Limit time Time limit (seconds)	Select Limit time to specify Time limit as the maximum amount of time (in seconds) that may elapse before exiting. If both Limit messages and Limit time are selected, the tool will exit when the first of these thresholds is exceeded, even if more messages are available on the queue. The specified time and message limits should be considered an approximation. The SQS Read tool cannot interrupt a call to `receiveMessage`, so it's possible for the number of received messages to exceed message limits.
Visibility timeout (seconds)	Period of time after a message is received during which Amazon SQS prevents other consumers from receiving and processing the message. The minimum is 0 seconds. The maximum is 12 hours.
Wait time (seconds)	The duration in seconds that the tool will wait for messages to be returned from the queue. Interaction between Wait time and Visibility timeout: a long wait time combined with a short visibility timeout can lead to a message's visibility timing out before the message is passed to downstream tools for processing; this can result in the same message being received multiple times.
With attributes	Specifies which of the fixed metadata that SQS defines on every message to include in a separate data stream using the "A" connector. This is different and distinct from Message attributes. Default is All.
Message attributes	Specifies how message attributes are delivered: Inline: message properties are provided as a JSON document embedded in each input record. Connector: message attributes are provided in a separate data stream using the M1 connector. None: no message properties are defined.
With message attributes	If Message attributes is Inline or Connector, the list of message attributes to include.
FIFO queue	If selected, the queue specified by Queue name is treated as a FIFO queue. When FIFO queue is enabled, messages are immediately deleted on receipt. The Delete tool cannot delete messages received from a FIFO queue.
Override shared settings	See SQS tool shared settings.

Configure the SQS Read tool

Before configuring an SQS tool, you should have an SQS connection defined in shared settings.

Select the SQS Read tool.
Go to the Configuration tab on the Properties pane.
Enter the Queue name.
Optionally, Enable Trigger input, and select Commit to display the T connector.
Optionally, configure message handling options.
Optionally, select With attributes and specify which message attributes to include.
Select Message attributes and specify how message attributes are provided:
1. Select None if there are no defined message attributes.
2. Select Inline if message attributes are provided as a JSON document embedded in each input record.
3. Select Connector if message attributes are provided in a separate data stream.
If you selected Inline in step 6, select Message attributes field and select the field containing the JSON document describing the record's attributes.
If you selected Connector in step 6, select Commit to display the M1 connector, and then connect the desired output.
Optionally, select FIFO queue to treat the queue specified by Queue name as a FIFO queue. Note that when this option is is enabled, messages are immediately deleted on receipt. The SQS Delete tool cannot delete messages received from a FIFO queue.
Optionally, override shared settings.
Optionally, go to the Execution tab, and then set Web service options.

SQS Delete tool

The SQS Delete tool allows you to remove messages from a SQS queue.

The "M" Messages input connector accepts a stream of records that includes fields for the Message ID and Receipt handle.

The Delete tool cannot be used to delete messages received from a FIFO queue, as these are deleted immediately on receipt.

The "S" Success output connector emits a stream of records representing successfully deleted messages. Each record contains an ID field representing the message ID.
The "F" Failure output connector emits a stream of records representing messages that could not be deleted. Each record contains a single message ID field representing the message ID, a Code field representing the error code, and a Message field describing the error.

SQS Delete tool configuration parameters

The SQS Delete tool has a single set of configuration parameters in addition to the standard execution options.

Parameter	Description
Queue name	Name of the queue from which this tool will read messages.
Message ID field	Field provided via the M input connector containing the message ID of the message to delete.
Receipt handle field	Field provided via the M input connector containing the message's receipt handle.
Override shared settings	See SQS tool shared settings.

Configure the SQS Delete tool

Before configuring an SQS tool, you should have an SQS connection defined in shared settings.

Select the SQS Delete tool.
Go to the Configuration tab on the Properties pane.
Enter the Queue name.
Select the Message ID field containing the message ID of the message to delete.
Select the Receipt handle field containing the message's receipt handle.
Optionally, override shared settings.
Optionally, go to the Execution tab, and then set Web service options.