JMS tools

The Java Message Service (JMS) is a system for transferring data as messages across multiple platforms using Apache ActiveMQ or IBM MQ providers. Participating applications can run independently of each other, at different speeds and times, in different locations, and without having a direct connection.

Data Management has three tools for interacting with JMS:

You can deploy multiple JMS tools in your dataflow to increase throughput via parallelism. This may be especially useful when the JMS Receive tool is configured for client-acknowledged mode, which sacrifices throughput for reliability. However, observe the following guidance when using "ganged" JMS Receive and JMS Acknowledge tools:

The JMS Receive and JMS Acknowledge tools are designed to work in collaborating pairs. Each Acknowledge tool must be configured to acknowledge messages from one and only one Receive tool. Likewise, each Receive tool expects its messages to be acknowledged by a single Acknowledge tool.

JMS tool shared settings

Data Management's JMS tools use shared settings, which allows you to define a single set of configuration properties (typically access credentials) to share across multiple tools in your Data Management Site. You can override these settings on a per-tool basis by opening the Shared settings section on the tool's Properties pane, selecting Override shared settings, and specifying values for that specific tool.

To define JMS shared tool settings

Open the Tools folder under Settings in the repository.
Select the JMS tab, and then go to the Properties pane.
Configure the tool properties for your environment:

Provider	JMS Provider: either IBMMQ or ACTIVEMQ.
Host	Host name or IP address of the JMS server.
Port	Port on which the JMS server listens.
Channel	If Provider is IBMMQ, the IBMMQ channel (typically a Server Connection Channel used by clients to connect to a Queue Manager).
Use SSL	If selected, JMS tools require connections to be secured with SSL.
Cipher suite	If Use SSL is selected, the list of available SSL ciphers for IBM MQ. These are provided by the Java virtual machine and may vary between systems. Check with your system administrator to determine which SSL ciphers are supported by your JMS broker.
TrustStore location	If Use SSL is selected, path to the TrustStore that contains certificates from certificate authorities, which verify certificates presented by the server in the SSL connection.
TrustStore password	If Use SSL is selected, password (or Key Vault reference) for the specified TrustStore.
Authenticate using credentials	If selected, JMS tools attempt to create an authenticated JMS connection using specified the Username and Password.
Username	If Authenticate using credentials is selected, user name to provide when creating an authenticated JMS connection.
Password	If Authenticate using credentials is selected, password (or Key Vault reference) to provide when creating an authenticated JMS connection.
Prefetch	If Provider is ACTIVEMQ, sets ActiveMQ's prefetch limit. You may need to tune this property if you suffer from "Slow Consumer" symptoms.
QueueManager name	If Provider is IBMMQ, the name of the IBMMQ QueueManager.

To configure default shared tool settings from an JMS tool's Properties pane, open the Shared settings section, and select Edit default settings.

To override JMS tool shared settings

Select the desired JMS tool, and then go to the Configuration tab on the Properties pane.
Open the Shared settings section, select Override shared settings, and specify new values for the tool.

JMS tool message properties

The JMS Send tool and JMS Receive tool support setting and getting user-defined message properties when sending and receiving messages. Properties may be defined inline, or via a separate data stream that you connect to the tool. Additionally, system-generated message properties can also be retrieved; these properties begin with the JMS prefix.

When sending large volumes of messages with properties, the Inline strategy is preferred as it uses memory more efficiently.

Inline properties

When Message properties is set to Inline, message properties are exchanged as JSON documents in the TextVar field specified by JSON properties field.

Sending messages with inline properties

When using the JMS Send tool to send messages, Properties are specified as name/value pairs:

JSON

{
    "fullName" : "Han Solo",
    "occupation" : "nerf herder",
    "balance" : 0,
    "witted" : 0.5,
    "scruffy-looking" : true
}

You can use the JSON Input tool to create these documents, but they must be provided to the JMS Send tool as TextVar data, so you'll need to perform a conversion using the DecodeTextBytes function. If your document is simple, you can build a string using the Calculate tool.

The JMS Send tool will infer the JSON data type and convert it to the corresponding JMS data type.

Receiving messages with inline properties

When using the Receive tool to receive messages, properties are represented as an array of objects:

JSON

[
    { 
        "Name": "fullName", 
        "String": "Han Solo" 
    },
    { 
        "Name": "occupation", 
        "String": "nerf herder" 
    },
    { 
        "Name": "balance", 
        "Integer": 0 
    },
    { 
        "Name": "witted", 
        "Double": 0.5 
    },
    { 
        "Name": "scruffy-looking", 
        "Boolean": true 
    }
]

Each object represents a property with a Name attribute, accompanied by a value attribute for the appropriate JMS data type — one of Boolean, Byte, Integer, Long, Short, Double, Float, and String.

If message properties will be consumed by a downstream tool, they should be parsed into records using the JSON Input tool. When configuring a JSON Input tool to consume message properties, use this sample document for analysis:

JSON

[
    { 
        "Name": "fullName", 
        "Boolean": null, 
        "Byte": null, 
        "Integer": null, 
        "Long": null, 
        "Short": null, 
        "Double": null, 
        "Float": null, 
        "String": "Han Solo" 
    },
    { 
        "Name": "occupation", 
        "Boolean": null, 
        "Byte": null, 
        "Integer": null, 
        "Long": null, 
        "Short": null, 
        "Double": null, 
        "Float": null, 
        "String": "nerf herder" 
    },
    { 
        "Name": "balance", 
        "Boolean": null, 
        "Byte": null, 
        "Integer": 0, 
        "Long": null, 
        "Short": null, 
        "Double": null, 
        "Float": null, 
        "String": null 
    },
    { 
        "Name": "witted", 
        "Boolean": null, 
        "Byte": null, 
        "Integer": null, 
        "Long": null, 
        "Short": null, 
        "Double": 0.5, 
        "Float": null, 
        "String": null 
    },
    { 
        "Name": "scruffy-looking", 
        "Boolean": true, 
        "Byte": null, 
        "Integer": null, 
        "Long": null, 
        "Short": null, 
        "Double": null, 
        "Float": null, 
        "String": null 
    }
]

Each object in the above JSON shows the superset of possible attributes for a message property. The objects returned by the JMS Receive tool are "homogeneous but sparse" — that is, each object will contain only one of the available data type values.

Properties via a connector

Message properties can also be provided via a separate data stream using the Message properties connector.

To connect a JMS tool to a message properties data stream:

Select the desired JMS Send or JMS Receive tool, and then go to the Configuration tab on the Properties pane.
Select Message properties and select Connector.
Select Commit to display the "M1" input or output connector, and then connect the desired input or output tool.

Records provided to the connector must match the following schema:

Field	Data Management type	Description
`ID`	`TextVar`	Identifier that can be used to associate this property with the message to which it belongs via the message's Correlation ID.
`Name`	`TextVar`	Key of this attribute.
`Boolean`	`Boolean`	Value field holding the value of this property, represented as a Boolean data type.
`Byte`	`Integer(1)`	Value field holding the value of this property, represented as a Integer(1) data type.
`Double`	`Float(8)`	Value field holding the value of this property, represented as a Float(8) data type.
`Float`	`Float(4)`	Value field holding the value of this property, represented as a Float(4) data type.
`Integer`	`Integer(4)`	Value field holding the value of this property, represented as a Integer(4) data type.
`Long`	`Integer(8)`	Value field holding the value of this property, represented as a Integer(8) data type.
`Short`	`Integer(2)`	Value field holding the value of this property, represented as a Integer(2) data type.
`String`	`TextVar`	Value field holding the value of this property, represented as a TextVar data type.

Note that records should set a value on one of the value fields.

JMS acknowledgement modes

The JMS tools support three message acknowledgment modes:

Auto: In auto-acknowledged mode, messages are acknowledged immediately on receipt. This removes them from the destination, and prevents re-delivery to any other message consumers. This mode is suitable when throughput is important, and your solution can tolerate some messages being "dropped on the floor" if intermediary processing fails. This is sometimes called "No more than once" delivery semantics.
Client: In client-acknowledged mode, messages must be acknowledged explicitly after receipt (typically following the successful completion of some intermediary processing operation). Messages remain ineligible for re-delivery to other consumers until either the message is acknowledged, or the session in which that message was consumed is closed. This mode is suitable when reliability is more important than throughput; it can be used to ensure that messages are processed precisely once. This is also known as "Once and only once" delivery semantics. However, this requires some consideration during solution design because the JMS API specifies that acknowledging any message consumed during a session will acknowledge all messages consumed during that session.
Lazy: In lazy-acknowledged mode, messages are acknowledged eventually. This usually involves a time interval or message quota, but it's left to the discretion of the provider. This acknowledgment mode reduces session-related overhead and can yield increased throughput and improved memory efficiency. However, it can also result in messages being delivered multiple times, so your solution should be designed to handle re-delivery of the same message. For that reason, this acknowledgment mode is often referred to as "at-least once" delivery.

JMS Send tool

The JMS Send tool sends messages to a JMS queue or topic.

The "M" Messages input connector accepts a stream of records that includes fields for the message ID and body.
The optional "M1" Messages Properties input connector accepts a separate data stream of message properties.
The "S" Success output connector emits a stream of records representing successfully sent messages. Each record contains a single ID field: the message ID.
The "F" Fail output connector emits a stream of records representing messages that could not be sent. Each record contains a single ID field: the message ID.

JMS Send tool configuration parameters

The JMS Send tool has a single set of configuration parameters in addition to the standard execution options:

Enable trigger output	If selected, the tool will emit a record to the "T" Trigger output when it is finished sending messages. See Trigger input and output.
Destination type	Type of JMS destination to which messages will be sent: QUEUE: Allows one-directional communication. Each queue acts as an intermediary between message producers and consumers, storing sent messages until they are received. Each message is received by a single recipient. TOPIC: Provides one-directional communication using subscriptions. In contrast to queues, in which each message is processed by a single consumer, topics and subscriptions provide a one-to-many form of communication, in a publish/subscribe pattern. A single topic can have multiple subscriptions.
Destination name	Name of the queue or topic to which this tool will send messages.
Message properties	Specifies how message properties are delivered: Inline: message properties are provided as a JSON document embedded in each input record delivered via the tool's "M" Message connector Connector: message properties are provided in a separate data stream using the "M1" Message Properties connector. You must select Provide correlation ID and specify a Correlation ID field. None: no message properties are defined.
Provide correlation ID	Select Provide correlation ID to associate an arbitrary ID with each message. If your records do not include a field suitable for use as a natural key, generate one by using an upstream Number Records tool. Additionally, if Message properties is Connector, you must select Provide correlation ID and specify a Correlation ID field.
Correlation ID field	Field containing join keys that associate incoming message properties with the messages to which they belong.
JSON properties field	If Message properties is Inline, the field containing the JSON document describing the record's properties.
Message body field	Field in the record provided via the "M" Messages input connector whose contents will be sent as the message body.
Override shared settings	See JMS tool shared settings.

Configure the JMS Send tool

Before configuring a JMS tool, you should have a JMS connection defined in shared settings.

Select the JMS Send tool, and then go to the Configuration tab on the Properties pane.
Optionally, Enable Trigger output.
Select Destination type: either QUEUE or TOPIC.
Select Destination name and specify the name of the queue or topic to which this tool will send messages.
Select Message properties and specify how message properties are provided:
- Select None if there are no defined message properties.
- Select Inline if message properties are provided as a JSON document embedded in each input record.
- Select Connector if message properties are provided in a separate data stream.
If you selected Inline in step 5:
- Select JSON properties field and select the field containing the JSON document describing the record's properties.
- Optionally, select Provide correlation ID and then specify a Correlation ID field.
If you selected Connector in step 5:
Select Commit to display the M1 connector, and then connect the desired input tool.
Select Provide correlation ID and then specify a Correlation ID field.
Select Message body field and select the field whose contents will be sent as the message body.
Optionally, override shared settings.
Optionally, select the Execution tab, and then set Report options and Web service options.

JMS Receive tool

The JMS Receive tool receives messages from an JMS queue or topic.

The "M" Messages output connector emits a stream of records that includes fields for the message ID and body.
The optional "M1" Messages Properties output connector emits separate data stream of message properties.

JMS Receive tool configuration parameters

The JMS Receive tool has a single set of configuration parameters in addition to the standard execution options:

Input trigger	Specifies input triggering behavior, allowing fine-grained control over the sequencing of tool execution. The Receive tool supports three input triggering modes: None: No input triggering. The tool will immediately begin polling for messages when the project is run. Conditional: The tool will sprout a "T" Trigger connector. The tool will not begin polling for messages until all records have been read from input. If no messages are supplied to the input, the tool will not run. Unconditional: The tool will sprout a "T" Trigger connector. The tool will not begin polling for messages until all records have been read from input. Even if no messages are supplied to the input, the tool will still run.
Destination type	Type of JMS destination from which messages will be received: QUEUE: Allows one-directional communication. Each queue acts as an intermediary between message producers and consumers, storing sent messages until they are received. Each message is received by a single recipient. TOPIC: Provides one-directional communication using subscriptions. In contrast to queues, in which each message is processed by a single consumer, topics and subscriptions provide a one-to-many form of communication, in a publish/subscribe pattern. A single topic can have multiple subscriptions.
Destination name	Name of the queue or topic from which this tool will receive messages.
Acknowledgment mode	Specifies how message acknowledgment is handled: Auto: Messages will be acknowledged automatically and immediately on receipt, removing them from the destination, and preventing delivery to any other message consumers. Client: Messages must be acknowledged explicitly by a downstream JMS Acknowledge tool. Lazy: Messages will be acknowledged automatically at some point after receipt. See JMS acknowledgement modes for a detailed discussion.
Unacknowledged message buffer	If Acknowledgment mode is Client, the buffer size for messages awaiting acknowledgment. Increase this value if your project fails with the JMS Exception "The number of pending messages has exceeded the configured limit." This can indicate that slow intermediary processing is causing messages to bottleneck between your JMS Receive and JMS Acknowledge tools, allowing messages to be received much faster than they can be processed and acknowledged. If this value is too high, your project may fail with an `Out Of Memory` error. If this happens, decrease the value, or increase the project's JVM memory in project settings.
Selector	Specifies an optional JMS message selector, allowing messages to be filtered using an SQL-like syntax.
Message properties	Specifies how message properties are delivered: Inline: message properties are provided as a JSON document embedded in each input record delivered via the tool's "M" Message connector Connector: message properties are provided in a separate data stream using the "M1" Message Properties connector. None: no message properties are defined.
Receive timeout	Specify Receive timeout as the maximum duration in seconds that the tool should wait for additional messages to arrive when the queue is empty.
Limit messages Message limit	Select Limit messages to specify Message limit as the maximum number of messages to read before exiting.
Limit time Time limit	Select Limit time to specify Time limit as the maximum amount of time (in seconds) that may elapse before exiting. Note that this is different from the Receive timeout parameter in that this time limit will be enforced even if more messages are available on the queue. If both Limit messages and Limit time are selected, the tool will exit when the first of these thresholds is exceeded, even if more messages are available on the queue. If you do not specify a Time limit or Message limit, the Receive tool will run indefinitely. You should usually set a limit for one or both.
Override shared settings	See JMS tool shared settings.

Configure the JMS Receive tool

Before configuring a JMS tool, you should have a JMS connection defined in shared settings.

Select the JMS Receive tool, and then go to the Configuration tab on the Properties pane.
Optionally, select Trigger input and configure input triggering behavior.
Select Destination type: either QUEUE or TOPIC.
Select Destination name and specify the name of the queue or topic to which this tool will send messages.
Select Acknowledgment mode and specify how message acknowledgment is handled:
- Auto: Messages will be acknowledged automatically and immediately on receipt.
- Client: Messages must be acknowledged explicitly by a downstream JMS Acknowledge tool.
- Lazy: Messages will be acknowledged automatically at some point after receipt.
Optionally, specify a Selector.
Select Message properties and specify how message properties are provided:
- Select None if there are no defined message properties.
- Select Inline if message properties are provided as a JSON document embedded in each input record.
- Select Connector if message properties are provided in a separate data stream.
If you selected Connector in step 7, select Commit to display the "M1" connector, and then connect the desired input tool.
Optionally, configure message handling options.
Note that if you do not specify a Time limit or Message limit, the Receive tool will run indefinitely. You should usually set a limit for one or both.
Optionally, override shared settings.
Optionally, select the Execution tab, and then set Report options and Web service options.

JMS Acknowledge tool

The JMS Acknowledge tool acknowledges messages delivered to an upstream JMS Receive tool whose Acknowledgment mode is set to Client.

The JMS Receive and JMS Acknowledge tools are designed to work in collaborating pairs. Each Acknowledge tool must be configured to acknowledge messages from one and only one Receive tool. Likewise, each Receive tool expects its messages to be acknowledged by a single Acknowledge tool.

The input connector accepts a record stream representing messages to be acknowledged. Each record must include a Message ID field.
The output connector emits a stream of records representing successfully acknowledged messages. Each record contains a Message ID field.
The "T" Trigger output connector emits a single record indicating whether the messages were successfully acknowledged. Each record contains a Success field.

If acknowledgment fails for any message, acknowledgment will fail for all messages. Keep this in mind as you design your solution, as you may wish to orchestrate your dataflow using automations that break large volumes of message into smaller batches.

JMS Acknowledge tool configuration parameters

The JMS Acknowledge tool has a single set of configuration parameters in addition to the standard execution options:

Consumer

Upstream JMS Receive tool that will provide messages to be acknowledged.

The JMS Receive and JMS Acknowledge tools are designed to work in collaborating pairs. Each JMS Acknowledge tool must be configured to acknowledge messages from one and only one Receive tool. Likewise, each JMS Receive tool expects its messages to be acknowledged by a single Acknowledge tool.

Message ID field

Field in the record provided via the "M" input connector containing the message ID of the message to remove.

Configure the JMS Acknowledge tool

The JMS Acknowledge tool requires that you have an upstream JMS Receive tool configured with Acknowledgment mode set to Client.

If acknowledgment fails for any message, acknowledgment will fail for all messages. Keep this in mind as you design your solution, as you may wish to orchestrate your dataflow using automations that break large volumes of message into smaller batches

To configure the JMS Acknowledge tool:

Select the JMS Acknowledge tool, and then go to the Configuration tab on the Properties pane.
Select Consumer and select an upstream JMS Receive tool that will provide messages to be acknowledged.
Select Message ID field and select the field containing the JMSMessageID of the message to remove.
Optionally, connect a downstream tool to the "T" Trigger output, which will emit a record when it is finished removing messages.
Optionally, select the Execution tab, and then set Report options and Web service options.