External Transform

Overview

The External Transform group includes tools for integrating external programs into the Data Management data stream using flat or CSV format data.

CSV Streaming Transform

With the CSV Streaming Transform tool, you can:

Call an executable that generates CSV-format data and use its output as input to a Data Management process.
Send CSV-format data from a Data Management process to a specified executable for use as input.
Send data from a Data Management process to an executable for processing and then "catch" the output, returning it to the same Data Management project.

CSV Streaming Transform tool configuration parameters

The CSV Streaming Transform tool has three sets of configuration parameters in addition to the standard execution options.

Input

Parameter	Description
Write records to	Defines data source for the executable. This is optional and defaults to (none).
Quote condition	The conditions under which the quote character is applied. Must be one of Automatic (as needed), Always (text only), Always (everything), or Never. Defaults to Automatic.
Quote missing fields	If selected, empty fields have quotes applied according to the specified Quote condition.
Write a field name header	If selected, field names will be output as the first line of the CSV file.
End of line	The character sequence identifying the line termination. One of `Cr Lf` (Windows, OS/2), `Lf` (UNIX), or `Cr` (Mac). This defaults to `Cr Lf`.

Output

Parameter	Definition
Read records from	Defines data target for the executable. This is optional and defaults to (none).
Sample file	If selected, configure the field layout using the specified sample file instead of configuring fields directly.
Limit records	If selected, limits the number of records read.
Use repository schema	If selected, configure the field layout using the specified Schema instead of configuring fields directly.
Schema	If Use repository schema is selected, a schema must be specified.
Separator	The field delimiter. Can be specified by Character or Code. This is optional and defaults to a comma.
Quote character	If defined, one or more characters that quote or escape the data. If no quote character is specified, there is no escapement. If one of `{[` or `<` are specified, the corresponding closing bracket is treated as the closing quote.
Allow newlines in fields	If selected, files with embedded newlines in the fields are accepted as input and any newlines encountered are ignored.
Skip lines	If defined, skip leading lines in the file, such as field names contained in a first line. This is optional and defaults to 0.
Trim	If selected, leading/trailing whitespace is removed from input data.

Process

Parameter	Definition
Executable	Path and name of the desired executable.
Arguments	Arguments or parameters to be sent to the executable. To specify files or pipes as input or output, use the markers `%IN%` and `%OUT%`. This is optional.
Standard input Standard output Standard error	Define how to handle standard buffered input/output and error streams. If you specified a file or pipe on the Input or Output tabs, you can optionally specify Process I/O settings.
Stdin file name Stdout file name Sterr file name	If you selected From file or To file as a Standard input/output/error option, you must specify a file name.
Ignore exit code	If selected, ignores the exit code of the spawned process. By default, non-zero exit codes are interpreted as errors.
Working directory	If specified, the current working directory of the spawned process.
Error handling	Configures how errors in the input data will be handled: Ignore error: accept the record and ignore the error. Issue warning: accept the record but send a warning to the Message Viewer and/or Error Report connector. Reject record: reject the record and send a warning to the Message Viewer and/or Error Report connector. Abort project: abort the project.
Parsing errors	Specify how to handle input containing invalid XML that cannot be processed. The Issue warning option is not available for this type of error.
Conversion errors	Specify how to handle input data that cannot be correctly converted to its specified data type (such as converting "Jane Smith" to a Date).
Truncation errors	Specify how to handle input with truncated field values (such as storing "John Smithson" in a Unicode(1) field).
Missing fields	Specify how to handle input with fields that are missing or contain empty values.
Extra fields	Specify how to handle input with extra elements that are not defined in the tool's configuration.
Send errors to Message Viewer	If selected, all error types configured to either Issue warning or Reject record will send warnings to the Message Viewer. The Report warnings option must be selected on the Execution tab. If selected, all error types configured to either Issue warning or Reject record will send warnings to the Message Viewer, even if the Report warnings option on the Execution tab is not selected.
Send errors to Error Report connector	If selected, creates an E "Error Report" output connector. All error types configured to either Issue warning or Reject record will send detailed error messages to that connector. Attach a downstream Data Viewer or other output tool to review these messages.

Configure the CSV Streaming Transform tool

To configure the CSV Streaming Transform tool:

Select the CSV Streaming Transform tool.
Optionally, select the Input tab and specify how to handle input to the executable.

Write records to...	Description
Standard I/O	Writes data to the executable's standard buffered input/output (stdio). When this option is selected, you cannot define processing options using the Standard input and Standard output controls on the Process tab (below).
Named file	Creates a file with the Name you specify for the data input to/output from the executable.
Temp file	Creates a temporary file for the data input to/output from the executable.
Named pipe	Creates pipe with the Name you specify for the data input to/output by the executable.
System named pipe	Creates a pipe with a system-specified name for the data input to/output by the executable.

Format	Description
Quote condition	Specify how fields are enclosed in quote characters: Automatic (as needed) Always (text only) Always (everything) Never
Quote empty fields	Select Quote empty fields to enclose blank fields in the quote characters.
Write a field name header	Select Write a field name header to write field names to the first line of the flat file.
End of line	End of line sequence to match the operating system that will read the resulting file: `CR-LF` (carriage return followed by line-feed, for DOS, Windows, and OS/2) `LF` (line-feed, for UNIX) `CR` (carriage return, for Mac)

Optionally, select the Output tab on the Properties pane and specify how to handle input to the executable. Choices are the same as those on the Input tab (step 2). Follow the procedure for configuring a CSV input. You can specify a sample input file to make configuring the expected output fields easier.
Select the Process tab and specify the desired Executable file. If the requested executable cannot be launched and is on on a Linux server with Ignore exit code set, the failure to launch will not be shown or logged.
Optionally, specify Arguments or parameters to be sent to the executable. If you will specify files or pipes as input, you should include the marker %IN% to denote the Input location. Similarly, include the marker %OUT% to denote the Output location. These markers will be replaced by the designated names at runtime.
Optionally, specify a Working directory for the spawned child process. This will not affect interpretation of any paths specified for input or output.
If you specified a file or pipe on the Input or Output tabs, you can optionally specify Process I/O settings. These define how to handle standard buffered input/output and error streams:
- Standard input directs the executable's standard buffered input (stdio) to the file you specify.
- Standard output directs the executable's standard buffered output (stdio) as specified:
  - Log as info reports messages as Data Management job information.
  - Log as warnings reports messages as Data Management warnings.
  - Log as errors reports messages as Data Management warnings.
  - To file writes messages to the file you specify.
Optionally, select Ignore exit code. Normally, Data Management tests the exit code of the spawned process. If the exit code is non-zero, it is considered an error. However, some programs (notably robocopy) use non-zero exit codes to mean different kinds of success. If you are specifying such an executable, enable the Ignore exit code option.
Configure error handling.
Optionally, go to the Execution tab and Enable trigger input and output, configure reporting options, or set Web service options.

Flat File Streaming Transform

With the Flat File Streaming Transform tool, you can:

Call an executable that generates flat file format data and use its output as input to a Data Management process.
Send flat file format data from a Data Management process to a specified executable for use as input.
Send data from a Data Management process to an executable for processing and then "catch" the output, returning it to the same Data Management project.

Flat File Streaming Transform tool configuration parameters

The Flat File Streaming Transform tool has three sets of configuration parameters in addition to the standard execution options.

Input

Parameter	Definition
Write records to	Defines data source for the executable. This is optional and defaults to (none).
End of line	The method for determining the end of the record. This is optional and defaults to Cr Lf (Windows).

Output

Parameter	Definition
Read records from	Defines data target for the executable. This is optional and defaults to (none).
Sample file	If selected, configure the field layout using the specified sample file instead of configuring fields directly.
Limit records	If selected, limits the number of records read.
Use repository schema	If selected, configure the field layout using the specified Schema instead of configuring fields directly.
Schema	If Use repository schema is selected, a schema must be specified.
Line break	The method for determining the end of the record. This is optional and defaults to Newlines only.
Record size	If specified, sets the length of the records. The columns in the data sample will visually line up when the record size is correctly specified. Does not apply if Line break is set to Newlines only.
Data offset	The starting position of the record data, if other than 0.
Field	Name that uniquely names the field amongst all fields in the record.
Type	Sub-configuration describing the type of the new field. See Configuring Data Types.

Process

Parameter	Definition
Process	Path and name of the desired executable.
Arguments	Arguments or parameters to be sent to the executable. To specify files or pipes as input or output, use the markers `%IN%` and `%OUT%`. This is optional.
Working directory	If specified, the current working directory of the spawned process.
Standard input Standard output Standard error	Define how to handle standard buffered input/output and error streams. If you specified a file or pipe on the Input or Output tabs, you can optionally specify Process I/O settings.
Name	If you selected From file or To file as a Standard input/output/error option, you must specify a file name.
Ignore exit code	If selected, ignores the exit code of the spawned process. By default, non-zero exit codes are interpreted as errors.
Error handling	Configures how errors in the input data will be handled: Ignore error: accept the record and ignore the error. Issue warning: accept the record but send a warning to the Message Viewer and/or Error Report connector. Reject record: reject the record and send a warning to the Message Viewer and/or Error Report connector. Abort project: abort the project.
Parsing errors	Specify how to handle input containing invalid XML that cannot be processed. The Issue warning option is not available for this type of error.
Conversion errors	Specify how to handle input data that cannot be correctly converted to its specified data type (such as converting "Jane Smith" to a Date).
Truncation errors	Specify how to handle input with truncated field values (such as storing "John Smithson" in a Unicode(1) field).
Missing fields	Specify how to handle input with fields that are missing or contain empty values.
Extra fields	Specify how to handle input with extra elements that are not defined in the tool's configuration.
Send errors to Message Viewer	If selected, all error types configured to either Issue warning or Reject record will send warnings to the Message Viewer. The Report warnings option must be selected on the Execution tab. If selected, all error types configured to either Issue warning or Reject record will send warnings to the Message Viewer, even if the Report warnings option on the Execution tab is not selected.
Send errors to Error Report connector	If selected, creates an E "Error Report" output connector. All error types configured to either Issue warning or Reject record will send detailed error messages to that connector. Attach a downstream Data Viewer or other output tool to review these messages.

Configure the Flat File Streaming Transform tool

To configure the Flat File Streaming Transform tool:

Select the Flat File Streaming Transform tool.
Optionally, select the Input tab and specify how records are written to the executable.

Write records to...	Description
None	The executable does not read data from the tool's input connector.
Standard I/O	Writes data to the executable's standard buffered input/output (stdio). When this option is selected, you cannot define processing options using the Standard input and Standard output controls on the Process tab (below).
Named file	Creates a file with the Name you specify for the data input to/output from the executable.
Temp file	Creates a temporary file for the data input to/output from the executable.
Named pipe	Creates pipe with the Name you specify for the data input to/output by the executable.
System named pipe	Creates a pipe with a system-specified name for the data input to/output by the executable.

Format

Description

End of line

End of line sequence to match the operating system that will read the resulting file:

None (no line ending)
CR-LF (carriage return followed by line-feed, for DOS, Windows, and OS/2)
LF (line-feed, for UNIX)
CR (carriage return, for Mac)

Optionally, select the Output tab and specify how records are read from the executable. Choices are the same as those on the Input tab (step 2), then follow the procedure for configuring a Flat File input. You can specify a sample input file to make configuring the expected output fields easier.
Select the Process tab and specify the desired Process file. If the requested executable cannot be launched and is on on a Linux server with Ignore exit code set, the failure to launch will not be shown or logged.
Optionally, specify Arguments or parameters to be sent to the executable. If you will specify files or pipes as input, include the marker %IN% to denote the Input location. Similarly, include the marker %OUT% to denote the Output location. These markers will be replaced by the designated names at runtime.
Optionally, specify a Working directory for the spawned child process. This will not affect interpretation of any paths specified for input or output.
If you specified a file or pipe on the Input or Output tabs, you can optionally specify Process I/O settings. These define how to handle standard buffered input/output and error streams:
- Standard input directs the executable's standard buffered input (stdio) to the file you specify as Name.
- Standard output directs the executable's standard buffered output (stdio) as specified:
  - Log as info reports messages as Data Management job information.
  - Log as warnings reports messages as Data Management warnings.
  - Log as errors reports messages as Data Management warnings.
  - To file writes messages to the file you specify.
- Standard error is the data stream where external executables typically report error messages. This option controls how such messages are handled, with choices as listed above.
Optionally, select Ignore exit code. Normally, Data Management tests the exit code of the spawned process. If the exit code is non-zero, it is considered an error. However, some programs (most notably robocopy) use non-zero exit codes to mean different kinds of success. If you are specifying such an executable, enable the Ignore exit code option.
Configure error handling.
Optionally, go to the Execution tab and Enable trigger input and output, configure reporting options, or set Web service options.