Skip to main content
Skip table of contents

External Transform

Overview

The External Transform group includes tools for integrating external programs into the Data Management data stream using flat or CSV format data.

CSV Streaming Transform

With the CSV Streaming Transform tool, you can:

  • Call an executable that generates CSV-format data and use its output as input to a Data Management process.

  • Send CSV-format data from a Data Management process to a specified executable for use as input.

  • Send data from a Data Management process to an executable for processing and then "catch" the output, returning it to the same Data Management project.

CSV Streaming Transform tool configuration parameters

The CSV Streaming Transform tool has three sets of configuration parameters in addition to the standard execution options.

Input

Parameter

Description

Write records to

Defines data source for the executable. This is optional and defaults to (none).

Quote condition

The conditions under which the quote character is applied. Must be one of Automatic (as needed), Always (text only), Always (everything), or Never. Defaults to Automatic.

Quote missing fields

If selected, empty fields have quotes applied according to the specified Quote condition.

Write a field name header

If selected, field names will be output as the first line of the CSV file.

End of line

The character sequence identifying the line termination. One of Cr Lf (Windows, OS/2), Lf (UNIX), or Cr (Mac). This defaults to Cr Lf.

Output

Parameter

Definition

Read records from

Defines data target for the executable. This is optional and defaults to (none).

Sample file

If selected, configure the field layout using the specified sample file instead of configuring fields directly.

Limit records

If selected, limits the number of records read.

Use repository schema

If selected, configure the field layout using the specified Schema instead of configuring fields directly.

Schema

If Use repository schema is selected, a schema must be specified.

Separator

The field delimiter. Can be specified by Character or Code. This is optional and defaults to a comma.

Quote character

If defined, one or more characters that quote or escape the data. If no quote character is specified, there is no escapement. If one of {[ or < are specified, the corresponding closing bracket is treated as the closing quote.

Allow newlines in fields

If selected, files with embedded newlines in the fields are accepted as input and any newlines encountered are ignored.

Skip lines

If defined, skip leading lines in the file, such as field names contained in a first line. This is optional and defaults to 0.

Trim

If selected, leading/trailing whitespace is removed from input data.

Process

Parameter

Definition

Executable

Path and name of the desired executable.

Arguments

Arguments or parameters to be sent to the executable. To specify files or pipes as input or output, use the markers %IN% and %OUT%. This is optional.

Standard input
Standard output
Standard error

Define how to handle standard buffered input/output and error streams. If you specified a file or pipe on the Input or Output tabs, you can optionally specify Process I/O settings.

Stdin file name
Stdout file name
Sterr file name

If you selected From file or To file as a Standard input/output/error option, you must specify a file name.

Ignore exit code

If selected, ignores the exit code of the spawned process. By default, non-zero exit codes are interpreted as errors.

Working directory

If specified, the current working directory of the spawned process.

Error handling

Configures how errors in the input data will be handled:

  • Ignore error: accept the record and ignore the error.

  • Issue warning: accept the record but send a warning to the Message Viewer and/or Error Report connector.

  • Reject record: reject the record and send a warning to the Message Viewer and/or Error Report connector.

  • Abort project: abort the project.

Parsing errors

Specify how to handle input containing invalid XML that cannot be processed.

The Issue warning option is not available for this type of error.

Conversion errors

Specify how to handle input data that cannot be correctly converted to its specified data type (such as converting "Jane Smith" to a Date).

Truncation errors

Specify how to handle input with truncated field values (such as storing "John Smithson" in a Unicode(1) field).

Missing fields

Specify how to handle input with fields that are missing or contain empty values.

Extra fields

Specify how to handle input with extra elements that are not defined in the tool's configuration.

Send errors to Message Viewer

If selected, all error types configured to either Issue warning or Reject record will send warnings to the Message Viewer.

The Report warnings option must be selected on the Execution tab.

If selected, all error types configured to either Issue warning or Reject record will send warnings to the Message Viewer, even if the Report warnings option on the Execution tab is not selected.

Send errors to Error Report connector

If selected, creates an E "Error Report" output connector. All error types configured to either Issue warning or Reject record will send detailed error messages to that connector. Attach a downstream Data Viewer or other output tool to review these messages.

Configure the CSV Streaming Transform tool

To configure the CSV Streaming Transform tool:

  1. Select the CSV Streaming Transform tool.

  2. Optionally, select the Input tab and specify how to handle input to the executable.

Write records to...

Description

Standard I/O

Writes data to the executable's standard buffered input/output (stdio). When this option is selected, you cannot define processing options using the Standard input and Standard output controls on the Process tab (below).

Named file

Creates a file with the Name you specify for the data input to/output from the executable.

Temp file

Creates a temporary file for the data input to/output from the executable.

Named pipe

Creates pipe with the Name you specify for the data input to/output by the executable.

System named pipe

Creates a pipe with a system-specified name for the data input to/output by the executable.

Format

Description

Quote condition

Specify how fields are enclosed in quote characters:

  • Automatic (as needed)

  • Always (text only)

  • Always (everything)

  • Never

Quote empty fields

Select Quote empty fields to enclose blank fields in the quote characters.

Write a field name header

Select Write a field name header to write field names to the first line of the flat file.

End of line

End of line sequence to match the operating system that will read the resulting file:

  • CR-LF (carriage return followed by line-feed, for DOS, Windows, and OS/2)

  • LF (line-feed, for UNIX)

  • CR (carriage return, for Mac)

  1. Optionally, select the Output tab on the Properties pane and specify how to handle input to the executable. Choices are the same as those on the Input tab (step 2). Follow the procedure for configuring a CSV input. You can specify a sample input file to make configuring the expected output fields easier.

  2. Select the Process tab and specify the desired Executable file. If the requested executable cannot be launched and is on on a Linux server with Ignore exit code set, the failure to launch will not be shown or logged.

  3. Optionally, specify Arguments or parameters to be sent to the executable. If you will specify files or pipes as input, you should include the marker %IN% to denote the Input location. Similarly, include the marker %OUT% to denote the Output location. These markers will be replaced by the designated names at runtime.

  4. Optionally, specify a Working directory for the spawned child process. This will not affect interpretation of any paths specified for input or output.

  5. If you specified a file or pipe on the Input or Output tabs, you can optionally specify Process I/O settings. These define how to handle standard buffered input/output and error streams:

    • Standard input directs the executable's standard buffered input (stdio) to the file you specify.

    • Standard output directs the executable's standard buffered output (stdio) as specified:

      • Log as info reports messages as Data Management job information.

      • Log as warnings reports messages as Data Management warnings.

      • Log as errors reports messages as Data Management warnings.

      • To file writes messages to the file you specify.

  6. Optionally, select Ignore exit code. Normally, Data Management tests the exit code of the spawned process. If the exit code is non-zero, it is considered an error. However, some programs (notably robocopy) use non-zero exit codes to mean different kinds of success. If you are specifying such an executable, enable the Ignore exit code option.

  7. Configure error handling.

  8. Optionally, go to the Execution tab and Enable trigger input and output, configure reporting options, or set Web service options.

Flat File Streaming Transform

With the Flat File Streaming Transform tool, you can:

  • Call an executable that generates flat file format data and use its output as input to a Data Management process.

  • Send flat file format data from a Data Management process to a specified executable for use as input.

  • Send data from a Data Management process to an executable for processing and then "catch" the output, returning it to the same Data Management project.

Flat File Streaming Transform tool configuration parameters

The Flat File Streaming Transform tool has three sets of configuration parameters in addition to the standard execution options.

Input

Parameter

Definition

Write records to

Defines data source for the executable. This is optional and defaults to (none).

End of line

The method for determining the end of the record. This is optional and defaults to Cr Lf (Windows).

Output

Parameter

Definition

Read records from

Defines data target for the executable. This is optional and defaults to (none).

Sample file

If selected, configure the field layout using the specified sample file instead of configuring fields directly.

Limit records

If selected, limits the number of records read.

Use repository schema

If selected, configure the field layout using the specified Schema instead of configuring fields directly.

Schema

If Use repository schema is selected, a schema must be specified.

Line break

The method for determining the end of the record. This is optional and defaults to Newlines only.

Record size

If specified, sets the length of the records. The columns in the data sample will visually line up when the record size is correctly specified. Does not apply if Line break is set to Newlines only.

Data offset

The starting position of the record data, if other than 0.

Field

Name that uniquely names the field amongst all fields in the record.

Type

Sub-configuration describing the type of the new field. See Configuring Data Types.

Process

Parameter

Definition

Process

Path and name of the desired executable.

Arguments

Arguments or parameters to be sent to the executable. To specify files or pipes as input or output, use the markers %IN% and %OUT%. This is optional.

Working directory

If specified, the current working directory of the spawned process.

Standard input
Standard output
Standard error

Define how to handle standard buffered input/output and error streams. If you specified a file or pipe on the Input or Output tabs, you can optionally specify Process I/O settings.

Name

If you selected From file or To file as a Standard input/output/error option, you must specify a file name.

Ignore exit code

If selected, ignores the exit code of the spawned process. By default, non-zero exit codes are interpreted as errors.

Error handling

Configures how errors in the input data will be handled:

  • Ignore error: accept the record and ignore the error.

  • Issue warning: accept the record but send a warning to the Message Viewer and/or Error Report connector.

  • Reject record: reject the record and send a warning to the Message Viewer and/or Error Report connector.

  • Abort project: abort the project.

Parsing errors

Specify how to handle input containing invalid XML that cannot be processed.

The Issue warning option is not available for this type of error.

Conversion errors

Specify how to handle input data that cannot be correctly converted to its specified data type (such as converting "Jane Smith" to a Date).

Truncation errors

Specify how to handle input with truncated field values (such as storing "John Smithson" in a Unicode(1) field).

Missing fields

Specify how to handle input with fields that are missing or contain empty values.

Extra fields

Specify how to handle input with extra elements that are not defined in the tool's configuration.

Send errors to Message Viewer

If selected, all error types configured to either Issue warning or Reject record will send warnings to the Message Viewer.

The Report warnings option must be selected on the Execution tab.

If selected, all error types configured to either Issue warning or Reject record will send warnings to the Message Viewer, even if the Report warnings option on the Execution tab is not selected.

Send errors to Error Report connector

If selected, creates an E "Error Report" output connector. All error types configured to either Issue warning or Reject record will send detailed error messages to that connector. Attach a downstream Data Viewer or other output tool to review these messages.

Configure the Flat File Streaming Transform tool

To configure the Flat File Streaming Transform tool:

  1. Select the Flat File Streaming Transform tool.

  2. Optionally, select the Input tab and specify how records are written to the executable.

Write records to...

Description

None

The executable does not read data from the tool's input connector.

Standard I/O

Writes data to the executable's standard buffered input/output (stdio). When this option is selected, you cannot define processing options using the Standard input and Standard output controls on the Process tab (below).

Named file

Creates a file with the Name you specify for the data input to/output from the executable.

Temp file

Creates a temporary file for the data input to/output from the executable.

Named pipe

Creates pipe with the Name you specify for the data input to/output by the executable.

System named pipe

Creates a pipe with a system-specified name for the data input to/output by the executable.

Format

Description

End of line

End of line sequence to match the operating system that will read the resulting file:

  • None (no line ending)

  • CR-LF (carriage return followed by line-feed, for DOS, Windows, and OS/2)

  • LF (line-feed, for UNIX)

  • CR (carriage return, for Mac)

  1. Optionally, select the Output tab and specify how records are read from the executable. Choices are the same as those on the Input tab (step 2), then follow the procedure for configuring a Flat File input. You can specify a sample input file to make configuring the expected output fields easier.

  2. Select the Process tab and specify the desired Process file. If the requested executable cannot be launched and is on on a Linux server with Ignore exit code set, the failure to launch will not be shown or logged.

  3. Optionally, specify Arguments or parameters to be sent to the executable. If you will specify files or pipes as input, include the marker %IN% to denote the Input location. Similarly, include the marker %OUT% to denote the Output location. These markers will be replaced by the designated names at runtime.

  4. Optionally, specify a Working directory for the spawned child process. This will not affect interpretation of any paths specified for input or output.

  5. If you specified a file or pipe on the Input or Output tabs, you can optionally specify Process I/O settings. These define how to handle standard buffered input/output and error streams:

    • Standard input directs the executable's standard buffered input (stdio) to the file you specify as Name.

    • Standard output directs the executable's standard buffered output (stdio) as specified:

      • Log as info reports messages as Data Management job information.

      • Log as warnings reports messages as Data Management warnings.

      • Log as errors reports messages as Data Management warnings.

      • To file writes messages to the file you specify.

    • Standard error is the data stream where external executables typically report error messages. This option controls how such messages are handled, with choices as listed above.

  6. Optionally, select Ignore exit code. Normally, Data Management tests the exit code of the spawned process. If the exit code is non-zero, it is considered an error. However, some programs (most notably robocopy) use non-zero exit codes to mean different kinds of success. If you are specifying such an executable, enable the Ignore exit code option.

  7. Configure error handling.

  8. Optionally, go to the Execution tab and Enable trigger input and output, configure reporting options, or set Web service options.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.