Build automation: Projects as steps

Overview

A Project step is simply a Data Management repository project or macro embedded within an automation project. When the automation runs an embedded project, it can set variables, change the project's configuration, and define its inputs and outputs.

Configuring a Project step:

1. Go to the repository.

2. Browse to the folder containing the project or macro you want to add.

3. Select the desired project or macro and drag it onto the automation project window.

4. Select the project, and then select the Configuration tab on the Properties pane

5. Optionally, select Override 'run on' setting to indicate where the project step should run:

• Automation machine: run on the same computer that is running the automation itself.

• Cluster load balance: run on any machine in a defined Data Management cluster.

6. Optionally, define parameters:

• Expression defines the user-settable parameters as variables. The expressions can use any of the operations or functions defined in Data Management, just as you would use them in the Calculate or Filter tools.

• Store into defines the part of the project configuration to change. Select the three dot action menu next to this column to view the configuration locator and select which portion of which tool's configuration to change.

7. Optionally, select the Variables tab and map project variables.

8. Optionally, select the Execution tab and set execution options:

• Select Skip to bypass this step at run time.

• Select Ignore error on this step to suppress step failure and continue processing in case of error.

Mapping project variables

Data Management projects embedded as automation steps can map automation variables into and out of their global variables (including parameters). In variable mappings copy the automation variable to the specified project variable before execution. Out variable mappings copy the project's global variable to the automation variable after project execution completes. Both In and Out variable mapping schemas let you specify an expression, which can be used to compute the value assigned to the variable.

Use project-variable mapping to set the parameters of a project before it is executed. For example, if you have defined a project parameter called DEPARTMENT, and use that parameter to alter the project's execution (using processing steps, or ${} syntax internal to the project), you would map a value into the project's DEPARTMENT In Variable. Similarly, if you use a project to calculate a value called FILE_QUALITY—perhaps the percentage of records that pass the data-quality rules of a project—you would map the project's FILE_QUALITY Out Variable to an automation variable, where it can be used to make execution choices.

Note that for simple tasks like changing the input file of a project, it is easier to use project configuration.

Project reconfiguration

Often you want to use the values of automation variables to alter a project's behavior. While you can do this by mapping variables into and out of projects, reconfiguring a project offers an easier option.

For example, suppose you have a ForEach group looping over files, and you want to execute the same project on each file:

To link the input file for this project to the FILENAME variable produced by the ForEach group:

1. Select the project, and then go to the Configuration tab on the Properties pane.

2. Enter an expression (FILENAME in this case) in the Expression column.

3. Select the three dot action menu in the Store into column and view the configuration locator.

This lets you choose common configuration settings to replace in the project (Tool(DbfInput1)/data_file in this example). Select Advanced mode to see additional settings.

4. Select OK, and then Commit .

Do not change the name of a DLD Input or DLD Output file when it is used as a staging connector. Data Management sets this file name to a temporary file which it manages.

Debugging embedded projects

If you have a project that runs as expected when executed independently, but has problems when run as part of an automation, you can use the View context-customized project command to debug the project.

Before an automation runs a project, it can make changes to the project's configuration, inputs and outputs, and variables. View context-customized project shows the "internal" view of this embedded project, including staging connection tools and data connections, variable mapping, and dynamic configuration. This helps you identify problems in embedded projects that are only evident at run-time. Once you identify a problem, use the Edit project command to make changes in the project, and the Reload project command to ensure that your edits are reflected in the automation.