Minimize temp space requirements

Overview

Data Management can use a lot of temp space, especially if projects are not configured optimally. You can reduce Data Management's temp space usage by following these guidelines.

Don't use Left-to-Right merge

Using the Left-to-Right merge type in the Merge tool (formerly the Concatenate tool) forces Data Management to save records to disk that are not destined for the leftmost input. This uses a lot of extra temp space and reduces performance. Choosing a merge type other than left-to-right will greatly reduce temp space usage. See About the Merge tool for details.

Avoid sorting

Sorting uses a lot of temp space, and you should avoid it when possible. The following tools always produce sorted outputs. If you are connecting a tool downstream from one of these tools, and the sort keys are the same (or a subset), you may be able to eliminate a sort step.

• Join

• Unique

• Sort

• Rank

• Summarize

• Pair-to-Group

 Keep sorted data sorted

A common Data Management technique is to split data (using Filter or a similar tool), perform operations on the split data, and then merge it back together. If you choose a Left-to-Right or Greedy merge type, you may be losing an opportunity to preserve the record order and avoid downstream sorts. You can avoid this by configuring the Merge tool to bring the streams back together in order. For details, see Avoiding split-merge bottlenecks.

Configure Join tools correctly

If you are joining large record sets, and one of them is unique on the join keys but the other is not, connect the non-unique input to L and the unique input to R. This will run faster and reduce the temp space required by the Join tool.