Sort parallelism

Overview

Many tools in Data Management sort records to do their work, and sorting is often a bottleneck in processing of large data sets. To improve performance, Data Management has an option to inject "native" parallelism into certain tools that do a lot of sorting and other time-intensive tasks. These tools are:

• Join

• Unique

• Sort

• Rank

• Summarize

• Pair-to-Group

In addition, most of the AO Matching macros use these tools internally.

You may be able to improve the performance of these tools and macros by configuring the sort threads setting at the Site, Machine, and Project level.

When to enable sort parallelism

This additional parallelism only improves the processing of large data sets, and only on higher-performance computers. We recommend that you enable sort parallelism only if the following conditions are met:

1. At least two CPU cores are licensed.

2. At least two temp spaces are configured, or your temp space is a high-performance storage system.

3. You are processing at least 10 million records at a time.

Enable sort parallelism

Sort parallelism is an additional Tuning option in the Settings hierarchy:

Setting Sort parallelism to 1 causes non-parallel behavior. Setting it to 2 enables parallel sorting operations.

Generally, you should set Sort parallelism equal to the number of temp spaces, but no larger than 2. We recommend that you benchmark the effect of sort parallelism on your projects and data. The benefit will depend on several factors:

• The number of temp spaces defined, and the performance of your storage system.

• The number of licensed CPU cores.

• The nature of the projects you are running.

• The number of simultaneous projects being run, and computer load. If your computer gets heavy use from multiple users, sort parallelism may actually decrease performance due to additional resource contention.