Use performance logs

Data Management's log viewer can provide valuable guidance for performance tuning. It will tell you which macros and tools are using the most CPU, disk, and memory, so that you can pinpoint performance bottlenecks.

• Don't configure your project to use more threads than licensed CPU cores, or your performance numbers will be inaccurate.

• Always collect performance information on a computer that is idle except for the project being measured.

• Add more cores! Data Management scales very well on multiple CPU cores, and many performance-enhancing techniques rely on parallelization.

Example:

This example assumes we've run the following project:

1. View the Management Dashboard and select the Logs icon. Double-click the Projects entry at the bottom of the Logs list, locate the project run near the top of the Logs list, and select it. Notice the Messages button above the lower panel:

2. Click Messages and select CPU Performance to display each tool in the project, along with its record counts and CPU usage:

Note that tools embedded within macros are also displayed. Clicking the Time column sorts the display so that the tools and macros using the most time are at the top of the list.

If you find a tool that is consuming most of the time, consider these approaches:

• Try explicit parallelization of very slow tools using Split/Merge.

• Split a Calculate tool with a long list of calculations into two sequentially-connected Calculate tools, each with half of the calculations (this is known as "pipeline parallelism").

• If Sort/Join/Summarize tools are taking a lot of time, configure sort parallelism. Note that even with sort parallelism configured, these tools may still exhibit high CPU times even when they run faster, because this number aggregates all of the threads' times.

• Reduce the width of records flowing through all of the tools by eliminating fields not needed for calculations. Extra columns slow down everything.