Frequently asked questions

I changed my tool, but the change isn't visible in the Data Management client. Why?

Try the following:

Close and re-open any open project that uses your tool.
After rebuilding your JAR, be sure that you've copied it into java_plugins in your Data Management install directory (unless you're explicitly adding it to the classpath via Settings>Site>Java>Add JAR to classpath).
Select the Palette menu, and then choose Reset Palette.

Adding your SDK tool JAR to the Java classpath under Settings>Site>Java is no longer supported as of RPDM version 8.2.

If your changes still do not appear, try restarting the Site and Execution services:

Open the Windows Control Panel and navigate to the Administrative Tools section.
Double-click Services.
Find the entry for RedPointDM Version 8 Execution Service Site Service. Right-click on the entry, and choose Stop on the context menu.
Find the entry for RedPointDM Version 8 Execution Service. Right-click on the entry, and choose Restart on the context menu.
Return to the entry for RedPointDM Version 8 Execution Service Site Service. Right-click on the entry, and choose Start on the context menu.

How do I log messages to the Message Viewer in Data Management?

You can do this via the traceMessage method in net.redpoint.dataflow.transform.ToolContext. Your tool will have a reference to ToolContext via the context instance variable. Use it like the following example.

CODE

context.traceMessage("See you in the Message Viewer!");

How do I debug my tool?

You can do this using your Java IDE's debugger.

How can I make my tool configurable by users?

Tools in Data Management are configured via the Property panel. The SDK provides a facility for exposing your tool to the Property panel for configuration. See Configure your tool's user interface.

Can I set a custom icon for my tool?

Yes, the SDK loads custom tool icons from the classpath based on naming convention. See Defining a custom icon in Configuring your tool's user interface.

Where can I see some examples of tools built using the Data Management SDK?

The SDK includes a samples project. It includes examples of each of the tool types supported by the SDK.

When should I build a tool using the SDK instead of creating a macro?

When a macro isn’t flexible enough or doesn’t perform well, consider building an SDK tool. Some common cases include:

Integrating Java libraries and APIs.
Performing iterative or looping processing steps.
Building algorithms that cannot be easily expressed with Data Management’s palette of tools. For example, graph algorithms are very difficult to build directly in Data Management.
Processing data formats that are unknown to Data Management.

However, some tasks are very easy to do in macros, but hard to implement in the SDK:

Sorts
Joins
Summarization
Accessing databases

Why am I getting a ClassNotFoundException when I run my tool?

If your tool depends on third-party libraries, you'll need to ensure that those libraries are accessible to your tool when it is installed in Data Management. Bundle the dependencies into your SDK tool JAR using Maven's shade plug-in. The POM distributed with the SDK samples project includes a stanza that can be used as a starting point for working with maven-shade-plugin.

Data Management 8.2.1 introduced a new classloading model. In prior versions, you must take special care to avoid classpath conflicts.

Why does my tool work fine the first time I run it, but acts weird on subsequent runs?

This often indicates a tool lifecycle issue. Be sure that you're performing all necessary setup and tear-down in the prepare and cleanup methods.

When implementing the cleanup method, follow these guidelines:

Close anything that holds a resource (files, databases, etc.)
Set things that use a lot of memory to null
Abort and join any background threads

When implementing the prepare method, ensure that the tool is in the proper state before it runs—whether it's running for the first time, or on subsequent executions. This includes setting any member variables to their initial default state.

Keep in mind that under certain conditions, your tool's cleanup may be called without a corresponding calls to the prepare or "do" methods.

Why can't I modify a Schema in my tools?

Many Records can share the same Schema, and if Data Management allowed Schemas to be changed, this would result in an incompatibility. Because of this, the Schema instances in your tools are immutable. Consider the following rules when working with Schema instances in your tools:

Given a List of FieldSpec instances, you can construct a Schema.
Given a Schema, you can construct a Record.
Given a Schema, you can get its List of FieldSpec instances, but you cannot change it.
When implementing a Transform Tool, you will often want your result Schema to be a slight modification of the input Schema. To accomplish this, call getFieldSpecs() on the input Schema, derive an augmented List<FieldSpec> from the input list, and create a new Schema.

Why can't I make my own Records in my InputTool or TransformTool?

In order to simplify the interface to these tool types and to prevent errors like attempting to return the same Record twice, Data Management creates the result records for you. These records always match the Schema that you provided in your tool's configure method. Consider the following rules when working with Record instances in your tools:

Given a Record, you can get/set its field values, but you cannot change the underlying Schema.
For InputTool and TransformTool, you will be handed the result record to populate.
For GeneralTool, you need to create your own result records. Make sure to create a new, unique instance for each result record, and that you use the correct Schema to create the record.

How do I install SDK dependencies in my local repository?

If you've got a standalone Maven installation, run these commands from %SDK_HOME%.

CODE

mvn install:install-file -Dfile=lib\records-1.0.jar -DgroupId=redpoint -DartifactId=records -Dversion=1.0 -Dpackaging=jar
mvn install:install-file -Dfile=lib\dataitems-1.0.jar -DgroupId=redpoint -DartifactId=dataitems -Dversion=1.0 -Dpackaging=jar

This will install the records and dataitems JARs in your local repository, which are both required by the Data Management SDK.

If you're using Maven bundled with Eclipse, you can do this instead:

From the menu bar, select Run>Run Configurations.
Select Maven Build and choose New Run Configuration.
Name your run configuration Install Redpoint Records.
For Base Directory, select Browse Workspace and choose your SDK project.
For Goals, enter install:install-file.
Under Parameters, select Add and add the following name/value pairs:
- Name
- Value
- file${project_loc}\lib\records-1.0.jargroupIdredpointartifactIdrecordsversion1.0packagingjar
Select Apply, and then Close.
Make sure your project is selected in the Project Explorer, and then select Install Redpoint Records from the Run As button on the tool ribbon.

You should see a line like the following in the messages printed to the Eclipse console.

CODE

[INFO] Installing \PATH\TO\YOUR\PROJECT\lib\records-1.0.jar to C:\Users\YOURUSERNAME\.m2\repository\redpoint\records\1.0\records-1.0.jar

Next, load the dataitems dependency:

From the menu bar, select Run>Run Configurations.
Select Maven Build, highlight the Install Redpoint Records you created above, and choose Duplicate Run Configuration.
Change Name to Install RPDM Dataitems.
Under Parameters, change file to ${project_loc}\lib\records-1.0.jar and artifactId to dataitems.
Select Apply, and then Close.
Make sure your project is selected in the Project Explorer, and then select Install Redpoint Records from the Run As button on the tool ribbon.

This time, you should see the following in the messages printed to the Eclipse console.

CODE

[INFO] Installing \PATH\TO\YOUR\PROJECT\lib\dataitems-1.0.jar to C:\Users\YOURUSERNAME\.m2\repository\redpoint\dataitems\1.0\dataitems-1.0.jar

Now you can edit pom.xml, removing the scope and systemPath elements from the dataitems and records dependency elements so the dependencies stanza looks like the following.

CODE

<dependency>
  <groupId>redpoint</groupId>
  <artifactId>dataitems</artifactId>
  <version>1.0</version>
</dependency>
<dependency>
  <groupId>redpoint</groupId>
  <artifactId>records</artifactId>
  <version>1.0</version>
</dependency>

This will allow Maven to resolve these dependencies using the local repository, instead of going directly to the filesystem.

What is the Document type?

Document represents a BSON document. BSON is very similar to JSON, but is a binary format, and supports more data types than JSON. BSON is the fundamental data type used by MongoDB.

SDK tools can create new Documents, manipulate existing Documents, and extract scalar values from Documents. See the BSON spec and the Document javadoc for more information.