Skip to main content
Skip table of contents

Data Management overview

Overview

Data Management™ is a visual programming tool that makes it easy to create and run high-performance data-transformation projects. You select components from a palette of tools that perform different operations on your data. By configuring and connecting these tools, you create a custom data processing engine. Running your project is as simple as pushing a button.

The projects that you build using Data Management resemble a flowchart: records flow from the inputs, through tools and macros, and into the outputs. Each tool performs a simple function, such as sorting, filtering, or joining. By connecting the simple tools, you can create complex processes tailored to your specific needs.

Data Management is optimized to process an entire data set, rather than transactions or queries. As a result, it is often hundreds of times faster than transaction-oriented databases, even those running on powerful hardware.

Data Management is fast.

Speed matters. Data Management combines a short learning curve, raw processing power, and built-in productivity projects to give you access to the right data, right now!

Data Management is flexible.

Data Management's seamlessly integrated tools make it simple to tailor processes to fit your unique data profile.

Data Management is powerful.

Data Management easily tackles the toughest data integration and data quality problems. Built-in data analysis, task automation, and data re-engineering simplify data challenges by orders of magnitude.

Data Management is intuitive.

Data Management's visual development environment, low learning curve, and suite of "best practice" templates let you deliver powerful custom solutions in record time…effortlessly.

How it works

Data Management has two operational modes: data-flow projects and automations.

Data-flow projects

Data Management's data-flow projects visually connect data sources, transformations, and data outputs (collectively referred to as tools) using an in-memory streaming technology. For optimal performance, tools run in parallel and records are not staged to disk unless necessary. Data-flow projects are high performance—hundreds or even thousands of tools can be connected in a single project. Use a data-flow project to read data from sources, transform/integrate/analyze it, then write the results to outputs.

The Data Management data-flow framework includes comprehensive data re-engineering, parsing, pattern matching, address scrubbing, geocoding, matching, and householding capabilities. Data Management solutions are user-friendly, enabling both business and technical users to work independently or collaboratively.

Data Management reduces implementation complexity and time requirements with templates and macros that let you use best practices as a starting point. Data Management's flexibility makes it simple to quickly and efficiently tailor processes to fit your unique data profile. And Data Management incorporates robust data management capabilities—all accessed via an intuitive graphical development environment.

Data-flow projects can handle any number of data inputs, no matter how large. You can work with any data, with a practically unlimited number of fields, applying any business rules you like. Data-flow projects can be used as HTTP/XML/SOAP-based web services with the press of a button.

Automations

Notifications via SMS have been deprecated in version 9.4.5 and will not be supported in future Data Management releases.

Automations control stepwise execution, integrating the data-flow projects described above with other functions such as executing external programs, waiting for user review, transferring files via FTP, and determining the format of a file. Automations use success/failure logic for error handling and can send notifications to operators via email or SMS. Automations can loop over files and sets of values, and they have broad support for variables and parameters. Finally, they automatically support checkpoint and restart, so that if any steps fail in a multi-step procedure, only the failed steps need be restarted.

Use automations to solve problems like these:

  • Loop over all files in a folder, and run the same set of steps on each file.

  • Break a very long transformation process into smaller steps, so that a single failure does not require a complete restart.

  • Let the user enter parameters to control execution at start up such as file names, filter options, or report options.

  • Suspend execution midway and let the user review and edit data directly, for data stewardship approval, special-case corrections, and fuzzy-matching review.

  • Wait for files to appear in an upload directory, and automatically process them when they do.

  • Validate a file against many possible formats, and when a format matches, take appropriate action.

  • Remain "live" continuously, waiting for external factors to trigger execution (such as the appearance of a file or table, a calendar event, or an FTP data transfer).

What's new?

For a detailed list of what’s new and minor enhancements, refer to the Release notes.

Version 9.5

This release of Data Management includes the following improvements:

  • Added support for CentOS 9.

  • Added support for Windows Server 2022.

  • Added support for Windows 11.

  • Added support for AWS Redshift JDBC 2.1 driver.

  • Added support for MongoDB 6.x for MongoDB tools.

  • Added support for MongoDB 6.x Atlas.

  • Allowed the ML Predictor tool to be used in a published web-service.

  • Updated Data Management services on Linux to be managed as systemd units.

  • Implemented less explicit error messages for logon failures.

Version 9.4

This release of Data Management includes the following improvements:

  • Improved performance of SQL Server JDBC Bulk Loader.

  • New Excel Input and Excel Output tools.

  • Support for 2-way (mutual) SSL authentication in published web services and Web Service Call tools.

  • Support for JDBC FastLoad for Teradata.

  • Support for Amazon Aurora PostgreSQL.

  • Support for OAuth2 Service Principle authentication to Microsoft Azure Data Lake Storage Gen2 and Azure Blob Storage.

  • Support for Teradata 17.10.

  • Option to set output folder for Diagnostics Capture.

  • Kafka Input and Kafka Output tools can be used in published web services.

Version 9.3

This release of Data Management included the following improvements:

Version 9.2

This release of Data Management included the following improvements:

  • Enhancements to Avro Input and Avro Output tools:

  • Support for Avro schemas

  • Support for field input/output

  • Support for Avro over Kafka (additional Kafka serializers/deserializers).

  • New Hive Input2, Hive Output2, and Hive Execute2 tools for Hadoop clusters based on Hive version 3.0 or higher.

Getting help

We want to make your experience with Data Management as pleasant and productive as possible. Please email or call us when you need help. See Troubleshooting for a list of information we'll need in order to help you.

Contact us at:

Redpoint Global Inc.
34 Washington Street, Suite 205
Wellesley Hills, MA 02481 USA

Phone:

+1 781 725 0250

Fax:

781-694-1474

Email:

support@redpointglobal.com

Troubleshooting

When you contact us about a problem, please include the following information:

  • Product serial number and version number (view this by choosing About from the Help menu).

  • Your name, company name, telephone number, and email address.

  • Operating system and version number.

  • Exact wording of any error messages that appear.

  • Exactly what you were doing when the problem or error occurred.

  • How you tried to resolve the problem.

We may also ask you for:

  • A sample of the input data.

  • Your system configuration.

Using online Help

Data Management's online Help offers several ways to find information you may need to perform a particular task.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.

Contact Support Close