Release notes

DM-4052

When configuring a data connection, options are exposed to configure the encrypt, trustServerCertificate, and hostNameInCertificate connection string properties for all supported MSSQL derivatives (Azure SQL, Azure Synapse, SQL Server 2019, and SQL Server 2022)

DM-15754

Added support for Ubuntu 22.04 LTS

DM-16467

Added support for SQL Server 2022

DM-16642

Added support for MongoDB 8.x Atlas

DM-16840

Added support for service-type Snowflake users

DM-16849

Added support of MongoDB 8.x for MongoDB tools

DM-17048

Added an example for ScanDate function in the documentation

DM-16637

Fixed an issue that caused Live Automations with calendar-type Wait steps at weekly/monthly frequency to be unreliable

DM-16768

Fixed an issue that caused all clients to disconnect when Management Dashboard loads

DM-16954

Fixed an issue that caused the repository search function to fail to identify objects locked by a specific user

DM-16996

Improved Snowflake read performance when driver tracing is set to off

DM-17003

Fixed an issue that caused the Excel Output tool to throw a NullPointerException when writing fields containing error values

DM-17139

Fixed an issue that caused NoClassDefFoundError for EC2MetadataUtils from GetSecret with AWS Secrets Manager

DM-17160

Removed version from data providers for MSSQL

DM-4649

Updated city/state/zip reference data for AO Matching macros

DM-15273

Added support for MongoDB 7.x to MongoDB tools

DM-15876

When a Snowflake data provider is selected, the Region attribute is optional

DM-16231

Upgraded bundled JRE to 1.8.0_422-b05

DM-16365

Upgraded OpenSSL to latest 1.0.2 release

DM-16660

Added more detailed Wait step logging

DM-15363

Filesystem step can copy empty files to AWS S3

DM-15414

Salesforce Output tool now allows you to update fields to an empty value

DM-15671

When tracing is turned OFF for Snowflake data provider, log folders are no longer created

DM-15805

Improved download performance for FTP step when using SFTP

DM-15806

Fixed an issue that caused installation to fail on Windows Server 2022

DM-15859

Worker-only installs on Linux can now be upgraded from v9.4

DM-15865

Worker-only installs can now be performed on Linux

DM-15874

SERP module no longer truncates the Unit designator

DM-15875

Aborting automation no longer deletes remote files before download when FTP step performs MGET with remove option

DM-15953

Client no longer crashes when viewing extremely large logs

DM-15962

GET command in FTP step ignores Remove option

DM-15981

Redshift data providers now allow additional special characters in passwords

DM-16113

Fixed an issue that caused an automation to fail when the Snowflake data provider attempted to write its logging configuration file

DM-16369

Fixed an issue with interactive create and drop of temp table when target table is in non-default schema

DM-16635

Projects and automations run via rpdm_cmd no longer truncate variables passed via parameters option

DM-16636

Using Store Into when an automation Project step references project via variable no longer crashes Site server

DM-14012

Added support for CentOS 9

DM-10695

Added support for Windows Server 2022

DM-10009

Added support for Windows 11

DM-11525

Added support for AWS Redshift JDBC 2.1 driver

DM-11436

Added support for MongoDB 6.x for MongoDB tools

DM-13831

Added support for MongoDB 6.x Atlas

DM-15107

Enabled the ML Predictor tool to be used in a published web-service

DM-14937

Updated Data Management services on Linux to be managed as systemd units

DM-15009

Implemented less explicit error messages for logon failures

DM-15167

Incorrect documentation for Snowflake bulk data provider's Upload file size property

DM-15079

Token renewal bug in Azure Key Vault library sporadically causes secret retrieval failure

DM-15114

Same contents when saving multiple files from repository to disk

DM-15172

Snowflake bulk load not cleaning up stages

DM-15098

Client hangs when Calculate tool performs many cumulative non-sequential assignments

DM-14909

Review step in RDBMS mode doesn't persist options for Fields data grid

DM-14466

Client hangs when Calculate tool performs many cumulative assignments

DM-14597

Azure Storage Queue Send tool emits record to Failure connector with empty Exception field

DM-14485

Disambiguate file/folder handling semantics in Azure Blob Storage

DM-10376

Error when Excel Input tool reads formula cell containing CONCAT function

DM-14903

Using temp table with RDBMS Output tool causes error for Google BigQuery

DM-14809

RDBMS Input converts BigQuery STRING column to giant Unicode field

DM-14413

Excessive memory consumption in Excel Output tool

DM-14735

Objects removed from repository after Site server restart

DM-10978

Active automation lost when analyzing repository schema

DM-11426

Google BigQuery connector fails with dynamic query

DM-14658

Auto-connect favors Macro Output Proxy tool's New connector over Input

DM-14419

Unspecified GetSecret behavior with AWS Secret Manager

DM-14088

GetSecret function leaks memory with AWS Secrets Manager

DM-14456

JNI leaks memory when exception thrown

DM-14553

Azure Storage Queue tools must treat connection strings and SAS tokens as secrets

DM-14551

Azure Storage Queue tools fail with Default Azure Credentials authentication method

DM-14376

Cannot set AWS S3 secret key to vault reference

DM-14390

Secret control retains value when switching between key vault and password mode

DM-14212

Unauthenticated repository read causes client crash

DM-15241

Access denied error prevents FTP step from obtaining mutex lock on known hosts file

DM-14737

Salesforce tools do not clean up files in working directory for bulk operations

DM-14377

Parse Address tool does not treat PMB as unit designator

DM-15199

RDBMS Output tool attempts finalization after project abort or fail

DM-15266

OCI/ODBC data providers log errors to server trace for unsatisfied dependencies

DM-15334

Create Macro doesn't work with Auto Mapping

DM-15337

MPUT operation fails for FTP step with SFTP site hosted by AWS Transfer Family service

DM-10978

Analyzing repository schema won’t crash active automation

DM-11426

Dynamic queries using temporary table work with non-default dataset in Google BigQuery

DM-14735

temp folder cleanup won't delete repository objects on Site server restart

DM-14658

Auto-connect now favors Macro Output Proxy tool's Input connector

DM-14606

Azure Data Warehouse JDBC loader properly handles default values when treating empty strings as null

DM-14456

Better memory management across JNI boundary

DM-14088

GetSecret function no longer leaks memory with AWS Secrets Manager

DM-14212

Client won’t crash when attempting an unauthenticated repository read

DM-14419

Documentation improvements for GetSecret function with AWS Secret Manager

DM-14390

Secret control discards old value when switching between key vault and password mode

DM-14376

Secret key can be set to key vault reference when configuring AWS S3 authentication

DM-14553

Azure Storage Queue tools treat connection strings and SAS tokens as secrets

DM-14551

Azure Storage Queue tools can authenticate via Default Azure Credentials

DM-14466

Client won’t hang when Calculate tool performs many cumulative assignments

DM-11081

RDBMS Input can fetch schema for Snowflake view via Direct Table read

DM-11760

Salesforce Input tool can query objects and fields introduced in newer SFDC API versions.

DM-13452

Salesforce Input tool parses Time data types when the Real-time option is enabled.

DM-13531

Windows diagnostic capture includes more complete information.

DM-13596

Salesforce Output tool handles large result sets.

DM-13648

Snowflake data provider produces logs when Tracing is enabled (requires JDBC driver 3.14.x).

DM-13897

SQL Server Bulk Copy insert type retains precision of Float(8) fields.

DM-13909

SQL Server Bulk Copy insert type omits IDENTITY columns by default.

DM-13934

Snowflake data provider correctly calculates timezone offset when Snowflake and the Data Management host are in different timezones.

DM-14057

Snowflake bulk data provider can load tables in non-default schema.

DM-14061

New steps are no longer automatically added to an existing group when editing automations.

DM-14066

Automations with multiple variables correctly handle situations in which only one variable is set.

DM-14091

AO Compile Master Tool correctly breaks ties using specified priority field.

DM-14102

Disconnecting from a client session no longer retains selection state.

DM-14113

Spurious errors are no longer written to a disconnected client’s log.

DM-14130

Embedded JRE has been updated to 1.8.0_382-b05.

DM-12179

Support for Ubuntu 20.04 LTS.

DM-13041

Support for AWS S3 cross-account bucket access when manually specifying the bucket’s URI.

DM-12627

DBF Input tool now handles files containing zero-length fields.

DM-12797

Managed secrets are redacted from trace log files.

DM-12798

DBF Input tool correctly handles files when Unicode field exceeds 255 characters in length.

DM-12977

Improvements to reliability of Calendar type Wait step when configured for hourly and interval frequency.

DM-13142

Parquet Input tool now analyzes files with a limit of 100MB in S3 without a Hadoop connection.

DM-13265

RDBMS Output tool auto-commits changes to the Update and Key columns of the Schema mapping grid.

DM-9264

Calculate tool defaults field to correct data type on reassignment.

DM-3229

Calculate tool preserves type field assignment when autocomplete shows no options.

DM-9148

Assignments in Calculate tool reflect type changes to referenced fields.

DM-9264

Calculate tool defaults field to correct data type on reassignment.

DM-9270

Format difference between the Format and Scan properties no longer yields empty Date automation variable.

DM-9272

Changes made in the Select Fields data grid in Join and Join2 tools no longer require you to explicitly commit.

DM-9784

User Interface step now maintains values entered in the Date and DateTime controls.

DM-10177

RDBMS Output tool no longer sets values for unmapped fields in auto-increment columns in Snowflake databases.

DM-10633

SDK Transform tool now updates fields when a longer value is replaced with a shorter one..

DM-10803

FileSystem step now recovers from an error in subsequent iterations of a ForEach group.

DM-10805

Calendar type Wait step now considers timezone offset hen evaluating time with week/day-of-month frequency.

DM-10837

Improved Kafka read performance with registry schemas.

DM-10945

Project servers no longer crash when geocoding street intersection addresses without a specified city.

DM-11108

S3 no longer mis-identifies plain files as buckets under some conditions.

DM-11316

Improved performance when browsing Azure Blob Store containing a large number of blobs..

DM-11421

SDK tools now show existing Key Vault references for Secret properties.

DM-11443

RPDM client now correctly opens projects with hatched-style annotation.

DM-11467

Tool are now auto-commit property changes before running a project.

DM-11521

Data provider for Google BigQuery now provides additional context with error messages.

DM-11747

Parquet input tool now correctly identifies DateTime fields when no Hadoop connection is configured.

DM-9932

Orphaned annotations no longer prevent a project from loading.

DM-10243

Automations containing Project steps do not induce memory leakage in Execution server.

DM-10409

The Connection Replacement command in Advanced Repository Actions now works for automations as well as projects.

DM-10510

XML Input 2 tool interprets CDATA sections correctly.

DM-10577

File System Automation step can now create directories in Google Storage.

DM-9263

Reduced aggressiveness of temporary file cleanup to avoid failure of long-running projects.

DM-9807

Data Management installer for Windows is signed with a new longer-term trusted software publisher certificate.

DM-9818

Repository Cleanup action does not produce an error when it encounters a folder that was renamed and then deleted.

DM-9842

Credentials are redacted when JDBC connection strings are logged.

DM-10255

Corrected timestamps in parent path that may prevent opening previous repository item versions.

DM-10343

Users belonging to more than 100 groups on Linux are now able to access files that require secondary group permissions.

DM-10363

Data Management can now connect to SQL Server via the JDBC trusted data provider when running under a Windows Managed Service Account.

DM-9257

Data Management installer for Windows is signed with a trusted software publisher certificate.

DM-9375

Data Management installation on Windows completes successfully when the Microsoft Visual C++ Redistributable for Visual Studio 2015-2022 is present.

DM-1888

Improved performance of SQL Server JDBC Bulk Loader.

DM-2061, DM-8283

New Excel Input and Excel Output tools.

DM-5810, DM-5868

Support for 2-way (mutual) SSL authentication in published web services and Web Service Call tools.

DM-6127

Support for JDBC FastLoad for Teradata.

DM-6815

Support for Amazon Aurora PostgreSQL.

DM-7665

Options to disable specific SSL ciphers in C++ and Java.

DM-8023, DM-8024

Support for OAuth2 Service Principle authentication to Microsoft Azure Data Lake Storage Gen2 and Azure Blob Storage.

DM-8112

Option to override Site Settings to specify how many times a specific live automation can fail within a specified period of time before being disabled.

DM-8388

Support for Teradata 17.10.

DM-8384

Option to set output folder for Diagnostics Capture.

DM-8409

Kafka Input and Kafka Output tools can be used in published web services.

DM-8885

Data files with uncommon or missing suffix types that are saved to the repository can be viewed and edited.

DM-6172, DM-8178

JDBC Bulk Providers do not require that all input fields be mapped to output columns in target table.

DM-7320

Log connection string option for ODBC/JDBC data providers works correctly.

DM-7911

Salesforce Output tool supports use of Real-time option with Date fields.

DM-8133

Repository schemas created from Parallel Section CSV Input connector do not define persistent Splitting and Partition options.

DM-8233

Web Service Call tool does not duplicate headers if it encounters an error and retries the call.

DM-8244, DM-8282

Thread ID reuse does not cause Hadoop project run errors/failures.

DM-8245

Parallel Section input connector can display resource maps in the File browse dialog.

DM-8249

Errors in the evaluation of constant expressions are correctly reported.

DM-8333

Retry bug in Azure Data Lake Storage Gen2 no longer causes data loss.

DM-8347

Improvements in handling of case-sensitivity for object identifiers in Oracle.

DM-8372

JDBC SQL Server Bulk Copy insert type supports non-default schemas.

DM-8387

Synchronize repository command does not fail when a missing vault object is encountered.

DM-8427

Duplicate tool IDs generate a warning.

DM-8464

FileTimeStamp function returns correct local filesystem time.

DM-8532

Automation FTP step displays Password control does not require that Use SSL

DM-8569

FileSystem automation step’s Delete Directory operation works on blob storage systems.

DM-8571

Improvements to handling of field delimiters for Azure Data Warehouse JDBC Bulk.

DM-8636

Automation FTP step timeout does not result in abnormal automation server exit.

DM-8740

Multiple clients with identical configurations can operate on repository without causing client crash.

DM-8748

Choose Palette Items command works as expected.

DM-8796

Removed authentication token renewal for username/password and keytab Hadoop authentication strategies.

DM-8749 and others

Auto-commit of edited tool properties works correctly.

DM-8975

Users authenticating via Kerbero keytab can run projects on Hadoop.

DM-9008

Changes to Dynamic interface options in Project parameters are saved to project.

DM-9105

Connecting to SSL-enabled Data Management service via a non-SSL connection does not result in a memory leak.

DM-9106

Data type can be edited when redefining fields in Calculate tool.

DM-9155

The log4j2.formatMsgNoLookups JVM option is specified by default.

DM-8419, DM-8653, and others

Increased timeouts and retry attempts on operations for Azure Data Lake Storage/Azure Data Lake Storage Gen2 to accommodate occasional latency in responses from Azure.

DM-8436

When writing files to the Amazon S3 file system, all parts are correctly flushed before the file is closed.

DM-1733

New option in CoreCfg.properties to disable non-secure FTP/HTTP file transfer protocols at the site level.

DM-3097

Support for Oracle 19c.

DM-4181

Salesforce Input and Salesforce Output tools support the latest version of the Salesforce API (version 51.0).

DM-4954

Option to sort fields alphabetically in Change FieldTypes tool.

DM-6178

Support for Ubuntu 18 LTS.

DM-6728

Ask auto-commit option replaced by modeless notification, with Undo and Hide actions.

DM-6793

Support for Google BigQuery RDBMS data provider.

DM-6910

Support for SQL Server 2019

DM-6947
DM-7176

Support for Google Secrets Manager and AWS Secrets Manager credentials vaults.

DM-7136

Support for HDInsight 4.1

DM-7249

New tools for reading from and writing to Azure Storage Queues.

DM-7350

Support for Red Hat Enterprise Linux/CentOS 8.

DM-7611
and others

Enhanced diagnostic capture and trace logging.

DM-7895 and others

New error-handling options in CSV Input, CSV Streaming Transform, CSV Direct, Flat File Input, and Flat File Streaming Transform tools.

DM-7430
and others

FTP automation step does not issue HOST_KEY_NOT_VERIFIABLE error when Automatically accept SFTP server key as valid is enabled.

DM-7328

Dynamic interfaces created in Project Parameters retain the defined ACTION settings when projects are closed and reopened.

DM-7332

ForEach automation step recognizes defined Resource Map.

DM-7345

Save Layout command is enabled in Flat File Streaming Transform tool.

DM-7391

Avro schema files can be saved as Resource Maps.

DM-7439

Data Viewer’s Show/Hide Columns command is always available.

DM-7520

Wait automation step conforms to configured Calendar settings.

DM-7558

Keyboard shortcuts work as expected in Expression editor.

DM-7661

Trigger output of RDBMS Output tool waits for record flush before activating.

DM-7670

Data Management services with non-admin permissions can access configuration files following upgrade or repair (Windows only).

DM-7676

Data Management can list AWS S3 buckets across multiple regions.

DM-7685

CSV Input tool utilizes memory correctly when Error Report connector is enabled but not connected.

DM-7700

CSV Input Connector in a Parallel Section handles compressed input correctly.

DM-7713

Improved handling of missing and default values in variables.

DM-7716, DM-7752

Added ability to delete non-empty ADL Gen2 directories.

DM-7717

Improved performance analyzing and reading JSON files on AWS S3.

DM-7720
and others

Repository Search supports Last modified by and custom date filters.

DM-7756

Non-secure connection requests to SSL-enabled published web services do not block new connections (Windows only.)

DM-7757

Create Archive selection tree retains item selection when folders are collapsed.

DM-7814
and others

Published web services now conform to the HTTP/1.1 specification regarding HTTP 100 Continue response headers.

DM-7823

Optional field setting is preserved in Detect Format automation step properties.

DM-7912

MongoDB Updater tool handles nested documents correctly.

DM-8078

Projects terminate with error message upon load failure in the Azure Data Warehouse JDBC Bulk provider.

DM-8156, DM-8160

Parallel section with small or empty pre-sorted input connector does not fail with key-range data split list size=0 error.

DM-6631 and others

Enhancements to Avro Input and Avro Output tools:

Support for Avro schemas

Support for field input/output

DM-6732

Support for Avro over Kafka (including additional serializers/deserializers).

DM-6900

New GetSecret() function enables runtime access of secret values stored in an external secrets management vault.

DM-6974

New Hive Input2, Hive Output2, and Hive Execute2 tools for Hadoop clusters based on Hive version 3.0 or higher.

DM-2216

The Refresh command works for Data Management SDK-based tools that use shared settings.

DM-6495

New ResourcePathFromName() function returns the path value associated with mapped resources.

DM-6748

Option to specify the number of connection failure retry attempts added to Web Service Call tool.

DM-7081, DM-7251

YYYYMMDD date format is now fully supported.

DM-7399

Invalid repository object names do not stop Repository Sync when Continue syncing when errors are encountered option is enabled.

DM-7352

Open File browse dialog displays all files when File type is set to All files.

DM-7305
and others

Prior versions of projects and automations saved to repository can be opened from the Versions tab.

DM-7416, DM-7418

Avro and Parquet Input/Output tools correctly process DateTime values with divergent DST offsets.

DM-7336

The rpdm_shell utility no longer fails with "Mode was not set" errors under certain conditions.

DM-7343

ScanDate() and ScanDateTime() functions could return incorrect date/time when date string did not conform to specified format string.

DM-6456

Data Management system variable InstallDirectory returns full URI path to the Data Management installation location.

DM-6864

RDBMS Output tool using SQL Server JDBC Bulk Copy supports null date fields.

DM-7107

Data Management license validation accommodates minor changes in Site Server serial numbers.

DM-7125

Kafka Input tool stops reading records upon reaching defined limit.

DM-7191

File names with in-line dot characters retain .dlp or .dla suffix on project/automation save.

DM-7196

Automations now support the following Data Management system variables:

InstallDirectory
Hostname
DfsType
DfsRoot
LoadDateTime

DM-7199

Log files for failed automation projects are retained.

DM-7200

Disabling an input or output trigger also deletes any connections previously made to the trigger.

DM-7223

Restored ability to browse Amazon S3 and Google Cloud Storage buckets.

DM-7251

ForEach automation step recognizes YYYYMMDD date format.

DM-7305

Prior versions of projects and automations can be opened or deleted from the Versions tab of the Properties pane.

DM-895

Google BigQuery Input tool can extract the result of an arbitrary SQL query.

DM-1239

Improvements and new options for input file schema analysis.

DM-1454

All grids in the client have copy/paste capabilities.

DM-1462

Google BigQuery Input tool supports use of a wildcard to export data in excess of 1 GB as multiple files.

DM-2120

Notification automation step has new option to Stop and exit with error message.

DM-2793

SSL/TLS versions below 1.2 are disallowed by default (option to allow older versions can be set in CoreCfg.properties).

DM-3099

Support for IBM DB2 11.1.

DM-3157

Wait automation step’s Time limit option now specifies days as well as hours, minutes, and seconds.

DM-3507

Site and machine-level JVM memory settings can be configured for individual automations in the same way that they are for projects.

DM-3508

Record count displays during input file analysis are formatted with commas.

DM-4084
and others

Enhanced support for triggers on input/output and streaming tools. For input tools, the trigger connector can be connected to upstream tools and used to make the input tool wait for upstream processes to complete before starting. For output tools, the trigger option sends a single record to the trigger connector after all operations are complete. This can be used to sequence other operations by connecting to downstream tools that support a trigger input connector. When used with the Gateway tool, the trigger input/output behavior makes sequencing operations within a project straightforward and easy to visualize, allowing simpler action sequencing and write-before-read semantics without the use of tool priorities.

DM-4268

DM-3773

Abillity to restrict access to external executables used by the Execute automation steps and streaming transform tools (option can be set in CoreCfg.properties).

DM-4408

DM-4142

Enhanced repository search lets you filter repository objects by many different criteria. You can define custom searches to find specific property configurations. Use Data Management’s rpdm_shell utility to search the repository by object size.

DM-4654

Ability to copy properties from one tool and paste the configuration into another tool of the same kind.

DM-4724

You can set and override trace levels for specific projects, either from the command line or by defining project settings.

DM-4984

Personally-identifiable information (PII) is redacted by default from trace files and logs.

DM-519

Stream to Table tools now automatically sort input data by ID field (unless Do not sort by ID field is selected).

DM-5835

The JSON Input tool now supports map-enclosed arrays where each input file or field contains multiple JSON documents nested inside a single element.

DM-6231

Enhanced support for gzip-compressed file inputs.

DM-6254

New Compress and Decompress tools support field-by-field data compressing using gzip, zlib, and ZIP compression.

DM-6334

Ability to open projects and automations from their associated items in the Logs viewer on the Management Dashboard.

DM-6346

Optional integration with Azure Key Vault. With Azure Key Vault integration enabled, you have the option of storing secrets such as passwords as either encrypted values (the default in Data Management version 9.0 and later) or as Key Vault references. Your organization must have an Azure Key Vault instance provisioned and have access credentials.

DM-6402

Data Management now supports accessing Amazon S3 buckets via the use of an IAM role attached to an EC2 instance.

DM-6543

DM-6438

The Advanced Repository Action Path Replacement command can now be configured to replace or retain paths in a variety of repository objects, including projects/macros/automations, paths to data, and all other paths.

DM-6673

New MongoDB Array Updater and Cosmos DB Array Updater tools let you alter existing documents in a collection by adding data to an array as an atomic document database operation (MongoDB syntax $push). Enable the Unique result option to add a value to the target array only if the value does not already exist in the array (MongoDB syntax (addToSet).

DM-6725

Attempting to enable a Live Automation without valid credentials shows a warning before the Live Automation is committed.

DM-6861

Boolean types in the type-chooser interface and schema grids are displayed with format information.

DM-7087

Performance of Parquet Output tool is greatly improved.

DM-884

When Skip duplicates option is selected, Load Archive command can load archives containing duplicate packages.

DM-1004

When the Web Service Call tool is configured for Retry strategy of Never, the Retry count setting is ignored.

DM-1457

Min() and Max() functions no longer return null, except that Min() with an empty string returns the empty string.

DM-3001

CSV Direct tool handles poorly formatted email addresses and URL domain names.

DM-4675

Preconfigured tools containing macros are visible on the tool Palette.

DM-4690

Management Dashboard's Attach command opens and displays projects or automations in correct tab.

DM-4960

Data viewer’s Find Next and Find Previous commands work with doubled space characters.

DM-5446

RDBMS Output tool updates DateTime records on SQL Server (OLEDB) to expected scale.

DM-5826

RDBMS In-Database Processor tool accurately reports row counts that exceed 32,768.

DM-5920

MSSQL OleDB reports correct insert record count when project fails with error.

DM-6098

Analysis of large input files can be stopped before analysis completes.

DM-6166

Information displayed on Management Dashboard is sorted appropriately for different data types (numeric, date/time, text).

DM-6167

Users are prevented from nesting automations.

DM-6177

CSV Output tool's Split by Size option calculates split boundaries correctly when writing with gzip compression.

DM-6223

Data loads using the Snowflake JDBC bulk provider support field names containing a trailing backslash.

DM-6224

Data Management will abort the project run if the RDBMS Input tool encounters an exception while processing a result set using a JDBC data provider.

DM-6336

Automation Execute steps configured to log stdout/stderr as messages no longer fail when encountering very long lines.

DM-6486

Open File dialog displays a warning when a folder contains too many files to list, offering the user an opportunity to narrow the list using a custom file pattern.

DM-6514

Stream To Table tool configured with a non-existent ID field on input does not cause fatal error.

DM-6607

Management Dashboard does not display duplicate client sessions.

DM-6633

Snowflake JDBC bulk loader retains fractional seconds in DateTime field values.

DM-6641

The command line rpdm_sync operation supports use of the -test flag.

DM-6649

Preconfigured tools created in older versions of Data Management can be migrated when upgrading.

DM-6676

Document Extractor tool includes option to ignore time zone when converting String fields to DateTime type.

DM-6689

Input tools that have no (non-wildcarded) input file configured always display a configuration error.

DM-6726

Document Injector tool preserves the scan/format strings of input data types.

DM-6727

Client sessions are correctly released when session is closed.

DM-6802

Loading a repository schema can no longer alter tool configuration settings.

DM-6855

Automation Wait step configured to wait for a file does not intermittently stall.

DM-6880

Flat File Input tool preserves leading spaces in fields of type TextFixed.

DM-6892

The SQL Server JDBC Bulk Copy provider now correctly writes all Decimal value types.

DM-6983

Context-customized projects in automations can be viewed via Attach and View commands in Management Dashboard.

DM-7012

Automations that are open or running when a variable is changed in Site Settings do not need to be restarted to access Site variables.

DM-7016

Published web services correctly authenticate Initial request newly obtained OAuth2 tokens.

DM-7023

Synchronize from Disk command now updates the timestamp and key of modified projects doing sync operations.

DM-7027

Synchronize from Disk command correctly handles file names with in-line dot characters.

DM-7028

Merge tool correctly sets zero for missing values that are unsigned integers.

DM-7075

Automation Expression steps are protected from multi-thread data collisions caused by simultaneous read/modify actions.

DM-7078

Parquet Input and Output tools can read and write DateTime fields as Parquet Timestamps, with additional options for timezone adjustments.

DM-17008

Data Management’s Snowflake Bulk data provider interprets the literal string “NULL” or “null” as a logical NULL. When it encounters a record field whose value is a literal “NULL” or “null”, it will insert a logical NULL into the corresponding table column. This behavior stems from limitations in the bulk loader’s file format. Redpoint may explore addressing this in a future version of Data Management.

Customers whose business logic depends on inserting a literal “NULL” or “null” are advised to use the non-bulk Snowflake data provider and set Insert type to Single record or Block.

DM-14623

Projects created in earlier Data Management versions store Azure Storage Queue plaintext secrets.

Prior to Data Management 9.4.8, the Connection string and SAS token properties for Azure Storage Queue tools were presented in the Properties panel as Text fields. Consequently, when the tool configuration was persisted, values were stored as plain-text.

Beginning In Data Management 9.4.8, the Connection string and SAS token properties are treated as secrets and presented in the Properties panel as Password fields. This ensures those properties are encrypted when persisted in the tool configuration.

However, Data Management cannot programmatically migrate plain-text properties to encrypted secrets in existing projects or shared settings.

Manually reconfigure the Connection string and SAS token properties in projects that contain Azure Storage Queue tools, and in shared settings under Repository > Settings > Tools > Azure Storage Queue.

DM-12522

Retype tool configuration does not convert new fields.

Property auto-commit might disable the Convert new fields option when the Change Field Types tool is selected on the canvas. If new fields are detected in the upstream record during a subsequent project execution, it might not apply the specified Data type for new fields.

Commit the Change Field Types tool’s properties using the Commit button each time the Change Field Types tool is selected.

DM-14135

SQL Server Bulk Copy insert type fails when table contains NVARCHAR(MAX) columns.

Due to a limitation in JDBC, the SQL Server Bulk Copy insert type, does not support NVARCHAR columns whose lengths exceed 4000 characters.

Modify your table’s DDL. You can:

• Use VARCHAR(MAX) if the Unicode character set isn’t required.

• Set the NVARCHAR column’s LENGTH to 4000 or less.

If you cannot modify your table’s DDL, use the RDBMS Output tool’s Block insert type.

DM-13908

SQL Server Bulk Copy insert type cannot be used when table includes timestamp column.

The SQL Server Bulk Copy insert type cannot be used when the table includes a column whose data type is timestamp or rowversion. This type is generally used as a mechanism for version-stamping table rows with an incrementing number that does not actually represent a date or time.

Use the block inert type and omit timestamp and rowversion columns from the schema mapping.

DM-953

Using built-in SQL functions with RDBMS tool configured for Oracle JDBC causes error.

Due to an issue with Oracle’s JDBC driver, SQL statements that include Prepared Statement parameter replacement may fail with an error, because the JDBC driver is unable to replace parameters used as arguments to a SQL function in a Prepared Statement. For example, this statement:

SELECT * FROM oracle_tablename WHERE DUMMYDATE = TO_DATE(?,'yyyy-mm-dd') ORDER BY ID,DUMMYDATE

will get the following error:

Error creating field specification list: Unsupported feature: checkValidIndex

Do not use Prepared Statement parameter replacement with Oracle/JDBC.

DM-4850

Mismatch between Site Server and Execution Server security settings causes failures.

It is possible to create a multi-host Data Management site with incompatible security settings. An Execution Server configured with different core security settings than its Site Server will not run correctly.

Check that the Data Management configuration file CoreCfg.properties (located in the Data Management installation folder) has the same core security settings as the Site Server for the following properties:

• use_ssl

• validate_ssl_cert

• enable_advanced_security

• administrator_os_user

If necessary, stop the Execution service, edit the CoreCfg.properties file using a text editor, and then restart the Execution service.

DM-4939

Errors in tools embedded within macros are not shown in when “Show errors for selected tool only” is enabled.

The Show errors for selected tool only filter option on the Message Viewer does not display errors in tool that are contained within a selected macro.

None.

DM-5025

Cannot change repository view to specific time.

Setting the Site Server’s Repository View property to a defined Label or Specific time has no effect.

Open a specific version from the Version tab on the repository object’s Properties pane.

DM-5160

When Advanced Security is enabled, logons with no domain get an error.

On Windows, if advanced security mode is enabled, users with logons that do not include a domain (for example, username instead of domain\username) may be unable to open or create projects and automations, and may see an error message like errno6.

Either specify a Default domain in Site Settings, or specify a fully qualified OS User for each Data Management user.

DM-5456, DM-5491

Oracle identifiers are case-sensitive, which can cause bad behavior when there’s a case mismatch.

Oracle identifiers are case-sensitive, which can cause problems when using RDBMS Output tools configured to use the Oracle JDBC driver.

RDBMS Output tools writing to Oracle should enclose identifiers in quotes.

DM-5873

Repository archive created with Version 9.0.2 client and Version 9.0.1 server cannot be loaded

Repository archives created using a Version 9.0.2 Data Management client and a Version 9.0.1 Site Server are empty and cannot be used with the Load Archive command.

Upgrade the Data Management Site Server to Version 9.0.2 or later.

DM-7083

In SQL Server, conversion of a datetime2 data type to a datetime data type results in out-of-range value.

In SQL Server, attempting to update tables containing datetime2 fields may result in improper truncation of data when the Update option to Use temp table is set to Always or Batch Only. Data Management supports Insert and Update into existing tables with datetime2 fields but does not support creating tables with datetime2 fields.

Pre-create the temp table with the desired datetime2 field, instead of allowing Data Management to create it for you.

DM-7508

JDBC error reading from table with special characters in field name

An RDBMS Input tool configured to perform a direct table read using a SQL Server provider may return an Invalid column schema error when one of the target fields contains special characters.

Instead of using a direct table read, use Select * or Select [field_name].

DM-7992

Google BigQuery is slow when temporary tables aren't used for update and delete.

An RDBMS Output tool performing update or delete operations on a Google BigQuery target will run very slowly unless it is configured to use temp tables.

Configure the RDBMS Output tool to use temp tables on output, as shown below:

DM-8530

RDBMS Output tool does not consider character encoding when specifying column lengths for automatically created tables.

VARCHAR columns will accept multi-byte UTF-8 characters. However, in many RDBMS, VARCHAR column lengths are measured in bytes rather than characters. If you’re storing 10 single-byte characters in a VARCHAR column, its length need only be 10 bytes. However, If you’re storing double-byte characters, you’ll need 20 bytes.

The Automatically create missing tables option in the RDBMS Output tool determines the table schema using the input record, but does not account for the storage requirements of multi-byte characters when calculating the length of VARCHAR columns. The resulting DDL may create a table whose VARCHAR column lengths are insufficient for the record data.

Avoid using the Automatically create missing tables option when working with records that may contain multi-byte characters. Instead, select the Script table creation for missing tables option and edit the resulting DDL to ensure column lengths adequate to storing the field values.

DM-8533

RDBMS Input tool missing input node.

In Version 9.1.1 and later, the Input connector that provides input to dynamic queries for the RDBMS Input tool is no longer displayed by default.

To display the dynamic query input connector, insert a row in the Dynamic Query data grid, specify the SQL field, and click Commit.

DM-8778

RDBMS Output tool cannot display empty schemas.

Schemas that don't contain any tables are not included in the RDBMS Output tool’s list of available Schemas. This can cause problems when the RDBMS Output tool is configured to Automatically create missing tables and no tables exist in the destination schemas.

Manually enter the empty schema’s name in the Schema property, and the new table name in the Table property. Once the project is run and the new table created, select Refresh tables to display both the schema and the new table.

DM-8784

Long-running projects authenticating to Azure services via OAuth2 may be impacted by default token expiry

By default, OAuth2 tokens in Azure expire after an hour. This can create problems for long-running Data Management projects authenticating to Azure services via OAuth2.

Use a different authorization method (ADL2 access key, or ABS connection string), increase the token lifetime in the Azure portal, or edit the project to write smaller files.

DM-8792

Abandoned multipart uploads to cloud storage can leave behind blob parts or temp files.

If you cancel a write operation to cloud storage, the "parts" of the incomplete multi-part upload are not deleted immediately.

Amazon and Azure have garbage collection facilities to remove these parts after a period of time, but you may need to configure your storage provider to enable this. Google Cloud Storage does not appear to have a garbage collection facility, so you will need to use the storage manager web page to periodically check for uploads and remove them.

DM-8845

Configuration locator may not display tool/macro names for older projects.

If you attempt to reconfigure a pre-version-9.4.1 project or macro from an automation, the tool icons are shown but not the tool names, making it difficult to distinguish multiple instances of the same tool type.

In the current Data Management version, open the old project or macro and then save it.

DM-8934

Automation connector displays incorrectly.

Certain automation connections may display connectors incorrectly when the project is saved and reopened.

The automation will execute correctly. No workaround is necessary.

DM-10508

RDBMS Output tool schema mismatch when inserting very large Unicode or TextVar fields.

An RDBMS Output tool configured with any JDBC provider will fail with a schema mismatch error when an upstream Unicode or TextVar field sized ~10MB (≥10388608 bytes) is mapped to a database column.

Reduce the configured size of the upstream TextVar field to less than 10MB (10388608 bytes).