Data ingestion basics

There is a general pattern to ingest data regardless of the data itself. This page calls out the basics of the pattern, and subsequent sections will continue to elaborate on these basics. We will start with how the data is represented in the platform, followed by what feed layouts are and how they are used when configuring sources and feeds for loading the data.

How is the data represented within the platform?

• There is a standardized data model organized into subject areas.

• Subject areas may be common or specific to industries / use cases / client requirements.

• All Redpoint CDP deployments contain the core subject areas, which include Core feed layouts as well as additional industry- or client-specific data structures (refer to Retail feed layouts).

• This data model is standardized but extensible. Below are a few groupings of objects that make up the data model along with a visual that shows how the objects support each other to materialize the CDP for a retail client. This pattern would look similar for other verticals, with the elements in green reflecting those industry specifics.

  • Base and extensions tables

  • Identity resolution

  • Lookup tables

  • Aggregations

What is a feed layout?

• A feed layout is a template for importing data relative to a specific subject area (customers, transactions, loyalty, etc.).

  • All expected fields

  • Required/optional indicators

  • Keys

• There is a predefined set of feed layouts that include commonly used subject areas.

• Many feed layouts include associated extension tables to support extensibility.

• Custom feed layouts can be enabled for vertical-specific or client-specific use cases.

• Feed layouts support ingestion by enabling you to map source data into a standardized input format that Redpoint CDP expects to initiate a standard processing flow.

  • Validation

  • Profiling

  • Identity resolution

  • Transformation

Sources and feeds

• Sources typically represent individual source systems (e.g., CRM, Ecommerce, POS)

• Each source may provide data through one or more feeds (e.g., customers, transactions, locations)

• All feed data must be provided based on a specified feed layout

• Conduct an inventory of all desired sources and feeds as a part of the Redpoint CDP implementation process

Source and feed inventory

The following table is an example of how you can start to document your list of sources, feeds, and associated descriptions.

Source and feed inventory with feed layout alignment

The next step is to review the various sources and determine what data topic(s) and feed layout(s) that the source data aligns with.

Additional details on how to determine this alignment as well as next steps are discussed in the section Data Ingestion Details.

At a high level, these are your next steps:

1. Take the list of sources and target feed layouts and define a detailed mapping of the source data fields to the associated feed layout.

2. Take your source data and feeds and convert it to a feed layout.

3. In the Redpoint CDP Web UI, define each of the sources and feeds and provide some additional details related to the feed, such as the file names and the cadence that the source feed will be sent to Redpoint CDP.

4. Create a sample file and provide it to the platform for data ingestion. We recommend that you create a small sample file of 5-10 records to start, choosing records that represent a few different types of values for each of the fields that are being provided so that you can establish a baseline validation for each feed layout that will be provided.

Customer-specific lookup tables and associated values will need to be updated before data can be ingested for a feed layout that requires a customer-specific lookups.

Additional details related to testing feed layouts and loading data will be provided in other pages in this onboarding section.

The following diagram outlines the steps that we have discussed up to this point.