Data ingestion details

This section elaborates on data ingestion details and prepares you to load data to the CDP. Below are a few notes on things that apply generally to most or all of the feed layouts.

This section provides detailed information about the following:

Feed layouts: processing sequence and dependencies

Feed layout holding tables

Core feed layouts

Retail feed layouts

Data ingestion use case examples

Business unit code

Records that do not have a business unit (BU) assigned will be assigned a default BU from the lookup table which itself is set to enterprise by default. Thus, it will be the default BU if the lookup table default is not updated/changed. Although BU is not a required field in the feed layouts, a BU will be assigned for all records.

Feed layout dynamic formatting

Feed layout ingestion has a feature that allows you to generate a file without every one of the potential fields outlined in a feed layout. You only need to generate files that have the required fields as well as any fields that you have data to populate. So, if you do not have a value for a predefined field in the layout, you do not need to include the field in your input file. This is a useful feature that minimizes the size of the input file and makes it easier to review your input files if you do not have data for a lot of the predefined fields. You still need to provide the correct field header names that corresponded to the feed layout definition in order for the data to pass validation and get mapped to the appropriate column within the CDP.

Extension tables

Some subject areas have both a primary feed layout as well as an extension feed layout that provides customization. For example, there is a feed layout for party profile as well as a party profile extension feed layout. The purpose of the extension feed layouts and tables is to provide a way to ingest data that is not defined as a part of the primary feed layout but that is contained within the data. An example may be data on the number of children that a person has or if they have purchased a car or house recently. Those particular attributes are not a part of the party profile feed layout, so they could be loaded to the CDP via the party profile extension feed layout and table.

The extension tables are based on the primary key of the base table. So, in the case of party profile, the source_party_profile_id is the key field for the party profile extension table. In addition to the primary key for each of the extension tables, there are a series of attributes defined in the pattern of a key/value pairs. So, for example there are custom_field_name_1 and custom_field_value_1.

In the example where the client captures the number of children that their customers have, they could provide that data in the first custom field as shown in the following table. There are a series of these fields that are defined, grouped for various data types.

Fields 1-20: character data type
Fields 21-40: datetime data type
Fields 41-60: big integer data type
Fields 61-80: decimal data type

Once a custom field has been delegated to capture a specific attribute, it should not be used to capture a different attribute for a different party profile extension record. So, you should NOT provide number of children as custom field 22 for some records and number of grandchildren as the same column. A separate column should be used to capture number of grandchildren.

Feed Layout Field Name	Value Provided
`custom_field_name_22`	`NumberOfChildren`
`custom_field_value_22`	`3`
`custom_field_name_23`	`NumberOfGrandChildren`
`custom_field_value_23`	`1`

Holding tables and referential integrity

Holding tables are used to place records that pass validation but do not load to the final destination because they are missing a component to complete the referential integrity when loading the record. An example would be that you provide a transaction file that has a location ID that is not currently in the location table.

For details related to holding tables, refer to Feed layout holding tables.