LuxSci: Recovering from a failed Interactive Activity

Overview

This document walks through how to recover from a failed Interactive Activity (IA) execution with the error “Primary cache data loss detected. Configuration is set to fail on data loss” using the LuxSci plugin.

Scenario 1: Recovering an IA with no batches of emails sent yet

The following steps assume that the error is caused by the cache expiration while the IA waits for the next trigger coming from a successful workflow execution. This scenario can be resolved without creating an additional workflow.

1. Download and install Redis Insight at https://redis.io/downloads/#Redis_Insight.

2. Retrieve the cache values in the rpi_FailoverCache table using the query below:

SELECT [Key]
      ,[Value]
      ,[ExpiresAt]
FROM [Pulse].[dbo].[rpi_FailoverCache]
Where [Key] like 'LuxSci_%' AND [Key] like '%[ClientID]_[ChannelExecutionID]'

Sample result:

ClientID value can found in the rpi_Clients table.

ChannelExecutionID can be found in the ops db using query below:

SELECT [WorkflowID]
      ,[ChannelExecutionID]
  FROM [OFFERHISTORY_TABLE]
  where WorkflowID = [WORKFLOW_ID]

WORKFLOW_ID value can be found in the Interaction results window.

3. Open Redis Insight and connect to the Redis server.

4. Add the keys below with type String. The BatchSize value can be taken from the result in step 2.
   Key: LuxSci_Inflight_[ClientID]_[ChannelExecutionID]
Key type: String
TTL: 31536000
Value: “[]”
   

   Key: LuxSci_MailMerge_[ClientID]_[ChannelExecutionID]
Key type: String
TTL: 31536000
Value: “{\"LastBatchNumber\":0,\"BatchSize\":[BatchSize],\"BatchSuccess\":0,\"BatchFailed\":0,\"TemplateId\":0}”

5. Play the failed activity.

Scenario 2: Recovering an interactive activity with batches of emails already sent

The following steps can be performed if the error is caused by the cache expiration when the IA is played coming from an OOMKill or paused state. To avoid a duplicate send, this steps require identifying recipients that were already sent and creating a separate single workflow recurring activity to target the incoming recipients of the failed IA.

1. Download the CSV sent report from LuxSci:

   1. Login to LuxSci to download the sent report for the specific channel execution ID.

   2. Click Reports > Sent Email > From SMTP Server > Messages Sent.

   3. Click Custom Header Fields.

   4. Set chid:Contains = "{ChannelExecutionID}_{OfferTemplateInstanceID}_{Channel.ID:N}"
      The chid value is formatted as follows:
      chid: {ChannelExecutionID}_{OfferTemplateInstanceID}_{Channel.ID:N}_{RPContactID}
      For example: 421_421_fb751f1d523947429148a31be8eff8bb_19828
      Do not include the RPContactID from the chid search to return all the records from specific channel execution ID.
      You can use the following queries to get the values with the associated Workflow ID, which can be found in the Interaction results window.
      You can find ChannelExecutionID and OfferTemplateInstanceID in the ops db using this query:

      SQL

      SELECT 
          [WorkflowID],
          [ChannelExecutionID],
          [OfferTemplateInstanceID]
      FROM [OFFERHISTORY_TABLE]
      WHERE [WorkflowID] = [WORKFLOW_ID];

      You can find ChannelID in the ops db using this query:

      SQL

      SELECT 
          [ChannelExecutionID],
          [WorkflowAssociationInstanceID],
          [ChannelID]
      FROM [op_ChannelExecution]
      WHERE [WorkflowAssociationInstanceID] = [WORKFLOW_ID];

   5. Click Search.

   6. Click the CSV button to download the sent report.

2. Extract the RPContactID and ChannelExecutionID from the CSV file:
   Open the downloaded CSV file in Excel. Split the value of chid to get the RPContactID and ChannelExecutionID. 

3. Create a table in the data warehouse and insert the extracted RPContact and ChannelExecutionID.
   Sample query:
   SQL Server

   SQL

   CREATE TABLE RECOVERYDATA (
       [RPContactID]        BIGINT NOT NULL,
       [ChannelExecutionID] INT    NOT NULL
   );
   
   INSERT INTO RECOVERYDATA
   VALUES (11, 4232);

   Snowflake

   SQL

   CREATE TABLE DEMORES.PUBLIC.RECOVERYDATA (
       "RPContactID"        BIGINT,
       "ChannelExecutionID" INT
   );
   
   INSERT INTO DEMORES.PUBLIC.RECOVERYDATA
   VALUES (11, 4232);

4. Perform catalog sync in RPI.

5. Create attributes from the created table.

6. Create join between the Offer History (OH) table and the created table using RPContactID.

7. Create a selection rule to target the record in the created table; this will serve as the suppression when configuring the audience. For example:

8. Create a selection rule to target the failed IA input (Open, ClickThrough, etc.). For example:

9. Create an audience using the selection rule to target the failed IA input and suppression using the selection rule for the records that were already sent by the failed IA. For example:

10. Stop the failed workflow activity.

11. Create a single recurring workflow and configure the batch audience to target the input from the failed IA.

12. Execute the workflow. It should select the incoming recipient as targeted by the failed IA input and suppress the records that were sent previously by the failed IA.