Amazon SES Data Streams Configuration Guide

Overview

This topic covers configuring Amazon Simple Email Service (SES) Data Streams to SQS, S3 Bucket, and Callback Service API.

Prerequisites

The following resources are required prior to proceeding to the configuration sections:

• VPC

• Subnet

• Security Group

• IAM User

AWS SES to SQS data stream configuration

This section covers the configuration of streaming data from Simple Email Service (SES) to SQS (Simple Queue Service).

Requirements

The following resources are required to be provisioned:

• Simple Email Service (SES)

• Simple Notification Service (SNS)

• Simple Queue Service (SQS)

• IAM Policy to be attached on your IAM User:

  1. Go to your AWS IAM Console > <YOUR-USER> and select Permissions tab > Add permissions > Create inline policy:

     

  2. In the Policy Editor section, choose JSON and override the policy definition with the definition below, then click Next, Add your policy name, and click Create Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SQSPolicy",
      "Effect": "Allow",
      "Action": ["SQS:*"],
      "Resource": [
        "arn:aws:sqs:us-east-1:<AWS-ACCOUNT-ID>:<SQS-NAME>"
      ]
    },
    {
      "Sid": "SESPolicy",
      "Effect": "Allow",
      "Action": [
        "ses:SendRawEmail",
        "ses:SendTemplatedEmail",
        "ses:BatchGetMetricData",
        "ses:CancelExportJob",
        "ses:CreateConfigurationSet",
        "ses:CreateConfigurationSetEventDestination",
        "ses:CreateContact",
        "ses:CreateContactList",
        "ses:CreateCustomVerificationEmailTemplate",
        "ses:CreateDeliverabilityTestReport",
        "ses:CreateEmailIdentity",
        "ses:CreateEmailIdentityPolicy",
        "ses:CreateEmailTemplate",
        "ses:CreateExportJob",
        "ses:CreateImportJob",
        "ses:DeleteConfigurationSet",
        "ses:SendRawEmail",
        "ses:DeleteConfigurationSetEventDestination",
        "ses:DeleteContact",
        "ses:DeleteContactList",
        "ses:DeleteCustomVerificationEmailTemplate",
        "ses:DeleteEmailIdentityPolicy",
        "ses:DeleteEmailTemplate",
        "ses:DeleteSuppressedDestination",
        "ses:GetAccount",
        "ses:GetBlacklistReports",
        "ses:GetConfigurationSet",
        "ses:GetConfigurationSetEventDestinations",
        "ses:GetContact",
        "ses:GetContactList",
        "ses:GetCustomVerificationEmailTemplate",
        "ses:GetDeliverabilityTestReport",
        "ses:GetDomainDeliverabilityCampaign",
        "ses:GetDomainStatisticsReport",
        "ses:GetEmailTemplate",
        "ses:GetExportJob",
        "ses:GetImportJob",
        "ses:ListConfigurationSets",
        "ses:ListContactLists",
        "ses:ListContacts",
        "ses:ListCustomVerificationEmailTemplates",
        "ses:ListDomainDeliverabilityCampaigns",
        "ses:ListEmailIdentities",
        "ses:ListEmailTemplates",
        "ses:ListExportJobs",
        "ses:ListImportJobs",
        "ses:ListSuppressedDestinations",
        "ses:SendBulkEmail",
        "ses:SendCustomVerificationEmail",
        "ses:SendEmail",
        "ses:TestRenderEmailTemplate",
        "ses:UpdateConfigurationSetEventDestination",
        "ses:UpdateContact",
        "ses:UpdateContactList",
        "ses:UpdateCustomVerificationEmailTemplate",
        "ses:UpdateEmailTemplate",
        "ses:SendBulkTemplatedEmail",
        "ses:CreateTemplate",
        "ses:GetTemplate",
        "ses:DeleteTemplate",
        "ses:UpdateTemplate"
      ],
      "Resource": "*"
    }
  ]
}

Configuration

1. Go to your AWS SES Console and click Configuration Sets > Create Set:

   

   
   

2. Fill in the Configuration set name, choose your Sending IP pool, and click Create set:

   

3. Once you’ve successfully created a Configuration Set, click Event destinations > Add destination:

4. On the Select event types page, click Select all and Next:

5. On the Specify destination page, choose Amazon SNS, fill in the destination Name, select your SNS topic, and click Next:

6. On the Review page, once you validated your config, click Add destination:

7. On your AWS SQS Console, go to Queues > SNS subscriptions and click Subscribe to Amazon SNS topic:

8. On the Subscribe to Amazon SNS topic page, choose your SNS topic and click Save:

AWS SES to S3 Bucket data stream configuration

This section covers the configuration of streaming data from Simple Email Service (SES) to S3 Bucket.

Requirements

The following resources are required to be provisioned:

• Simple Email Service (SES)

• Amazon Data Firehose

• Amazon S3 Bucket

Configuration

1. Go to your Amazon Data Firehose console and click Firehose streams > Create Firehose stream:

   

   
   

2. In the Choose source and destination section, choose Direct PUT for Source and Amazon S3 for Destination:

   

3. In the Firehose stream name section, fill in your Firehose stream name:

   

4. In the Destination Settings section, hover over S3 bucket > Browse and choose your S3 Bucket. Next, optionally, if you want to stream data in a directory on the S3 Bucket, go to S3 Bucket prefix - optional section and fill in your desired directory. Click Create Firehose Stream:

   

5. Notice that there’s an error: Your Firehose stream was not created. This is expected. That first attempt created the IAM Role and IAM Policy for the configuration. Now, expand the error and take note of the IAM Role Name that has been created:

   

6. Now, open another instance of your browser and go to your AWS IAM Console > Roles and search for the IAM Role that you noted from the previous step, in this case: KinesisFirehoseServiceRole-my-firehose-s-us-east-1-1751410726925, then click the IAM Role:

   

7. On the Permissions tab, select Add permissions > Create inline policy:

   

8. On the Policy Editor section, click JSON and replace the policy definition with the IAM Role Policy below then click Next:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "FirehosePolicy",
      "Effect": "Allow",
      "Action": "firehose:PutRecordBatch",
      "Resource": "arn:aws:firehose:us-east-1:<AWS-ACCOUNT-ID>:deliverystream/<FIREHOSE-STREAM-NAME>"
    },
    {
      "Sid": "AssumeRolePolicy",
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::<AWS-ACCOUNT-ID>:role/<IAM-ROLE-NAME>"
    }
  ]
}

9. On the Policy details section, fill in the Policy name and click Create policy:

10. Now, go back to the AWS Console Firehose stream create page and click Create Firehose stream. You will now be able to do so:

11. Go to your Amazon SES console and click Configuration Sets > Create set:

12. Fill in the Configuration set name, choose your Sending IP pool and click Create set:

13. Once you’ve successfully created a configuration set, click Event destinations > Add destination:

14. On the Select event types page, click Select all and Next:

15. On the Destination options section, select the following:

    • Destination type = Amazon Kinesis Data Firehose

    • Fill in the Name of your destination

    • Delivery stream = Name of your firehose stream

    • IAM Role = the IAM Role you edited on the previous instructions, in this case KinesisFirehoseServiceRole-my-firehose-s-us-east-1-1751410726925

    • Click Next:

16. Once you’re done reviewing, click Add destination:

17. You will notice that you’ll get an error: The input you provided is invalid, and this is to be expected, because we need to set a trust relationship policy on the IAM Role:

18. Open a new tab on your browser and go to AWS console IAM > Roles and search for your IAM Role, in this case: KinesisFirehoseServiceRole-my-firehose-s-us-east-1-1751410726925, then click it:

19. On the Trust relationships tab, click Edit trust policy and replace the policy definition with the one below. Once done, click Update policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "firehose.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ses.amazonaws.com"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "AWS:SourceArn": "arn:aws:ses:us-east-1:<AWS-ACCOUNT-ID>:configuration-set/<SES-CONFIG-SET-NAME>",
          "AWS:SourceAccount": "<AWS-ACCOUNT-ID>"
        }
      }
    }
  ]
}

20. Once the trust policy is updated, head back SES Add event destination and click Add destination. The config should now push through.

AWS SES to Callback Service API data stream configuration

This section covers the configuration of streaming data from Simple Email Service (SES) to RPI Callback Service API.

Requirements

The following resources are required to be provisioned:

• Simple Email Service (SES)

• Simple Notification Service (SNS)

• Amazon Lambda Function

Configuration

1. Go to your AWS SES Console and select Configuration sets > Create set:

   

   
   

2. Fill in the Configuration set name, choose your Sending IP pool, and click Create set:

   

3. Once you’ve successfully created a Configuration Set, select Event Destinations > Add destination:

4. On the Select event types page, click Select all and Next:

5. On the Specify destination page, choose Amazon SNS, fill in the destination Name, select your SNS topic, and click Next:

6. On the Review page, once you’ve validated your config, click Add destination:

7. Now go to AWS Console Lambda > Functions and click Create function:

8. Set the settings below and click Create function:

   • Choose one of the following options to create your function: Use a blueprint

   • Blueprint name: Process messages pushed to an SNS topic (nodejs18.x)

   • Function name: Fill in your function name

   • Execution role: Create a new role with basic Lambda permissions

9. SNS trigger: Choose your SNS topic

10. Now, on the Code source, replace with the one below and click Deploy (Ctrl+Shift+U):

import https from 'https'
export const handler = async (event) => {
    // Parse the SNS message
    const snsMessage = JSON.parse(event.Records[0].Sns.Message);

    // Define the HTTPS endpoint details
    const endpointUrl = 'https://<RPI-CALLBACK-SERVICE-URL>'; // Replace with your actual endpoint
    const requestBody = JSON.stringify(snsMessage);

    const options = {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Content-Length': Buffer.byteLength(requestBody),
        },
    };

    return new Promise((resolve, reject) => {
        const req = https.request(endpointUrl, options, (res) => {
            let data = '';
            res.on('data', (chunk) => {
                data += chunk;
            });
            res.on('end', () => {
                if (res.statusCode >= 200 && res.statusCode < 300) {
                    console.log('Successfully forwarded message:', data);
                    resolve({
                        statusCode: 200,
                        body: JSON.stringify('Message forwarded successfully!'),
                    });
                } else {
                    console.error('Error forwarding message:', res.statusCode, data);
                    reject(new Error(`Failed to forward message: ${res.statusCode} - ${data}`));
                }
            });
        });

        req.on('error', (e) => {
            console.error('Request error:', e.message);
            reject(new Error(`Request error: ${e.message}`));
        });

        req.write(requestBody);
        req.end();
    });
  

11. Now, go to Configuration tab > Permissions, click the Role name, and it will open a new browser tab:

12. On the Permissions tab, choose Add permissions > Attach policies:

13. On the Other permissions policy section, search for AWSLambdaVPCAccessExecutionRole, tick the check box of the policy and click Add permissions:

14. Now, go back to AWS Console Lambda and click your function. Then go to Configuration tab > VPC and click Edit:

15. On the Edit VPC page, choose your VPC, Subnet, and Security groups, then click Save:

Lambda Function is now integrated on VPC.