Queue provider: Kafka - AWS MSK

Overview

Full details of the parallel steps to be undertaken within the RPI client application’s Configuration Workbench can be found in the RPI reference guide.

Setting up a Kafka instance in AWS

This section describes how to setup your Kafka instance in AWS. Please follow the steps below:

1. On your web browser, navigate to https://console.aws.amazon.com.

2. Create or log in to an AWS account.

3. Navigate to Amazon MSK service and select Create cluster.

4. Follow the instructions provided on the page.

5. Once the Cluster is created, select the cluster name.

6. Go to the Client Integration Information page to review the details.

7. In the Plaintext section, select Copy link.

8. Go to the Kafka Queue Provider Config page in Redpoint Interaction (RPI) and paste the copied link n the Bootstrap servers field.

Setting up AWS MSK

This section describes how to setup AWS MSK instance.

Create a MSK Cluster with IAM Auth

1. Log in to AWS Console.

2. Go to MSK service and select Create cluster.

3. For creation method, select Custom create.

4. Enter cluster name.

5. For cluster type, select Provisioned.

6. For the Apache Kafka version, select latest (recommended).

7. In Configuration Custom configuration and Create configuration.

auto.create.topics.enable=true
default.replication.factor=2
min.insync.replicas=2
num.io.threads=8
num.network.threads=5
num.partitions=1
num.replica.fetchers=2
replica.lag.time.max.ms=30000
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
unclean.leader.election.enable=true
zookeeper.session.timeout.ms=18000

8. In Networking, select your VPC, Public subnets, and security group. Select Next.

9. In Security, choose IAM role-based authentication and select Next.

10. After reviewing, select Create cluster.

Enable public access

1. Go to cluster properties.

2. Choose Edit Networking settings.

3. Select Enable Public Access.

Create IAM policy

1. Go to IAM service and choose Policies.

2. Select Create policy.

3. In Create policy, select JSON.

4. Paste the sample policy included below.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:CreateTopic",
                "kafka-cluster:DescribeCluster",
                "kafka-cluster:AlterTopic",
                "kafka-cluster:ReadData",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:DescribeGroup",
                "kafka-cluster:Connect",
                "kafka-cluster:WriteData"
            ],
            "Resource": [
                "arn:aws:kafka:*:656810741044:group/*/*/*",
                "arn:aws:kafka:*:656810741044:cluster/*/*",
                "arn:aws:kafka:*:656810741044:topic/*/*/*"
            ]
        }
    ]
}

5. Add tags.

6. Review your work and select Create policy.

Create IAM User

1. In IAM service, select Users.

2. Choose Add Users.

3. Enter the username.

4. Select Access key - Programmatic access.

5. Under Set permissions, select Attach existing policies directly.

6. Search for the policy you created and select the policy.

7. Select Next.

8. Add tags.

9. Review your work and select Create user.

Note: Don’t forget to download and save credentials. Use the Access Key and Secret Key to configure the Kafka Queue provided in Redpoint Interaction

Get Bootstrap Servers

1. Open the Amazon MSK console.

2. Choose the name of a cluster to view its description. The table shows all the clusters for the current region under this account.

3. On the Cluster summary page, choose View client information.