Amazon Web Services (AWS)

Overview

Amazon Web Services (AWS) is an Amazon subsidiary that provides on-demand cloud computing platforms. AWS hosts numerous products and services, including the Aurora relational database, the Redshift data warehouse product, and the Simple Storage Service (S3).

Amazon S3

Amazon S3 (Simple Storage Service) is a cloud computing data service offering from Amazon Web Services (AWS). Amazon S3 provides storage through web services interfaces.

Accessing S3 buckets

Data Management can access S3 resources in two ways:

• Bucket policies grant access when the request originates from approved IP addresses or Amazon Virtual Private Cloud (Amazon VPC) endpoints.

Your AWS administrator must configure the S3 buckets to ensure that the servers running Data Management have access to the desired buckets.

• Account-based access, where access is granted when the correct credentials are presented and authenticated. This may happen automatically via the default credential provider chain if the credentials are present on the Data Management servers either as system environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY), or in the default credential profiles file (typically something like ~/.aws/credentials). You may also configure the credentials explicitly across the entire Data Management site or on a user-by-user basis.

You must have Administrator privileges to do this.

Once access to S3 resources is configured, any Data Management browse dialog that supports DFS paths will display a file system labeled S3 Buckets at the root level of the browse window. Expand that item to display a list of all Buckets accessible via the configured AWS credentials.

S3 has no append operation. Enabling the Append to existing file option in a Flat File Output tool or a CSV Output tool will result in an error.

Accessing S3 buckets owned by another user

Cross-account buckets are not listed in the File Browser's S3 Bucket file system. To access a cross-account bucket, you must know the bucket name and enter it when configuring tools, steps, or macros.

1. Add a tool to your canvas that you want to access another user's bucket.

2. Open the Properties panel of the tool.

3. Enter the URI of the bucket you want to access in the File Browser field.
   

   

4. Select the Browse button. You will see both your priority bucket and the entered bucket.
   

   

Configure access to S3 buckets

To configure site-level access to S3 buckets

1. In the repository, open the Settings folder.

2. Open the Cloud folder.

3. Select the AWS icon.

4. Go to the Properties pane.
   

   

5. On the S3 Settings tab, select Expose buckets.

6. If you are not authenticating automatically (via bucket policies or default credential provider chain), open the Authentication section and enter one or more Access key and Secret key pairs. (The Secret key may be a Password or Key Vault reference.) If you have multiple sets of credentials, enter the Access key and Secret key pairs in order of their priority. The first set of credentials with access to a bucket will be used.

You must have Administrator privileges to do this.

7. Optionally, select Encryption type and configure client-side S3 encryption.

To configure user-level access to S3 buckets

1. In the repository, open the Users folder and select the desired user.

2. On the User tab of the Properties pane, find the AWS/S3 access section, and then enter the Access key and Secret access key (which may be a Password or Key Vault reference). If Access key and Secret access key pairs are also defined at the site level, these user-level credentials will have priority.

You must have Administrator privileges to do this.

3. Optionally, select Encryption type and configure client-side S3 encryption.

To configure cross-account access to S3 buckets

Beginning in version 9.4.6, Data Management can access buckets owned by another AWS account if IAM policies are properly configured. When attempting to access a cross-account bucket, Data Management will use the first pair of Access keys and Secret keys configured in S3 Settings. If no keys are configured, Data Management will attempt to authenticate using the AWS Default Credential Provider Chain. Follow the steps in the “To configure site-level access to S3 buckets section”, and add multiple Access keys and Secret keys in the order of priority, placing the bucket you wish to access first.

Configure client-side S3 encryption

AWS S3 supports both server-side (at rest) and client side (in transit) data encryption. Server-side encryption is managed entirely by AWS, and requires no Data Management configuration. If you need to encrypt data before transmitting it to or from AWS S3, you must configure client-side encryption in Data Management.

To configure client-side S3 encryption

1. Configure access to S3 buckets.

2. Select Encryption type and select your desired encryption strategy using the table below.

mkdir awsKeys
cd awsKeys 
openssl rand 128 > sym_keyfile.key

mkdir awsKeys
cd awsKeys

openssl genpkey -algorithm RSA \
    -pkeyopt rsa_keygen_bits:2048 \
    -pkeyopt rsa_keygen_pubexp:65537 | \
  
openssl pkcs8 -topk8 -nocrypt -outform der > aws-s3-private-key.p8

openssl pkey -pubout -inform der -outform der \
    -in aws-s3-private-key.p8 \
    -out aws-s3-public-key.spki