DynamoDB - data pipeline

user-pcr user-pcr
Mar 16, 2022 - 17:59
0 8
DynamoDB - data pipeline

The Data Pipeline allows you to export and import data to/from a table, file, or S3 bucket. This of course proves useful for backups, testing, and similar needs or scenarios.

In the export, you use the Data Pipeline console, which creates a new pipeline and starts an Amazon EMR (Elastic MapReduce) cluster to run the export. EMR reads data from DynamoDB and writes to the target. We will discuss EMR in detail later in this tutorial.

In an import operation, you use the Data Pipeline console, which creates a pipeline and starts the EMR to perform the import. It reads data from the source and writes to the destination.

Note. Export / import operations incur costs taking into account the services used, in particular EMR and S3.

Using the Data Pipeline

You must specify the action and resource permissions when using the data pipeline. You can use an IAM role or policy to define them. Users who perform import/export should note that they will need an active access key ID and a secret key.

IAM Roles for Data Pipeline

Two IAM roles are required to use the data pipeline −

  • DataPipelineDefaultRole - contains all the actions that you allow the pipeline to perform for you.

  • DataPipelineDefaultResourceRole - This is where you allow resources to be given to the pipeline.

DataPipelineDefaultRole - contains all the actions that you allow the pipeline to perform for you.

DataPipelineDefaultResourceRole - This is where you allow resources to be given to the pipeline.

If you are new to Data Pipeline, you should spawn each role. All previous users have these roles because of the existing roles.

Use the IAM console to create IAM roles for the data pipeline and complete the following four steps:

Step 1: Log in to the IAM console located at https://console.aws.amazon.com/iam/.

Step 2 − Select Roles from the toolbar.

Step 3 − Select Create New Role . Then enter DataPipelineDefaultRole in the Role Name field and select Next Step . In the AWS Service Roles list in the Role Type panel , navigate to the data pipeline and select Select . Select Create Role from the Browse panel .

Step 4 − Select Create New Role .