Skip to main content

Connect Astro to AWS data sources

Use the information provided here to learn how you can securely connect your Astro data plane to your existing AWS instance. A connection to AWS allows Astro to access data stored on your AWS instance and is a necessary step to running pipelines in a production environment.

Connection options​

The connection option that you choose is determined by the requirements of your organization and your existing infrastructure. You can choose a straightforward implementation, or a more complex implementation that provides enhanced data security. Astronomer recommends that you review all of the available connection options before selecting one for your organization.

Publicly accessible endpoints allow you to quickly connect Astro to AWS. To configure these endpoints, you can use one of the following methods:

When you use publicly accessible endpoints to connect Astro and AWS, traffic moves directly between your Astro data plane and the AWS API endpoint. Data in this traffic never reaches the control plane, which is managed by Astronomer.

Authorization options​

Authorization is the process of verifying a user or service's permissions before allowing them access to organizational applications and resources. Astro clusters must be authorized to access external resources from your cloud. Which authorization option that you choose is determined by the requirements of your organization and your existing infrastructure. Astronomer recommends that you review all of the available authorization options before selecting one for your organization.

To grant an Astro cluster access to a service that is running in an AWS account not managed by Astronomer, use AWS IAM roles. IAM roles on AWS are often used to manage the level of access a specific user, object, or group of users has to a resource. This includes an Amazon S3 bucket, Redshift instance, or secrets backend.

  1. In the Cloud UI, click Clusters and then copy the value displayed in the Cluster ID column for the Astro cluster that needs access to AWS service resources.

  2. Create an IAM role in the AWS account that contains your AWS service. See Creating a role to delegate permissions to an AWS service.

  3. In the AWS Management Console, go to the Identity and Access Management (IAM) dashboard.

  4. Click Roles and in the Role name column, select the role you created in step 2.

  5. Click the Trust relationships tab.

  6. Click Edit trust policy and update the arn value:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": [
                        "arn:aws:iam::<dataplane-AWS-account-ID>:role/AirflowS3Logs-<cluster-ID>"
                    ]
                },
                "Action": "sts:AssumeRole"
            }
        ]
    }

    To locate your <dataplane-AWS-account-ID> and <cluster-ID>, in the Cloud UI click Clusters. The <dataplane-AWS-account-ID> is located in the Account ID column and the cluster ID is located in the ID column.

    The Astro cluster data plane account includes the AirflowLogsS3-<clusterid> role. When you configure an Airflow connection for a Deployment, specify this role when you create the AWS Airflow Connection.

  7. Click Update policy.

  8. In the Airflow UI or as an environment variable on Astro, create an Airflow connection to AWS for each Deployment that requires the resources you connected. See Managing connections to Apache Airflow.

  9. Optional. Repeat these steps for each Astro cluster that requires access to external data services on AWS.

Hostname resolution options​

Securely connect your Astro data plane to resources running in other VPCs or on-premises through a resolving service.

Using Route 53 requires sharing a resolver rule with your Astro account. If this is a security concern, Astronomer recommends using Domain Name System (DNS) forwarding.

Use Route 53 Resolver rules to allow Astro to resolve DNS queries for resources running in other VPCs or on-premises.

Prerequisites​

  • An Amazon Route 53 Resolver rule. See Managing forwarding rules.
  • Permission to share resources using the AWS Resource Access Manager (RAM)

Share the Amazon Route 53 Resolver rule​

To allow Astro to access a private hosted zone, you need to share your Amazon Route 53 Resolver rule with your Astro AWS account.

  1. In the Route 53 Dashboard, click Rules below Resolver in the navigation menu.

  2. Select a Resolver rule and then click Details.

  3. Click Share and enter Astro in the Name field.

  4. In the Resources - optional section, select Resolver Rules in the Select resource type list and then select one or more rules.

  5. On the Associate permissions page, accept the default settings and then click Next.

  6. On the Grant access to principals page, select Allow sharing only within your organization, and then enter your Astro AWS account ID for your organization in the Enter an AWS account ID field. To get the Astro AWS account ID, go to the Cloud UI, click Settings, and then copy the value in the ID column for the Astro AWS account you want to share the Resolver rule with.

  7. Click Create resource share.

Contact Astronomer support for rule verification​

To verify that the Amazon Route 53 Resolver rule was shared correctly, submit a request to Astronomer support. With your request, include the Amazon Route 53 Resolver rule ID. To locate the Resolver rule ID, open the Route 53 Dashboard, and in the left menu click Rules below Resolver. Copy the value in the Resolver ID column.

Create a connection to confirm connectivity (optional)​

When Astronomer support confirms that the Amazon Route 53 Resolver rule was successfully associated with the Astro VPC, you can create a connection to the resource that is resolved by the shared rule. See Managing Connections.