Skip to main content

Don't miss the biggest Airflow event of the year: Airflow Summit 2023, Sept 19-21

Join Us →

Connect Astro to AWS data sources

Use this document to learn how you can connect an Astro cluster and its Deployments to your external AWS resources.

Connection options​

Publicly accessible endpoints allow you to quickly connect your Astro clusters or Deployments to AWS through an Airflow connection. If your cloud restricts IP addresses, you can add the external IPs of your Deployment or cluster to an AWS resource's allowlist.

If you have stricter security requirements, you can create a private connection to AWS in a few different ways.

After you crate a connection from your Deployment to AWS, you might also have to individually authorize Deployments to access specific resources. See Authorize your Deployment using workload identity.

Access a public AWS endpoint​

To facilitate communication between your Astro cluster or Deployment and your cloud, you can allowlist the external IPs for your cluster or Deployment in your cloud. If you have no other security restrictions, this means that any Deployment or cluster with an allowlisted external IP address can access your AWS resources through a valid Airflow connection.

Allowlist external IP addresses for a cluster​

  1. In the Cloud UI, click your Workspace name in the upper left corner, then click Organization Settings.
  2. Click Clusters, then select a cluster.
  3. In the Details page, copy the IP addresses listed under External IPs.
  4. Add the IP addresses to the allowlist of any external services that you want your cluster to access.

After you allowlist a cluster's IP address, all Deployments in that cluster are allowed to access your AWS resources.

Allowlist external IP addresses for a Deployment​

To grant access to your external resources on per-Deployment basis, or if you are using a standard cluster, allowlist the IPs only for specific Deployments. For each Deployment that you want to allowlist:

  1. In the Cloud UI, select a Deployment, then click Details.
  2. Copy the IP addresses under External IPs.
  3. Add the IP addresses to the allowlist of any external services that you want your Deployment to access.

When you use publicly accessible endpoints to connect to AWS, traffic moves directly between your Astro cluster and the AWS API endpoint. Data in this traffic never reaches the Astronomer-managed control plane.

Create a private connection between Astro and AWS​

Choose one of the following setups based on the security requirements of your company and your existing infrastructure.

info

This connection option is only available for dedicated Astro Hosted clusters and Astro Hybrid.

To set up a private connection between an Astro VPC and an AWS VPC, you can create a VPC peering connection. VPC peering ensures private and secure connectivity, reduces network transit costs, and simplifies network layouts.

To create a VPC peering connection between an Astro VPC and an AWS VPC, you must create a temporary assumable role. The Astro AWS account will assume this role to initiate a VPC peering connection.

  1. Open the AWS console of the AWS account with the external VPC and copy the following:

    • AWS account ID
    • AWS region
    • VPC ID of the external VPC
    • CIDR block of the external VPC

  2. Create a temporary role using the role creation stack template. In the Quick create stack template that opens, complete the following fields:

    • Stack name: Enter a meaningful name for your stack.
    • Peer Owner IDs: Enter your cluster's AWS account ID. To retrieve your cluster's AWS account ID on Astro Hosted, contact Astronomer support. To retrieve your cluster's AWS account ID on Astro Hybrid, click the name of your Workspace in the upper left corner of the Cloud UI, click Organization Settings, then click Clusters. Open your cluster and copy its Account ID.

  3. After the stack is created, go to the Stack info tab and copy the AssumeRole ARN from the Stack ID field.

  4. In the Cloud UI, click your Workspace name in the upper left corner, then click Organization Settings. Click Clusters, select your cluster, and copy the ID of the cluster.

  5. Contact Astronomer support and provide the following details:

    • AWS region of the external VPC from Step 1
    • VPC ID of the external VPC from Step 1
    • AWS account ID of the external VPC from Step 1
    • CIDR block of the external VPC from Step 1
    • Stack ID from Step 3
    • Astro cluster ID from Step 4

    Astronomer support will initiate a peering request and create the routing table entries in the Astro VPC.

  6. Wait for Astronomer support to send you the Astro VPC CIDR and VPC peering ID. Then, the owner of the external VPC needs to add a route in the external VPC, using the Astro VPC CIDR as the Destination and the VPC peering ID as the Target.

  7. (Optional) Delete the stack that you created. This will delete the temporary assumable role.

DNS considerations for VPC peering​

To resolve DNS hostnames from your external VPC, every Astro VPC has DNS Hostnames, DNS Resolutions, and Requester DNS Resolution enabled. See AWS Peering Connection settings.

If your external VPC resolves DNS hostnames using DNS Hostnames and DNS Resolution, you must also enable the Accepter DNS Resolution setting on AWS. This allows Astro clusters to resolve the public DNS hostnames of the external VPC to its private IP addresses. To configure this option, see AWS Documentation.

If your external VPC resolves DNS hostnames using private hosted zones, then you must associate your Route53 private hosted zone with the Astro VPC using instructions provided in AWS Documentation.

To retrieve the ID of any Astro VPC, contact Astronomer support. If you have more than one Astro cluster, request the VPC ID of each cluster.

Hostname resolution options​

Securely connect Astro to resources running in other VPCs or on-premises through a resolving service.

Using Route 53 requires sharing a resolver rule with your Astro account. If this is a security concern, Astronomer recommends using Domain Name System (DNS) forwarding.

Use Route 53 Resolver rules to allow Astro to resolve DNS queries for resources running in other VPCs or on-premises.

Prerequisites​

  • An Amazon Route 53 Resolver rule. See Managing forwarding rules.
  • Permission to share resources using the AWS Resource Access Manager (RAM)

Share the Amazon Route 53 Resolver rule​

To allow Astro to access a private hosted zone, you need to share your Amazon Route 53 Resolver rule with your Astro AWS account.

  1. In the Route 53 Dashboard, click Rules below Resolver in the navigation menu.

  2. Select a Resolver rule and then click Details.

  3. Click Share and enter Astro in the Name field.

  4. In the Resources - optional section, select Resolver Rules in the Select resource type list and then select one or more rules.

  5. On the Associate permissions page, accept the default settings and then click Next.

  6. On the Grant access to principals page, select Allow sharing only within your organization, and then enter your Astro AWS account ID for your organization in the Enter an AWS account ID field.

    To get the Astro AWS account ID, click the name of your Workspace in the upper left corner of the Cloud UI, then click Organization Settings. From the General page, copy the AWS External ID.

  7. Click Create resource share.

Contact Astronomer support for rule verification​

To verify that the Amazon Route 53 Resolver rule was shared correctly, submit a request to Astronomer support. With your request, include the Amazon Route 53 Resolver rule ID. To locate the Resolver rule ID, open the Route 53 Dashboard, and in the left menu click Rules below Resolver. Copy the value in the Resolver ID column.

Create a connection to confirm connectivity (optional)​

When Astronomer support confirms that the Amazon Route 53 Resolver rule was successfully associated with the Astro VPC, you can create a connection to the resource that is resolved by the shared rule. See Managing Connections.

See Also​

Was this page helpful?

Sign up for Developer Updates

Get a summary of new Astro features once a month.

You can unsubscribe at any time.
By proceeding you agree to our Privacy Policy, our Website Terms and to receive emails from Astronomer.