Skip to main content

Install Astro on AWS

This is where you'll find instructions for completing the Astro installation process, including prerequisites and the steps required for our team to provision resources in your network.

At a high-level, we'll ask that you come prepared with a new AWS account. From there, you can expect to:

  • Create an account on Astro.
  • Share AWS account information with our team.
  • Create a cross-account IAM role that Astro can assume within your new AWS account.

Astronomer will then create a cluster within your AWS account that hosts the resources and Apache Airflow components necessary to deploy DAGs and execute tasks. If you'd like to support more than 1 Astro cluster, contact Astronomer support.

For a complete list of the AWS resources that our team will provision in your AWS account, see Resource usage.

Prerequisites

Before completing this setup, make sure that you have:

  • A dedicated AWS account with minimum EC2 service quotas.

  • An AWS IAM user with the following permissions:

  • A subscription to the Astro Status Page. This will ensure that you're alerted in the case of an incident or scheduled maintenance.

Astro requires a clean AWS account with a minimum set of EC2 service quotas. For security reasons, the install process is not currently supported on an AWS account that has other tooling running in it. For instructions on creating a new AWS account, follow AWS documentation.

The required EC2 service quotas are:

QuotaCodeQuotaNameMinimum Value
L-1216C47ARunning On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances40
L-34B43A08All Standard (A, C, D, H, I, M, R, T, Z) Spot Instance Requests40

These are required to mitigate near term capacity risks and ensure a smooth onboarding experience on Astro. If you need to modify or increase a specific quota, see Amazon’s documentation on requesting a quota increase.

Once your AWS account is created, proceed to Step 1.

tip

If you have one or more existing AWS accounts, you can use AWS Organizations to manage billing, users, and more in a central place. For more information on how to add your Astro AWS account to your AWS Organization, read Amazon's documentation.

VPC peering prerequisites (Optional)

If any AWS resources are on a private network, you can choose between two options:

  • Allow traffic via the public internet and use allow-lists for communication.
  • Create a VPC Peering connection between Astronomer's VPC and the VPCs for your broader network.

If you want to continue with the second option, you'll additionally need:

  • A CIDR block (RFC 1918 IP Space) no smaller than a /19 range. You must ensure it does not overlap with the AWS VPC(s) that you will be peering with later. The default CIDR range is 172.20.0.0/19.
  • VPC Name / ID for peering with Astronomer (accessible through the AWS VPC console).
  • The IP addresses of your DNS servers.

Step 1: Access Astro

To begin the Astro install process, first create an account at https://cloud.astronomer.io/.

When you first authenticate to Astro, you can sign in to the Cloud UI with a Google account, a GitHub account, or an email and password.

Astro login screen

If you're the first person from your team to authenticate, the Astronomer team will add you as a Workspace Admin to a new Workspace named after your Organization. From there, you'll be able to add other team members to that Workspace without Astronomer's assistance.

tip

After completing your initial installation, we recommend setting up an identity provider (IdP) so that users can log in to Astro through your IdP.

Step 2: Retrieve an external ID from the Cloud UI

In the Cloud UI, open the Settings tab and copy the value in AWS EXTERNAL ID. This external ID is a unique identifier that Astro uses to connect to your AWS account. Save the external ID as a secret or in another secure format. See How to use an external ID when granting access to your AWS resources to a third party.

You must be an Organization Owner to view the external ID. If you are not an Organization Owner, the field will not appear in the Cloud UI.

Step 3: Create a cross-account IAM role for Astro

Use the external ID you saved from Step 2 to create a cross-account IAM role for Astro. Astronomer recommends using the AWS Management Console to create the role.

  1. Open the Astronomer cross-account role CloudFormation template.

  2. Enter the external ID that you copied in Step 2 in the ExternalId field.

  3. Select the I acknowledge that AWS CloudFormation might create IAM resources with custom names checkbox.

  4. Click Create Stack.

To provision additional Clusters after completing your initial installation, see Create a Cluster.

caution

Some AWS regions that Astronomer supports are disabled by default on AWS, including:

  • ap-east-1 - Asia Pacific (Hong Kong)
  • me-south-1 - Middle East (Bahrain)

If you're setting up your first cluster in any of these regions, you need to complete the additional setup described in Create a cluster.

Step 4: Provide setup information to Astronomer

After creating the AWS account, provide Astronomer support with the following information:

  • Your AWS Account ID.
  • Your preferred Astro cluster name.
  • The AWS region that you want to host your cluster in.
  • Your preferred node instance type.
  • Your preferred maximum node count.

If you do not specify configuration preferences, Astronomer will create a cluster with two m5.xlarge nodes and a maximum node count of 20 in us-east-1. For information on all supported regions, configurations, and defaults, see AWS resource reference.

Step 5: Wait for Astronomer to complete the install

After you've created the cross-account IAM role for Astro, contact Astronomer support. Astronomer support will finish creating the cluster in your AWS account.

This process can take some time. Wait for confirmation that the installation was successful before proceeding to the next step.

Note: If you need to VPC peer with Astronomer, additionally provide the following information to your Astronomer representative:

  • Subnet CIDRs (RFC 1918 IP Space).
  • VPC Name/ID and region for peering with Astronomer. This is accessible through the AWS VPC console.
  • The IPs of your DNS servers.

You then need to accept a VPC peering request from Astronomer after Astro is installed. To accept the request, follow Creating and accepting a VPC peering connection in AWS documentation.

Once VPC peered with Astronomer, configure and validate the following to ensure successful network communications between Astro and your resources:

Step 6: Create a Deployment

When Astronomer confirms that your Astro cluster has been created, you are ready to create a Deployment and start deploying DAGs. Log in to the Cloud UI again and create a new Deployment. If the installation is successful, your new Astro cluster is listed as an option below the Cluster menu:

Cloud UI New Deployment screen

Next steps

Now that you have an Astro cluster up and running, take a look at the docs below for information on how to start working in Astro: