Skip to main content

Install Astro on AWS using a cross-account role

caution

This feature is in Private Preview and is an alternative to the standard AWS installation process. For the standard installation, see Install Astro on AWS

To install Astro in a dedicated AWS account owned by your organization, you'll complete the following tasks:

  • Create an account on Astro.
  • Share AWS account information with Astronomer support.
  • Create a cross-account IAM role that Astro can assume within your new AWS account.

Astronomer support will create a cluster within your AWS account to host the resources and Airflow components necessary to deploy DAGs and execute tasks. If you need more than one Astro cluster, contact Astronomer support.

Prerequisites

  • A dedicated AWS account with the minimum EC2 service quotas.
  • A subscription to the Astro Status Page. This ensures that you'll be notified when there are incidents or when maintenance is scheduled.
  • The following domains added to your organization's allowlist for any user and CI/CD environments:
    • https://cloud.astronomer.io/
    • https://astro-<your-org>.datakin.com/
    • https://<your-org>.astronomer.run/
    • https://api.astronomer.io/
    • https://images.astronomer.cloud/
    • https://auth.astronomer.io/
    • https://updates.astronomer.io/
    • https://install.astronomer.io/

Astro requires a clean AWS account with the minimum EC2 service quotas. For security reasons, the install process is not currently supported on an AWS account that has other tooling running in it. For instructions on creating a new AWS account, see How do I create and activate a new AWS account?.

The following table lists the required EC2 service quotas.

QuotaCodeQuotaNameMinimum Value
L-1216C47ARunning On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances40
L-34B43A08All Standard (A, C, D, H, I, M, R, T, Z) Spot Instance Requests40

These quotas are required to mitigate near term capacity risks and simplify the Astro onboarding experience. If you need to modify or increase a specific quota, see Request a quota increase.

tip

If you have one or more existing AWS accounts, you can use AWS Organizations to manage billing, users, and more in a central place. For more information on how to add your Astro AWS account to your AWS Organization, see Inviting an AWS account to join your organization.

VPC peering prerequisites (Optional)

The following options are available when your AWS resources are on a private network:

  • Allow traffic through the public internet and use allow-lists for communication.
  • Create a VPC Peering connection between the Astronomer VPC and the VPCs for your network.

If you want to continue with the second option, you'll additionally need:

  • A CIDR block (RFC 1918 IP Space) no smaller than a /19 range. You must ensure it does not overlap with the AWS VPC(s) that you will be peering with later. The default CIDR range is 172.20.0.0/19.
  • VPC Name / ID for peering with Astronomer (accessible through the AWS VPC console).
  • The IP addresses of your DNS servers.

Step 1: Access Astro

  1. Go to https://cloud.astronomer.io/ and create an account, or enter your email address, and then click Continue.

  2. Select one of the following options to access the Cloud UI:

    • Enter your password and click Continue.
    • To authenticate with an identity provider (IdP), click Continue with SSO, enter your username and password, and then click Sign In.
    • To authenticate with your GitHub account, click Continue with GitHub, enter your username or email address, enter your password, and then click Sign in.
    • To authenticate with your Google account, click Continue with Google, choose an account, enter your username and password, and then click Sign In.

    If you're the first person in your Organization to authenticate, you'll be granted Organization owner permissions. You can create a Workspace and add other team members to the Workspace without the assistance of Astronomer support. See Create a Workspace and Add a user. To integrate an identity provider (IdP) with Astro, see Set up an identity provider.

Step 2: Retrieve an external ID from the Cloud UI

You must be an Organization Owner to view the external ID. If you are not an Organization Owner, the AWS External ID field will not appear in the Cloud UI.

  1. In the Cloud UI, click the Settings tab.

  2. Click Show in the AWS External ID field and then click Copy. This external ID is a unique identifier that Astro uses to connect to your AWS account.

  3. Save the external ID as a secret or in another secure format. See How to use an external ID when granting access to your AWS resources to a third party.

Step 3: Create a cross-account role

Use the external ID to create a cross-account IAM role for Astro. Astronomer recommends using the AWS Management Console to create the role.

  1. Open the Astronomer cross-account role CloudFormation template.

  2. Enter the external ID that you copied in Step 2 in the ExternalId field.

  3. Select the I acknowledge that AWS CloudFormation might create IAM resources with custom names checkbox.

  4. Click Create Stack.

Notification of changes to the cross-account role

Occasionally, Astronomer makes changes to its policies to ensure the continued operation and development of Astro.

Users with an Organization Owner role will receive an email notification from Astronomer 14 days before any changes are made to the policies governing the cross-account IAM role that expands user access. Notifications will include an explanation of the changes being made and why the change was necessary.

Astronomer can reduce the access available to the cross-account role without notification.

Monitor the cross-account role for changes (optional)

You can use CloudTrail to monitor changes to policies within the cross-account role. Access to CloudTrail has been limited to prevent the accidental modification or deletion of CloudTrail logs by Astronomer support. The following table lists the events that you should monitor.

Event NamesResource
AttachRolePolicy , DetachRolePolicyroleName = astronomer-remote-management
SetPolicyVersion , CreatePolicyVersionpolicyArn = "arn:aws:iam::*:policy/AstronomerCrossAccountRole"

To monitor changes to the cross-account role policy, create an Amazon CloudWatch alarm. See Creating CloudWatch alarms for CloudTrail events. When you create the metric filter in the CloudWatch console, on the Define pattern page, in Create filter pattern, enter the following for Filter pattern:

{ ($.eventName = AttachRolePolicy || $.eventName = DetachRolePolicy || $.eventName = SetPolicyVersion || $.eventName = CreatePolicyVersion) && ($.requestParameters.policyArn = "*AstronomerCrossAccountRole"  || $.requestParameters.roleName = astronomer-remote-management) }

Step 4: Provide setup information to Astronomer

After creating the AWS account, provide Astronomer support with the following information:

  • Your AWS Account ID.
  • Your preferred Astro cluster name.
  • The AWS region that you want to host your cluster in.
  • Your preferred node instance type.
  • Your preferred maximum node count.

If you do not specify configuration preferences, Astronomer creates a cluster with m5.xlarge nodes and a maximum node count of 20 in us-east-1. For information on all supported regions, configurations, and defaults, see AWS cluster configurations.

To provision additional clusters after completing your initial installation, see Create a cluster.

caution

Some AWS regions that Astronomer supports are disabled by default on AWS, including:

  • ap-east-1 - Asia Pacific (Hong Kong)
  • me-south-1 - Middle East (Bahrain)

If you're setting up your first cluster in any of these regions, you need to complete the additional setup described in Create a cluster.

Provide VPC peering information (optional)

If you need to VPC peer with Astronomer, provide the following information to your Astronomer representative:

  • Subnet CIDRs (RFC 1918 IP Space).
  • VPC Name/ID and region for peering with Astronomer. This is accessible through the AWS VPC console.
  • The IPs of your DNS servers.

Step 5: Astronomer support creates the cluster

After you've created the cross-account IAM role for Astro, contact Astronomer support. Astronomer support creates the cluster in your AWS account.

This process can take some time. Wait for confirmation from Astronomer support that the cluster has been created before creating a Deployment.

If you submitted a VPC peering request, you'll need to accept the request from Astronomer after Astro is installed. To accept the request, see Create a VPC peering connection.

When VPC peering with Astronomer is complete, configure and validate the following items to ensure successful network communications between Astro and your resources:

Step 6: Create a Deployment and confirm the install

When Astronomer support confirms that your Astro cluster has been created, Astronomer recommends that you create a Deployment and deploy DAGs. See Create a Deployment.

To confirm a successful installation, in the Cloud UI select a Workspace and on the Deployments page click Deployment. The Astro cluster created by Astronomer support appears as an option in the Cluster list.

Next steps