Skip to main content

Create a New Cluster on Astro

Overview

The Astro install typically starts with 1 Cluster per Organization. Depending on your use case, however, your team can choose to configure multiple Astro Clusters. This could enable a few benefits, including:

  • Clusters in different regions
  • Different Clusters for development and production environments

Within a single Workspace, you can host Deployments across multiple Clusters. For example, you might have a production Deployment running in a production Cluster and a development Deployment running in a development Cluster. Both of those Deployments can be in the same Workspace.

This guide provides instructions for provisioning additional Clusters within your Astro Organization.

Prerequisites

To create an Astro Cluster, you need to have:

  • An activated Data Plane.
  • Permissions to configure IAM in the dedicated account for Astro on your cloud.

Step 1: Submit a Request to Astronomer

To create a new Astro Cluster on AWS for your Organization, you must first reach out to your Astronomer representative. For each new Cluster that you want to provision, provide our team with the following information:

  • Your AWS Account ID.
  • Your preferred Astro Cluster name.
  • The AWS region that you want to host your Cluster in.
  • Your preferred node instance type.
  • Your preferred max node count.

If not specified, Astronomer will create a Cluster with two m5.xlarge nodes and a maximum node count of 20 in us-east-1 by default. For information on all supported regions, configurations, and defaults, see AWS Resource Reference.

From there, your Astronomer representative will provide you with a unique External ID for each new Cluster. Make note of this value for the next step.

Step 2: Edit Your AWS Trust Policy

In the AWS IAM console, edit the astronomer-remote-management trust relationship to include new trust policies and ensure that Astronomer has permission to manage your new Cluster.

To do this, add the External ID that corresponds to your new Cluster to the existing sts:ExternalId grouping. For example, your policy for two Astro Clusters might look like the following, with <External-ID-2> being the External ID for your new Cluster:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::406882777402:root"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": [
"<External-ID-1>",
"<External-ID-2>"
]
}
}
}
]
}

Once you've modified your trust policy, click Update Trust Policy in the AWS Console to apply the new trust relationship.

Additional Setup for AWS Regions that are Disabled by Default

Some AWS regions that Astronomer supports are disabled by default on AWS. These regions are:

  • af-south-1 - Africa (Cape Town)
  • ap-east-1 - Asia Pacific (Hong Kong)
  • me-south-1 - Middle East (Bahrain)

To create a cluster in one of these regions, complete the following additional setup in your AWS account:

  1. In the AWS IAM console, update the astronomer-remote-management trust relationship to include permissions for enabling and disabling your desired region as described in the AWS Documentation:

    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Principal": {
    "AWS": "arn:aws:iam::406882777402:root"
    },
    "Action": "sts:AssumeRole",
    "Condition": {
    "StringEquals": {
    "sts:ExternalId": [
    "<External-ID-1>",
    "<External-ID-2>"
    ]
    }
    }
    }
    {
    "Sid": "EnableDisableRegion",
    "Effect": "Allow",
    "Action": [
    "account:EnableRegion",
    "account:DisableRegion"
    ],
    "Resource": "*",
    "Condition": {
    "StringEquals": {"account:TargetRegion": "<your-aws-region>"}
    }
    },
    {
    "Sid": "ViewConsole",
    "Effect": "Allow",
    "Action": [
    "aws-portal:ViewAccount",
    "account:ListRegions"
    ],
    "Resource": "*"
    }
    ]
    }
  2. In the AWS Management Console, enable the desired region as described in AWS documentation.

  3. Upgrade your global endpoint session token to version 2, which is valid in all AWS regions, by running the following command via the AWS CLI:

    aws iam set-security-token-service-preferences --global-endpoint-token-version v2Token

Step 3: Confirm with Astronomer

Astronomer will notify you once your Cluster has been created. Once your Cluster is available, you should be able to configure a new Deployment within that Cluster.