Skip to main content

Install Astro on Azure

To complete the installation process, you'll:

  • Add the Astronomer Service Principal to your Azure Active Directory (Azure AD) instance.
  • Assign the Astronomer Service Principal an Owner role to your subscription.
  • Register Microsoft Azure features.

When you've completed the installation process, Astronomer support creates a cluster within your Azure subscription to host the resources and Apache Airflow components necessary to deploy DAGs and execute tasks.

For more information about managing Azure subscriptions with the Azure CLI, see How to manage Azure subscriptions with the Azure CLI.

Prerequisites

  • A clean Azure subscription. For security reasons, Azure subscriptions with existing tooling running aren't supported. Also, the subscription must be included in an Azure management group that doesn't apply Azure policies. See What are Azure management groups.
  • An Azure AD user with the following role assignments:
  • Microsoft Azure CLI or Azure Az PowerShell module. See How to install the Azure CLI and Install the Azure Az PowerShell module.
  • A minimum quota of 48 Standard Ddv5-series vCPUs in the deployment region. You can use Dv5-series vCPUs, but you'll need 96 total vCPUs composed of 48 Ddv5-series vCPUs and 48 Dv5-series vCPUs. To adjust your quota limits up or down, see Increase VM-family vCPU quotas.
  • A subscription to the Astro Status Page. This ensures that you're alerted when an incident occurs or when scheduled maintenance is planned.

For more information about the resources required to run Astro on Azure, see Azure Resource Reference.

Step 1: Access Astro

  1. Go to https://cloud.astronomer.io/ and create an Astronomer account.

  2. Go to https://cloud.astronomer.io/login, enter your email address, and then click Continue.

  3. Select one of the following options to access the Cloud UI:

    • Enter your password and click Continue.
    • To authenticate with an identity provider (IdP), click Continue with SSO, enter your username and password, and then click Sign In.
    • To authenticate with your GitHub account, click Continue with GitHub, enter your username or email address, enter your password, and then click Sign in.
    • To authenticate with your Google account, click Continue with Google, choose an account, enter your username and password, and then click Sign In.

    If you're the first person in an Organization to authenticate, you're added as a Workspace Admin to a new Workspace named after your Organization. You can add other team members to the Workspace without the assistance of Astronomer support. See Add a user. To integrate an identity provider (IdP) with Astro, see Set up an identity provider.

Step 2: Prepare for data plane activation

The data plane is a collection of Astro infrastructure components that run in your cloud and are managed by Astronomer. This includes a central database, storage for Airflow tasks logs, and the resources required for task execution.

  1. Run the following command to log in to your Azure account:

    az login
  2. Run the following command to select your Azure subscription:

    az account set -s <subscription-id>
  3. Run the following command to add the Astronomer Service Principal to Azure AD:

    az ad sp create --id a67e6057-7138-4f78-bbaf-fd9db7b8aab0
  4. Run the following commands to get details about the Azure subscription and create a new role assignment for the Astronomer service principal:

    subid=$(az account show --query id --output tsv)
    az role assignment create --assignee a67e6057-7138-4f78-bbaf-fd9db7b8aab0 --role Owner --scope /subscriptions/$subid
  5. Run the following commands to register the EncryptionAtHost feature:

    az feature register --namespace "Microsoft.Compute" --name "EncryptionAtHost"
    while [ $(az feature list --query "[?contains(name, 'Microsoft.Compute/EncryptionAtHost')].{State:properties.state}" -o tsv) != "Registered" ]
    do
    echo "Still waiting for Feature Registration (EncryptionAtHost) to complete, this can take up to 15 minutes"
    sleep 60
    done
    echo "Registration Complete"
    az provider register --namespace Microsoft.Compute

Step 3: Provide setup information to Astronomer support

After you've prepared your environment for data plane activation, provide Astronomer support with the following information:

  • Your preferred Astro cluster name.
  • Your Azure TenantID and SubscriptionID.
  • Optional. Your preferred node instance type. The default is Standard_D4d_v5.
  • Optional. Your preferred Postgres Flexible Server instance type. The default is Standard_D4ds_v4.
  • Optional. Your preferred maximum node count.
  • Optional. Your custom CIDR ranges for Astronomer service connections. The default is 172.20.0.0/19.

If you don't specify a preferred configuration for your organization, Astronomer support creates a cluster in CentralUS with the default configurations for Astro on Azure. See Azure resource reference.

Step 4: Astronomer support creates the cluster

After you provide Astronomer support with the setup information for your organization, Astronomer support creates your first cluster on Azure.

Wait for confirmation from Astronomer support that the cluster has been created before creating a Deployment.

Step 5: Create a Deployment

When Astronomer confirms that your Astro cluster has been created, you can create a Deployment and start deploying DAGs. Log in to the Cloud UI and create a new Deployment. If the installation is successful, your new Astro cluster is listed as an option in the Cloud UI Cluster list:

Cloud UI New Deployment screen

Next steps