Authorize an Astro Deployment to cloud resources using workload identity

When you create an Airflow connection from a Deployment to access cloud resources, Airflow uses your connection details to access those services. You can add credentials to your Airflow connections to authenticate, but it can be risky to add secrets like passwords to your Airflow environment.

To avoid adding secrets to your Airflow connection, you can directly authorize your Astro Deployment to access AWS or GCP cloud services using workload identity. Astronomer recommends using a workload identity in most cases to improve security and avoid managing credentials across your Deployments. If you have less strict security requirements, you can still use any of the methods described in Airflow connection guides to manage your connection authorization.

This guide explains how to authorize your Deployment to a cloud using workload identity. For each Deployment, you will:

• Authorize your Deployment to your cloud services.
• Create an Airflow connection to access your cloud services.

tip

Watch the Astro Academy Customer Workload Managed Identity Learning Byte video to learn more about managed identities and how to set up passwordless authentication for GCP.

Prerequisites

The Astro cluster running your Deployment must be connected to your cloud's network. See Networking overview.

What is workload identity?

A workload identity is a Kubernetes service account that provides an identity to your Deployment. The Deployment can use this identity to authenticate to a cloud's API server, and the cloud can use this identity to authorize the Deployment to access different resources.

Setup

AWS

Step 1: Authorize the Deployment in your cloud

To grant a Deployment access to a service that is running in an AWS account not managed by Astronomer, use AWS IAM roles to authorize your Deployment's workload identity. IAM roles on AWS are often used to manage the level of access a specific user, object, or group of users has to a resource, such as Amazon S3 buckets, Redshift instances, and secrets backends.

To authorize your Deployment, create an IAM role that is assumed by the Deployment's workload identity:

1. In the Astro UI, select your Deployment and then click Details. Copy the Deployment's Workload Identity.

2. In the AWS account that contains your AWS service, create an IAM role. See Creating a role to delegate permissions to an AWS service.

3. In the AWS Management Console, go to the Identity and Access Management (IAM) dashboard.

4. Click Roles and in the Role name column, select the role you created in Step 2.

5. Click Trust relationships.

6. Click Edit trust policy and paste the workload identity you copied from Step 1 in the trust policy. Your policy should look like the following:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "AWS": [
                       "<workload-identity-role>"
                   ]
               },
               "Action": "sts:AssumeRole"
           }
       ]
   }

7. Click Update policy.

Repeat these steps for each Astro Deployment that needs to access your AWS resources.

Step 2: Create an Airflow connection

Now that your Deployment is authorized, you can connect it to your cloud using an Airflow connection. Either create an Amazon Web Services connection in the Astro UI or the Airflow UI for your Deployment and specify the following fields:

• Connection Id: Enter a name for the connection.

• Extra:

  {
  "role_arn": "<your-role-arn>",
  "region_name": "<your-region>"
  }

If you don't see Amazon Web Services as a connection type in the Airflow UI, ensure you have installed its provider package in your Astro project's requirements.txt file. See Use Provider in the Astronomer Registry for the latest package.

GCP

Attach a service account to your Deployment

You can attach a custom GCP service account to your Deployment to grant the Deployment all of the service account's permissions.

Using service accounts provides the greatest amount of flexibility for authorizing Deployments to your cloud. For example, you can use existing service accounts on new Deployments, or your can attach a single service account to multiple Deployments that all have the same level of access to your cloud.

1. Create a service account in the GCP project that you want your Deployment to access. Grant the service account any permissions that the Deployment will need in your GCP project. Copy the service account ID to use later in this setup.

2. In the Cloud UI, select your Deployment, then click Details. In the Advanced section, click Edit.

3. In the Workload Identity menu, select Customer Managed Identity

4. Enter your GCP service account ID when prompted, then copy and run the provided gcloud CLI command.

5. Click Update Deployment. The service account is now selectable as a workload identity for the Deployment.

6. Complete one of the following options for your Deployment to access your cloud resources:

   • Create a Google Cloud connection type in Airflow and configure the following values:

     • Connection Id: Enter a name for the connection.
     • Impersonation Chain: Enter the ID of the service account that your Deployment should impersonate.

   • To access resources in a secrets backend, run the following command to create an environment variable that grants access to the secrets backend:

   astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "project_id": "<your-secret-manager-project-id>", "impersonation_chain": "<your-gcp-service-account>"}

Alternative setup: Authorize your Deployment through GCP service account impersonation

If your organization has requirements over how service accounts are managed outside of your cloud, you can manually configure GCP service account impersonation to allow your Deployment's default workload identity to impersonate a service account in your GCP project.

1. Create a service account in the GCP project that you want your Deployment to access. Grant the service account any permissions that the Deployment will need in your GCP project. Copy the service account ID to use later in this setup.

2. In the Astro UI, select your Deployment, then click Details. Copy the Deployment's Workload Identity.

3. In the Google Cloud Console, open the IAM & Admin > Service Accounts menu, then open the service account you just created.

4. In the Actions column, click Manage Permissions, then click Grant Access. In the modal that appears, enter your Deployment's workload identity service account in the Add Principals field and select the Service Account Token Creator in the Assign Roles field.

5. Complete one of the following options for your Deployment to access your cloud resources:

   • Create a Google Cloud connection type in Airflow and configure the following values:
     • Connection Id: Enter a name for the connection.
     • Impersonation Chain: Enter the ID of the service account that your Deployment should impersonate.

   Note that this implementation requires apache-airflow-providers-google >= 10.8.0. See Add Python, OS-level packages, and Airflow providers.

   • Specify the impersonation chain in code when you instantiate a Google Cloud operator. See Airflow documentation. Note that if you configure both a connection type and an operator, the operator-level configuration takes precedence.
   • To access resources in a secrets backend, run the following command to create an environment variable that grants access to the secrets backend:

   astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "project_id": "<your-secret-manager-project-id>", "impersonation_chain": "<your-gcp-service-account>"}

Alternative setup: Grant an IAM role to your Deployment workload identity

Complete this alternative setup if you don't have an existing Google service account that your Deployment workload identity can impersonate.

Step 1: Authorize the Deployment in your cloud

To grant a Deployment access to a service that is running in a GCP account not managed by Astronomer, use your Deployment's workload identity. Workload identity is a service account in GCP that's used to manage the level of access for a specific user, object, or group of users to a resource, such as Google BigQuery or a GCS bucket.

To authorize your Deployment, grant the required access to your Deployment's workload identity:

1. In the Astro UI, select your Deployment, then click Details. In the Workload Identity dropdown menu, select Default Identity. Then, copy the workload identity that appears next to the dropdown menu.

2. Grant your Deployment's workload identity an IAM role that has access to your external data service. To do this with the Google Cloud CLI, run:

   gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT --member=serviceAccount:<workload-identity> --role=roles/viewer

   To grant your workload identity an IAM role using the Google Cloud console, see Grant an IAM role.

Repeat these steps for each Deployment that needs to access your GCP resources.

Step 2: Create an Airflow connection

Now that your Deployment is authorized, you can connect it to your cloud using an Airflow connection. Either create a Google Cloud connection in the Astro UI or the Airflow UI for your Deployment and specify the following fields:

• Connection Id: Enter a name for the connection.
• Project Id: Enter the ID of your Google Cloud Project where your services are running.

If you don't see Google Cloud as a connection type in the Airflow UI, ensure you have installed its provider package in your Astro project's requirements.txt file. See Use Provider in the Astronomer Registry for the latest package.

Azure

In this setup, you'll authorize an existing user-assigned managed identity to a resource on Azure, then give permissions to your Deployment to assume that managed identity.

Prerequisites

• A Microsoft Entra ID tenant with Global Administrator or Application Administrator privileges.
• A user-assigned managed identity on Azure. See Azure documentation.
• The Azure CLI.

caution

You can only use the same user-assigned managed identity for up to five Deployments. If you need to authorize more than five Deployments to Azure, you need to create more than one user-managed identity. For more information, see Microsoft Entra documentation.

Step 1: Authorize the managed identity in Azure

1. In your Azure portal, open the resource that your managed identity needs access to. Then, select Access control (IAM).
2. Click Add > Add role assignment.
3. Select the role for your managed identity, then click Next.
4. In the Assign access to section, select Managed identity. Click + Select Members and choose your managed identity. After you add your managed identity, click Next.
5. Review and finalize the assignment.

Step 2: Configure your Deployment

1. In your Azure portal, open the Managed Identities menu.
2. Search for your managed identity, click Properties, then copy its Name, Client ID, Tenant ID, and Resource group name.
3. In the Astro UI, select your Deployment, click Details, then click How to Configure... under Workload Identity.
4. In Managed Identity, enter the Name of the managed identity you assigned to the resource.
5. In Resource Group, enter the Resource group name that your managed identity belongs to.
6. Using the Azure CLI, copy and run the provided command in your local terminal.
7. After the command completes, click Close on the modal in the Astro UI.
8. (Optional) repeat Steps 4 - 8 for any other Deployments that need to be authorized to Azure.

Step 3: Create an Airflow connection

1. In the Astro UI, click Environment in the main menu to open the Connections page.
2. Click + Connection to add a new connection for your Workspace.
3. Search for Azure, then select the Managed identity option.
4. Configure your Airflow connection with the information you copied in the previous steps.
5. Link the connection to the Deployment(s) where you configured your managed identity.

Any DAG that uses your connection will now be authorized to Azure through your managed identity.

See also

• Manage Airflow connections and variables
• Deploy code to Astro