Set up Azure Key Vault as your secrets backend

This topic provides setup steps for configuring Azure Key Vault as a secrets backend on Astro.

If you use a different secrets backend tool or want to learn the general approach on how to integrate one, see Configure a Secrets Backend.

Prerequisites

• A Deployment.
• The Astro CLI.
• An Astro project.
• An existing Azure Key Vault linked to a resource group.
• Your Key Vault URL. To find this, go to your Key Vault overview page > Vault URI.

If you do not already have Key Vault configured, read Microsoft Azure documentation.

Step 1: Register Astro as an app on Azure

Follow the Microsoft Azure documentation to register a new application for Astro.

At a minimum, you need to add a secret that Astro can use to authenticate to Key Vault.

Note the value of the application's client ID and secret for Step 3.

Step 2: Create an access policy

Follow the Microsoft documentation to create a new access policy for the application that you just registered. The settings you need to configure for your policy are:

• Configure from template: Select Key, Secret, & Certificate Management.
• Select principal: Select the name of the application that you registered in Step 1.

Step 3: Set up Key Vault locally

In your Astro project, add the following line to your requirements.txt file:

apache-airflow-providers-microsoft-azure

Add the following environment variables to your .env file:

AIRFLOW__SECRETS__BACKEND=airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>", "tenant_id": "<your-tenant-id>", "client_id": "<your-client-id>", "client_secret": "<your-client-secret>"}

To find your your client ID in Azure Portal, go to App Registration page > Application (Client) ID. To find your tenant ID, go to App Registration page > Directory (tenant) ID. To find your client secret, go to App Registration Page > Certificates and Secrets > Client Secrets > Value.

This configuration tells Airflow to look for variable information at the airflow/variables/* path in Azure Key Vault and connection information at the airflow/connections/* path. You can now run a DAG locally to check that your variables are accessible using Variable.get("<your-variable-key>").

By default, this setup requires that you prefix any secret names in Key Vault with airflow-connections or airflow-variables. If you don't want to use prefixes in your Key Vault secret names, set the values for sep, "connections_prefix", and "variables_prefix" to "" within AIRFLOW__SECRETS__BACKEND_KWARGS.

Step 4: Deploy to Astro

1. Run the following commands to export your environment variables to Astro.

   astro deployment variable create --deployment-id <your-deployment-id> --load --env .env

   In the Astro UI, mark AIRFLOW__SECRETS__BACKEND_KWARGS as Secret. See Set environment variables in the Astro UI.

2. Run the following command to push your updated requirements.txt file to Astro:

   astro deploy --deployment-id <your-deployment-id> 

3. (Optional) Remove the environment variables from your .env file, or store your .env file so that your credentials are hidden, for example with GitHub secrets.