Skip to main content

Configure an External Secrets Backend on Astro

Overview​

Apache Airflow variables and connections often contain sensitive information about your external systems that should be kept secret in a secure, centralized location that complies with your organization's security requirements. While secret values of Airflow variables and connections are encrypted in the Airflow metadata database of every Deployment, Astronomer recommends integrating with a secrets backend tool.

Integrating a secrets backend tool on Astro allows you to:

  • Store Airflow variables and connections in a centralized location alongside secrets from other tools and systems used by your team, including Kubernetes secrets, SSL certificates, and more.
  • Comply with internal security postures and policies that protect your organization.
  • Recover in the case of an incident.
  • Automatically pull Airflow variables and connections that are already stored in your secrets backend when you create a new Deployment instead of having to set them manually in the Airflow UI.

To meet these requirements, Astro supports integration with a variety of secret backend tools. This guide provides setup steps for configuring the following tools as secrets backends on Astro:

  • Hashicorp Vault
  • AWS Systems Manager Parameter Store
  • AWS Secrets Manager
  • Google Cloud Secret Manager
  • Azure Key Vault

All secrets backend integrations are set per Deployment on Astro.

info

If you enable a secrets backend on Astro, you can continue to define Airflow variables and connections either as environment variables or in the Airflow UI as needed. If set via the Airflow UI, variables and connections are stored as encrypted values in Airflow's metadata database.

When Airflow checks for the value of an Airflow variable or connection, it does so in the following order of precedence:

  1. Secrets backend
  2. Environment variable
  3. Set via the Airflow UI
tip

Setting Airflow connections via secrets requires knowledge of how to generate Airflow connection URIs. If you plan to store Airflow connections on your secrets backend, read the Apache Airflow documentation for guidance on how to generate a connection URI.

Setup​

This topic provides steps for how to use Hashicorp Vault as a secrets backend for both local development and on Astro. To do this, you will:

  1. Create an AppRole in Vault which grants Astro minimal required permissions.
  2. Write a test Airflow variable or connection as a secret to your Vault server.
  3. Configure your Astro project to pull the secret from Vault.
  4. Test the backend in a local environment.
  5. Deploy your changes to Astro.

Prerequisites​

To use this feature, you need:

If you do not already have a Vault server deployed but would like to test this feature, we recommend that you either:

Step 1: Create a Policy and AppRole in Vault​

To use Vault as a secrets backend, we recommend configuring a Vault AppRole with a policy that grants only the minimum necessary permissions for Astro. To do this:

  1. Create a Vault policy with the following permissions:

    path "secret/data/variables/*" {
      capabilities = ["read", "list"]
    }

    path "secret/data/connections/*" {
      capabilities = ["read", "list"]
    }

  2. Create a Vault AppRole and attach the policy you just created to it.

  3. Retrieve the role-id and secret-id for your AppRole by running the following commands:

    vault read auth/approle/role/<your-approle>/role-id
    vault write -f auth/approle/role/<your-approle>/secret-id

    Save these values for Step 3.

Step 2: Write an Airflow Variable or Connection to Vault​

To test whether your Vault server is set up properly, create a test Airflow variable or connection to store as a secret. You will use this secret to test your backend's functionality in Step 4, so it can be either a real or placeholder value.

To store an Airflow variable in Vault as a secret, run the following Vault CLI command with your own values:

vault kv put secret/variables/<your-variable-key> value=<your-variable-value>

To store a connection in Vault as a secret, run the following Vault CLI command with your own values:

vault kv put secret/connections/<your-connection-id> conn_uri=<connection-type>://<connection-login>:<connection-password>@<connection-host>:5432

To confirm that your secret was written to Vault successfully, run:

# For variables
$ vault kv get secret/variables/<your-variable-key>
# For connections
$ vault kv get secret/connections/<your-connection-id>

Step 3: Set Up Vault Locally​

In your Astro project, add the Hashicorp Airflow provider to your project by adding the following to your requirements.txt file:

apache-airflow-providers-hashicorp

Then, add the following environment variables to your Dockerfile:

# Make sure to replace `<your-approle-id>` and `<your-approle-secret>` with your own values.
ENV AIRFLOW__SECRETS__BACKEND="airflow.contrib.secrets.hashicorp_vault.VaultBackend"
ENV AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_path": "connections", "variables_path": "variables", "config_path": null, "url": "https://vault.vault.svc.cluster.local:8200", "auth_type": "approle", "role_id":"<your-approle-id>", "secret_id":"<your-approle-secret>"}'

This tells Airflow to look for variable and connection information at the secret/variables/* and secret/connections/* paths in your Vault server. In the next step, you'll test this configuration in a local Airflow environment.

warning

If you want to deploy your project to a hosted Git repository before deploying to Astro, be sure to save <your-approle-id> and <your-approle-secret> securely. We recommend adding them to your project's .env file and specifying this file in .gitignore.

When you deploy to Astro in Step 4, you can set these values as secrets via the Cloud UI.

info

By default, Airflow uses "kv_engine_version": 2, but we've written this secret using v1. You can change this to accommodate how you write and read your secrets.

For more information on the Airflow provider for Hashicorp Vault and how to further customize your integration, read the Apache Airflow documentation.

Step 4: Run an Example DAG to Test Vault Locally​

To test Vault, write a simple DAG which calls your test secret and add this DAG to your project's dags directory. For example, you can use the following DAG to print the value of a variable to your task logs:

from airflow import DAG
from airflow.hooks.base import BaseHook
from airflow.models import Variable
from airflow.operators.python import PythonOperator
from datetime import datetime

def print_var():
    my_var = Variable.get("<your-variable-key>")
    print(f'My variable is: {my_var}')

    conn = BaseHook.get_connection(conn_id="<your-connection-key>")
    print(conn.get_uri())

with DAG('example_secrets_dag', start_date=datetime(2022, 1, 1), schedule_interval=None) as dag:

  test_task = PythonOperator(
      task_id='test-task',
      python_callable=print_var
)

Once you've added this DAG to your project:

  1. Run astro dev restart to push your changes to your local Airflow environment.

  2. In the Airflow UI (http://localhost:8080/admin/), trigger your new DAG.

  3. Click on test-task > View Logs. If you ran the example DAG above, you should should see the contents of your secret in the task logs:

    {logging_mixin.py:109} INFO - My variable is: my-test-variable

Once you confirm that the setup was successful, you can delete this example DAG.

Step 5: Deploy on Astro​

Once you've confirmed that the integration with Vault works locally, you can complete a similar set up with a Deployment on Astro.

  1. In the Cloud UI, add the same environment variables found in your Dockerfile to your Deployment environment variables. Specify AIRFLOW__SECRETS__BACKEND_KWARGS as secret to ensure that your Vault credentials are stored securely.

    warning

    Make sure to remove the surrounding single quotation marks ('') from AIRFLOW__SECRETS__BACKEND_KWARGS and the double quotation marks ("") from all other environment variable values defined in your Dockerfile. If you add these values with the quotation marks included in your Dockerfile, your configuration will not work on Astro.

  2. In your Astro project, delete the environment variables from your Dockerfile.

  3. Deploy your changes to Astro.

Now, any Airflow variable or connection that you write to your Vault server can be successfully accessed and pulled by any DAG in your Deployment on Astro.