Configure a secrets backend

Apache Airflow variables and connections often contain sensitive information about your external systems that you need to keep in a secrets backend tool, which stores secrets in a secure and centralized location. Unlike other management strategies, such as using Environment Variables or working with connections and variables in the Airflow UI, secrets backends require a third-party secrets manager. This means that you can use a secrets manager administered by your organization for existing security protocols, or you need to choose and set up a secrets backend.

This document explains the available secrets backend integrations supported by Astro and how Airflow finds connections and variables if you use multiple strategies to manage them.

See Manage connections and variables to learn more about your available options and decide whether using a secrets backend complies with your organization's security requirements.

Available integrations

Secrets backend integrations can be configured individually with each Astro Deployment by someone with Workspace Operator permissions.

Using secrets to set Airflow connections requires knowledge of how to generate Airflow connections in URI or JSON format. See Import and export Airflow connections and variables for guidance on how to export your connections and variables based on where they are stored.

Astro integrates with the following secrets backend tools:

• AWS Secrets Manager
• AWS Systems Manager Parameter Store
• Azure Key Vault
• Google Cloud Secret Manager
• Hashicorp Vault

How Airflow finds connections or variables

tip

If you need to access your secrets backend from your local Airflow, you can mount your user credentials to a local Airflow environment. While this implementation is not recommended for Astro Deployments, it lets you quickly test pipelines with data hosted in your cloud. See Authenticate to cloud services.

If you configure a secrets backend on Astro, you can still continue to define Airflow variables and connections as either environment variables or in the Airflow UI. Airflow checks for the value of an Airflow variable or connection in the following order:

1. Secrets backend
2. Environment variables
3. The Airflow UI