Airflow connections are used for storing credentials and other information necessary for connecting to external services. Airflow variables are a generic way to store and retrieve arbitrary content or settings as a simple key value store within Airflow.
Use this document to select the right Airflow connection and variable management strategies for your team.
Airflow supports several different methods for managing connections and variables. Each of these strategies has benefits and limitations related to their security and ease of use. The strategies you choose should be compatible with both your local environments and Astro Deployments, allowing you to import and export objects between the two contexts.
For in-depth information on creating and managing connections, see Connection Basics.
- A locally hosted Astro project created with the Astro CLI. See Create a project.
- A Deployment on Astro. See Create a Deployment.
Choose a connection and variable management strategy
The following table suggests possible management strategies for specific use cases.
|I'm getting started and want to quickly create Airflow objects||Airflow UI|
|I prefer to manage my Airflow variables in a Git repository and to upload directly to Airflow||Airflow UI|
|I need to keep my connections and variables stored in a centralized and secure location.||Secrets backend|
|I don't have a secrets backend, but I still want some security and permissions attached to Airflow objects.||Environment variables|
Because variables and connections serve different purposes in Airflow, you might want to use a different strategy for each object type. For example, you can use a secrets backend for connections and use combination of a
json files and the Airflow UI for variables.
Compare connection and variable management strategies
The following sections explain the benefits, limitations, and implementations of each strategy in more detail.
The quickest way to create Airflow connections and variables is through the Airflow UI. This experience is the same for both local Airflow environments and Astro Deployments. Astronomer recommends this method if you're just getting started with Airflow, you want to get your DAGs running quickly, or if you want to export connections in a URI/JSON format.
- The UI has features for correctly formatting and testing your connections.
- It's easy to change variables or connections to test different use cases on the fly.
- You can export and import your variables from the Airflow UI using JSON files and the Astro CLI. See Import and export connections and variables.
- Connections and variables are encrypted and stored in the Airflow metadata database.
- You cannot export or import connections from the UI for security reasons.
- Managing many connections or variables can become unwieldy.
- In a local environment, you lose your connections and variables if you delete your metadata database with
astro dev kill.
A secrets backend is the most secure way to store connections and variables. You can access a secrets backend both locally and on Astro by configuring the appropriate credentials in your Airflow environment. Astronomer recommends this method for all staging and production deployments. See the following documentation for setup steps:
- Store objects in a centralized location alongside other secrets used by your organization.
- Comply with internal security postures and policies that protect your organization.
- Recover objects if your Airflow environments go down.
- Share secrets across different Airflow environments.
- Allow selective access to connections and variables by using
- Limit the number of open connections to your metadata database, especially if you are using your connections and variables outside of task definitions.
- A third-party secrets manager is required.
- Separate configurations might be required for using a secrets backend locally and on Astro.
- You cannot use the Airflow UI to view connections and variables.
- You are responsible for ensuring that secrets are encrypted.
You can use Airflow's system-level environment variables to store connections and variables. This strategy is recommended when you don't have a secrets backend, but you still want to take advantage of security and RBAC features to limit access to connections and variables. You can configure system-level environment variables both locally and on Astro. For setup steps, see:
- If you use an
.envfile for your local Airflow environment and your local metadata database is corrupted or accidentally deleted, you still have access to all of your connections and variables.
- You can export environment variables from a local Airflow environment to Astro using the Astro CLI. See Import and export connections and variables.
- You can override Airflow variables set in the Airflow UI. See Environment variable priority
- You can create your connections and variables as environment variables from the Cloud UI. See Use environment variables.
- Environment variables marked as Secret are encrypted in the Astronomer control plane. See How environment variables are stored on Astro for details.
- This approach limits the number of open connections to your metadata database, especially if you are using your connections and variables outside of task definitions.
- You can't view connections and variables from the Airflow UI.
- You must restart your local environment using
astro dev restartwhenever you make changes to your
- The environment variables are defined in plain text in your
- Connections must be formatted as either a URI or serialized JSON.
- Environment variables are not as secure or centralized compared to a secrets backend.
- You cannot directly export your environment variables from the Cloud UI to a local Airflow environment. See Import and export Airflow objects.
While it's possible to manage Airflow connections and variables with these strategies, Astronomer doesn't recommend them at scale:
- You can use the Airflow REST API to programmatically create Airflow connections and variables for a Deployment. Airflow objects created with the API are stored in the Airflow metadata database and visible in the Airflow UI.
- For local Astro projects, you can use
airflow_settings.yamlfor defining your connections and variables. See Configure
airflow_settings.yamlfor more details.