Environment Variables on Astronomer can be used to set both Airflow configurations (reference here) or custom values, which are then applied to your Airflow Deployment either locally or on Astronomer.
Environment Variables can be used to set any of the following (and much more):
- SMTP to enable email alerts
- Airflow Parallelism and DAG Concurrency
- A Secret Backend to manage your Airflow Connections and Variables
- Store Airflow Connections and Variables
- Customize your default DAG view in the Airflow UI (Tree, Graph, Gantt etc.)
This guide will cover the following:
- How to set Environment Variables on Astronomer
- How Environment Variables are stored on Astronomer
- How to store Airflow Connections and Variables as Environment Variables
Set Environment Variables on Astronomer
On Astronomer, there are 3 ways to set Environment Variables:
- via your
.envfile (Local Only)
- via your
- via the Software UI
Read below for instructions on how to configure them via all 3 methods.
Note: While Environment Variables on Astronomer are the equivalent of updating your
airflow.cfg, you are NOT able to bring your own
airflow.cfgfile on Astronomer and configure it directly.
.env (Local Only)
The Astronomer CLI comes with the ability to bring in Environment Variables from a specified
.env file, which was automatically generated when you initialized an Airflow project on Astronomer via
$ astro dev init.
To add Environment Variables locally,
- Find your
.envfile in your Airflow project directory
- Add your Environment Variables of choice to that
- Rebuild your image to apply those changes by running
$ astro dev start --env .env
.env file, insert the value and key, ensuring all-caps for all characters. For example:
Note: If your Environment Variables contain secrets you don't want to expose in plain-text, you may want to add your
.gitignoreif and when you deploy these changes to your version control tool.
Confirm your Environment Variables were Applied
To confirm that the Environment Variables you just set were applied to your Airflow Deployment locally, first run:
This will output 3 Docker containers that were provisioned to run Airflow's 3 primary components on your machine: The Airflow Scheduler, Webserver and Postgres Metadata Database.
Now, create a Bash session in your scheduler container by running:
docker exec -it <scheduler-container-name> /bin/bash
If you run
ls -1 following this command, you'll see a list of running files:
bash-5.0$ ls -1
Dockerfile airflow.cfg airflow_settings.yaml dags include logs packages.txt plugins requirements.txt unittests.cfg
This should output all Environment Variables that are running locally, some of which are set by you and some of which are set by Astronomer by default.
Note: You can also run
cat airflow.cfgto output all contents in that file.
Multiple .env Files
The CLI will look for
.env by default, but if you want to specify multiple files, make
.env a top-level directory and create sub-files within that folder.
In other words,your project might look like the following:
If you're working on an Airflow project locally but intend to deploy to Astronomer and want to commit your Environment Variables to your source control tool, you can set them in your
Dockerfile. This file was automatically created when you first initialized your Airflow project on Astronomer (via
$ astro dev init).
Note: Given that this file will be committed upstream, we strongly recommend witholding Environment Variables containing sensitive credentials from your
Dockerfileand instead inserting them via your
.envfile locally (while adding the file to your
.gitignore) or setting them as 'secret' via the Software UI, as described in a dedicated section below.
To add Environment Variables, insert the value and key in your
Dockerfile beginning with
ENV, ensuring all-caps for all characters. With your Airflow image commonly referenced as a "FROM" statement at the top, your Dockerfile might look like this:
Once your Environment Variables are added,
$ astro dev stopand
$ astro dev startto rebuild your image and apply your changes locally OR
$ astro deployto apply your changes to your running Airflow Deployment on Astronomer
Note: Environment Variables injected via the
Dockerfileare mounted at build time and can be referenced in any other processes run during the docker build process that immediately follows
$ astro deployor
$ astro dev start.
Environment Variables applied via the Software UI only become available once the docker build process has been completed.
via the Software UI
The last way to add Environment Variables on Astronomer is to add them via the Software UI. For Environment Variables that you only need on Astronomer and not locally, we'd recommend using this method.
To set them,
- Navigate to the Software UI
- Go to "Deployment" > "Variables"
- Add your Environment Variables
Note: Input for all configurations officially supported by Airflow are pre-templated, but you're free to specify your own values.
Mark Environment Variables as "Secret"
On Astronomer, users have the ability to mark any Environment Variable as "secret" via the UI. For those who have Environment Variables containing potentially sensitive information (e.g. SMTP password, S3 bucket, etc.), we'd recommend leveraging this feature.
To do so from the "Variables" tab,
- Enter a Key
- Enter a Value
- Check the "Secret?" box
- Click "Add"
- Press "Deploy Changes"
Once changes are deployed, Environment Variables marked as "secret" will NOT be available in plain-text to any user in the Workspace.
A few additional notes:
- Workspace Editors and Admins are free to set an existing non-secret Env Var to "secret" any time
- To convert a "secret" Env Var to a "non-secret" Env Var, you'll be prompted to enter a new value
- If you export Environment Variables via JSON, "secret" values will NOT render in plain-text
- Users cannot add a new variable that has the same key as an existing variable
Read below for more detail on how Environment Variables are encrypted on Astronomer.
Note: As noted above, Workspace roles and permissions apply to actions in the "Variables" tab. For a full breakdown of permissions for each role, reference Astronomer's "Roles and Permissions" doc.
Precedence amongst 3 Methods
Given the ability to set Environment Variables across 3 different methods potentially simultaneously, it's worth noting the precedence each take.
On Astronomer, Environment Variables will be applied and overridden in the following order:
- Software UI
- .env (Local Only)
- Default Airflow Values (
In other words, if you set
AIRFLOW__CORE__PARALLELISM with one value via the Software UI and you set the same Environment Variable with another value in your
Dockerfile, the value set in the Software UI will take precedence.
How Environment Variables are Stored on Astronomer
All values for Environment Variables that are added via the Software UI are stored as a Kubernetes Secret, which is encrypted at rest and mounted to your Deployment's Airflow pods (Scheduler, Webserver, Worker(s)) as soon as they're set or changed.
Environment Variables are not stored in Airflow's Metadata Database and are not stored in Astronomer's platform database. Unlike other components, the Astronomer Houston API fetches them from the Kubernetes Secret instead of the platform's database to render them in the Software UI.
For information on how Airflow Connections and Variables are encrypted on Astronomer, refer to this forum post.
Adding Airflow Connections and Variables via Environment Variables
For users who regularly leverage Airflow Connections and Variables, we'd recommend storing and fetching them via Environment Variables.
As mentioned above, Airflow Connections and Variables are stored in Airflow's Metadata Database. Adding them outside of task definitions and operators requires an additional connection to Airflow's Postgres Database, which is called every time the Scheduler parses a DAG (as defined by
processor_poll_interval, which is set to 1 second by default). By adding Connections and Variables as Environment Variables, you can refer to them more easily in your code and lower the amount of open connections, thus preventing a strain on your Database and resources.
Read below for instructions on both.
The Environment Variable naming convention for Airflow Connections is:
For example, consider the following Airflow Connection:
- Connection ID:
- Connection URI:
Here, the full Environment Variable would read:
You're free to set this Environment Variable via an
.env file locally, via your Dockerfile or via the Software UI as explained above. For more information on how to generate your Connection URI, refer to Airflow's documentation.
The Environment Variable naming convention for Airflow Variables is:
For example, consider the following Airflow Variable:
- Variable Name:
Here, the Environment Variable would read:
Note: The ability to store and fetch Airflow Variables was introduced in Airflow 1.10.10 and is not available in earlier versions.