On Astro, an Environment Variable is a key-value configuration stored in a configuration file that applies to a specific Deployment. You can use environment variables to configure custom environment variables for your Deployment, customize core settings of Airflow and its pre-installed providers, or store Airflow connections and variables.
Some scenarios where you might use environment variables in your Deployment include:
- Adding a token or a URL that is required by your Airflow DAGs or tasks.
- Integrating with Datadog or other third-party tooling to export Deployment metrics.
- Specifying a tag that's added to all resources created by the Deployment and indicates whether a resource is for development or production.
Some examples of customizing core settings of Airflow or any of its pre-installed providers include:
- Changing the import timeout of DAGBag using
- Setting up an SMTP service to receive Airflow alerts by email.
Some environment variables on Astro are set globally and cannot be overridden for individual Deployments, while others are used by Astro Runtime to enhance your Airflow experience. For more information on these, see Global environment variables.
Choose a strategy
Environment variables can be used in many different contexts on Airflow. To choose the right management and implementation strategy for your specific use case, it's helpful to know how Astro prioritizes and stores environment variables for each available management option.
On Astro, you can manage environment variables from two different locations:
- Your Deployment's Variables tab in the Cloud UI. This is the fastest and easiest way to set an environment variable. See Using the Cloud UI for setup steps.
- Your Astro project
Dockerfile. Storing environment variables in your Dockerfile allows you to manage them as code in a version control tool like GitHub. However, environment variables stored in the Dockerfile do not appear in the Cloud UI and might be harder to reference from DAG code. Using the Dockerfile is recommended for more complex production use cases, such as implementing a secrets backend. See Using your Dockerfile for setup steps.
At the local development level, you can use your Astro project
.env file to set and test environment variables. When you're ready to push these environment variables to a Deployment, you can use the Astro CLI to export and store them in the Cloud UI for your Deployment.
How environment variables are stored in the Cloud UI
When you set a non-secret environment variable in the Cloud UI, Astronomer stores the variable in a database that is hosted and managed by Astronomer.
When set a secret environment variable in the Cloud UI, the following happens:
- Astro generates a manifest that defines a Kubernetes secret, named
env-secrets, that contains your variable's key and value.
- Astro applies this manifest to your Deployment's namespace.
- After the manifest is applied, the key and value of your environment variable are stored in a managed etcd cluster at rest within Astro.
This process occurs every time you update the environment variable's key or value.
Environment variables marked as secret are stored securely by Astronomer and are not shown in the Cloud UI. However, it's possible for a user in your organization to create or configure a DAG that exposes secret values in Airflow task logs. Airflow task logs are visible to all Workspace members in the Airflow UI and accessible in your Astro cluster's storage.
To avoid exposing secret values in task logs, instruct users to not log environment variables in DAG code.
Environment variable priority
On Astro, environment variables set in the Cloud UI take precedence over environment variables set in your Dockerfile.
For example, if you set
AIRFLOW__CORE__PARALLELISM with one value in the Cloud UI and you set the same environment variable with another value in your
Dockerfile, the value set in the Cloud UI takes precedence.
Similarly, in local development, environment variables set in your
.env file take precedence over environment variables set in your Dockerfile.
Example use cases
For most use cases, Astronomer recommends using the Cloud UI to store your environment variables for the following reasons:
- It's easy to use.
- It has built-in security for secret environment variables.
- You can import and export environment variables using the Astro CLI.
There are some scenarios when you might want to use a mix of methods or strategies other than the Cloud UI. The following table prescribes specific methods for various common use cases.
|You are a new Astro user. You just created a Deployment and want to integrate your Hashicorp Vault secrets backend to test DAGs.||Cloud UI for ease of use and visibilty.|
|Your team has multiple environment types and you use an ||Cloud UI for visibility.|
|You want to standardize a core set of Airflow environment configurations across multiple environments.|
|You want to use version control to manage your environment variables from a code repository.|
|You want to carry over environment variables when you promote code from a development environment to a production environment.|
|You are part of a production support team analyzing DAG failures and want to turn on debugging-related environment variables temporarily.||Cloud UI for ease of use.|
|You're developing locally, and you want to use a secret credential in your DAGs.||Use the |
|You use environment variables to store your Airflow connections and variables, and you have to reconfigure these between Deployments based on the environment type.||Cloud UI for visibility.|
|You want the features of the Cloud UI for environment variables, but you also keep track of the non-secret environment variables in your code repository.||Use the Astro CLI and an automation script to add or update the non-secret environment variables in your Deployment from your code repository.|