Skip to main content

Set environment variables on Astro

An environment variable on Astro is a key-value configuration that is applied to a specific Deployment. You can use environment variables to set Airflow configurations and custom values for Deployments on Astro. For example, you can use environment variables to:

You can use environment variables to configure the same keys and values that you would configure as Airflow variables. However, the variables that you configure in Astro are exposed as environment variables and managed differently than Airflow variables. For more information, see How environment variables are stored on Astro.

Some environment variables on Astro are set globally and cannot be overridden for individual Deployments. For more information on these environment variables, see Global environment variables.

Set environment variables in the Cloud UI​

astro cli

If you prefer to work with the Astro CLI, you can create and update environment variables using the astro deployment variable create and astro deployment variable update commands. See CLI command reference.

  1. In the Cloud UI, select a Workspace, click Deployments, and then select a Deployment.

  2. Click the Variables tab.

  3. Click Edit Variables.

  4. Enter an environment variable key and value. For sensitive credentials that should be treated with an additional layer of security, select the Secret checkbox. This will permanently hide the variable's value from all users in your Workspace.

  5. Click Add.

  6. Click Save Variables to save your changes. Your Airflow scheduler, webserver, and workers restart. After saving, it can take up to two minutes for new variables to be applied to your Deployment.

Edit existing values​

After you set an environment variable key, only the environment variable value can be modified. You can modify environment variables that are set as secret. However, the variable value is never shown. When you modify a secret environment variable, you'll be prompted to enter a new value.

  1. In the Cloud UI, select a Workspace, click Deployments, and then select a Deployment.

  2. Click the Variables tab.

  3. Click Edit Variables.

  4. Click Edit value next to the value you want to edit.

    Edit value location

  5. Modify the variable's value, then click Done editing.

    Done editing location

  6. Click Save Variables to save your changes. Your Airflow scheduler, webserver, and workers restart. After saving, it can take up to two minutes for updated variables to be applied to your Deployment.

How environment variables are stored on Astro​

Non-secret environment variables set in the Cloud UI are stored in a database that is managed by Astronomer and hosted in the Astro control plane. When you configure a secret environment variable in the Cloud UI, the following methodology is used:

  • Astro generates a manifest that defines a Kubernetes secret, named env-secrets, that contains your variable's key and value.
  • Astro applies this manifest to your Deployment's namespace.
  • After the manifest is applied, the key and value of your environment variable are stored in a managed etcd cluster at rest within Astro.

This process occurs every time you update the environment variable's key or value. To use a secret environment variable value in a task running on the Kubernetes executor or the KubernetesPodOperator, you need to mount the value from the Astro kubernetes secret to your Kubernetes Pod. See:

caution

Environment variables marked as secret are stored securely by Astronomer and are not shown in the Cloud UI. However, it's possible for a user in your organization to create or configure a DAG that exposes secret values in Airflow task logs. Airflow task logs are visible to all Workspace members in the Airflow UI and accessible in your Astro cluster's storage.

To avoid exposing secret values in task logs, instruct users to not log environment variables in DAG code.

Set environment variables in your Dockerfile​

If you want to store environment variables using an external version control tool, Astronomer recommends setting them in your Dockerfile. This file is automatically created when you first initialize an Astro project using astro dev init.

Environment variables added to a Dockerfile are mounted at build time and can be referenced in any other build process that follows astro deploy or astro dev start. Environment variables applied in the Cloud UI only become available once the Docker build process is completed.

caution

Environment variables set in your Dockerfile are stored in plain text. For this reason, Astronomer recommends storing sensitive environment variables using the Cloud UI or a third party secrets backend. For more information, see Configure a secrets backend.

To add environment variables, declare an ENV command with the environment variable key and value. For example, the following Dockerfile sets two environment variables:

FROM quay.io/astronomer/astro-runtime:8.4.0
ENV AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG=1
ENV AIRFLOW_VAR_MY_VAR=25

After you add your environment variables, use one of the following options to deploy your changes:

  • Run astro dev restart to rebuild your image and apply your changes locally.
  • Run astro deploy to apply your changes to your Deployment on Astro.

Environment variable priority​

On Astro, environment variables are applied and overridden in the following order:

For example, if you set AIRFLOW__CORE__PARALLELISM with one value in the Cloud UI and you set the same environment variable with another value in your Dockerfile, the value set in the Cloud UI takes precedence.

Add Airflow connections and variables using environment variables​

If you regularly use Airflow connections and variables, Astronomer recommends storing and fetching them with environment variables instead of adding them to the Airflow UI.

Airflow connections and variables are stored in the Airflow metadata database. Calling them outside of task definitions and operators requires an additional connection to the Airflow metadata database which is used every time the scheduler parses a DAG.

By adding connections and variables as environment variables, you can lower the amount of open connections and improve the performance of your database and resources.

Airflow connections​

In Astro Runtime version 4.2 and earlier, use the Airflow connection URI format to store connections as environment variables. The naming convention for Airflow environment variable connections is:

  • Key: AIRFLOW_CONN_<CONN_ID>
  • Value: <connection-uri>

For example, consider the following Airflow connection:

  • Connection ID: MY_PROD_DB
  • Connection URI: my-conn-type://login:password@host:5432/schema

To store this connection as an environment variable, you create an environment variable with the key AIRFLOW_CONN_MY_PROD_DB and the value my-conn-type://login:password@host:5432/schema.

In Astro Runtime version 5.0 and later, you can also use JSON format to store connections. See JSON format example. When using the JSON format in environment variables, the JSON object must be defined in a single, unbroken line.

info

Airflow connections set with environment variables do not appear in the Airflow UI. They can only be seen and updated in the Cloud UI or your Dockerfile.

Airflow variables​

Use environment variables to store values you would normally store as Airflow variables. This gives you better control and security of custom values. To fetch a Deployment environment variable value from a DAG, you must format the variable key as AIRFLOW_VAR_<VAR_NAME>.

For example, consider the following Airflow variable:

  • Variable name: My_Var
  • Value: 2

To store this Airflow variable as an environment variable, you create an environment variable with the key AIRFLOW_VAR_MY_VAR and the value 2.

You can then use the following Python functions in the top level of your DAG code to fetch the variable value:

  • Variable.get('<VAR_NAME>'): This method is more secure for fetching secret values. However, this method can affect performance because it makes a request to the Airflow metadata database every time your DAGs are parsed, which can occur every 30 seconds. See DAG writing best practices for more information about avoiding repeated requests in top level code.

  • os.getenv('AIRFLOW_VAR_<VAR_NAME>','<default-value>'): This method is faster because it reduces the number of Airflow metadata database requests. However, it's less secure. Astronomer does not recommend using os.getenv with secret values because calling these values with the function can print them to your logs.

    Replace <default_value> with a default value to use if Airflow can't find the environment variable. Typically, this is the value you defined for the environment variable in the Cloud UI.

Set Airflow configurations using environment variables​

caution

Some Airflow configurations should not be overridden because Astro already uses them. See Global environment variables for a list of all non-configurable environment variables.

You can use Astro environment variables to set Airflow environment variables.

For example, to set AIRFLOW__CORE__PARALLELISM in your Deployment, you would configure the following environment variable:

  • Key: AIRFLOW__CORE__PARALLELISM
  • Value: 64

See the Airflow Configurations Reference for a list of all possible configurations.

Sign up for Developer Updates

Get a summary of new Astro features once a month.

You can unsubscribe at any time.
By proceeding you agree to our Privacy Policy, our Website Terms and to receive emails from Astronomer.