Upgrade Astro Runtime

New versions of Astro Runtime are released regularly to support new Astro and Apache Airflow functionality. To take advantage of new features and bug and security fixes, Astronomer recommends upgrading Astro Runtime as new versions are available.

Upgrade considerations

Consider the following when you upgrade Astro Runtime:

• All versions of the Astro CLI support all versions of Astro Runtime. There are no dependencies between the two products.
• Upgrading to certain versions of Runtime might result in extended upgrade times or otherwise disruptive changes to your environment. To learn more, see Version-specific upgrade considerations.
• Astronomer does not support downgrading a Deployment on Astro to a lower version of Astro Runtime.

To stay up to date on the latest versions of Astro Runtime, see Astro Runtime release notes. For more information on Astro Runtime versioning and support, see Astro Runtime versioning and lifecycle policy. For a full collection of Astro Runtime Docker images, go to the Astro Runtime repository on Quay.io.

Version upgrade considerations

This is where you'll find the upgrade considerations for specific Astro Runtime versions. This includes breaking changes, database migrations, and other considerations.

Runtime 6 (Airflow 2.4)

Smart Sensors were deprecated in Airflow 2.2.4 and removed in Airflow 2.4.0. If your organization is still using Smart Sensors, you'll need to start using deferrable operators. See Deferrable operators.

Runtime 5 (Airflow 2.3)

Astro Runtime 5.0.0, based on Airflow 2.3, includes changes to the schema of the Airflow metadata database. When you first upgrade to Runtime 5.0.0, consider the following:

• Upgrading to Runtime 5.0.0 can take 10 to 30 minutes or more depending on the number of task instances that have been recorded in the metadata database throughout the lifetime of your Deployment on Astro.

• Once you upgrade successfully to Runtime 5, you might see errors in the Airflow UI that warn you of incompatible data in certain tables of the database. For example:

  Airflow found incompatible data in the `dangling_rendered_task_instance_fields` table in your metadata database, and moved...

  These warnings have no impact on your tasks or DAGs and can be ignored. If you want to remove these warning messages from the Airflow UI, reach out to Astronomer support. If requested, Astronomer can drop incompatible tables from your metadata database.

For more information on Airflow 2.3, see "Apache Airflow 2.3.0 is here" or the Airflow 2.3.0 changelog.

Prerequisites

• An Astro project.
• An Astro Deployment.
• The Astro CLI.

info

If you're upgrading a local Airflow environment, you don't need an Astro Deployment and you can skip steps 3-4.

(Optional) Pin provider package versions

Major Astro Runtime upgrades can include major upgrades to built-in provider packages. These package upgrades can sometimes include breaking changes for your DAGs. See the Apache Airflow documentation for a list of all available provider packages and their release notes.

For the most stable upgrade path, Astronomer recommends pinning all provider package versions from your current Runtime version before upgrading. To check the version of all provider packages installed in your Runtime version, run:

docker run --rm quay.io/astronomer/astro-runtime:<current-runtime-version> pip freeze | grep apache-airflow-providers

After reviewing this list, pin the version for each provider package in your Astro project requirements.txt file. For example, Runtime 7.4.1 uses version 4.0.0 of apache-airflow-providers-databricks. To pin this version of the Databricks provider package when you upgrade to a later version of Runtime, you add the following line to your requirements.txt file:

apache-airflow-providers-databricks==4.0.0

Update Your Dockerfile

1. In your Astro project, open your Dockerfile.

2. Change the Docker image in the FROM statement of your Dockerfile to a new version of Astro Runtime.

   To upgrade to the latest version of Runtime, for example, change the FROM statement in your Dockerfile to:

   FROM quay.io/astronomer/astro-runtime:7.4.1

   You must always specify the major, minor, and patch version of any given Astro Runtime version.

Test Astro Runtime locally

Astronomer recommends testing new versions of Astro Runtime locally before upgrading a Deployment on Astro.

1. Save the changes to your Dockerfile.

2. Open your project directory in your terminal and run astro dev restart. This restarts the Docker containers for the Airflow webserver, scheduler, triggerer, and Postgres metadata database.

3. Access the Airflow UI of your local environment by navigating to http://localhost:8080 in your browser.

4. Confirm that your local upgrade was successful by scrolling to the bottom of any page. You should see your new Astro Runtime version in the footer as well as the version of Airflow it is based on.

   

5. (Optional) Run DAGs locally to ensure that all of your code works as expected. If you encounter errors after your upgrade, it's possible that your new Astro Runtime version includes a breaking provider package change. If you encounter one of these breaking changes, follow the steps in Upgrade or pin provider package versions to check your provider package versions and, if required, pin the provider package version from your previous Runtime version in your requirements.txt file.

(Optional) Upgrade and test provider packages

If you pinned provider package versions before your upgrade, upgrade your provider packages by changing the pinned version in your requirements.txt file. Test east provider package upgrade locally before deploying to Astro.

Deploy to Astronomer

To push your upgraded project to an Astro Deployment, run:

astro deploy

For more information about deploying to Astro, see Deploy code.

caution

Once you upgrade to a Deployment on Astro to a new version of Astro Runtime, you cannot roll back or downgrade to a lower version. If you attempt to do so, you will see an error in the Astro CLI and your request to deploy will not succeed.

Confirm your upgrade on Astro

1. In the Cloud UI, select a Workspace, click Deployments, and then select a Deployment.

2. Click Open Airflow.

3. In the Airflow UI, scroll to the bottom of any page. You should see your new Runtime version in the footer:

   

   You will also see an Image tag for your deploy. This tag is shown only for Deployments on Astro and is not generated for changes in a local environment.