On Astronomer, the process of pushing up your code to an individual Airflow Deployment involves customizing a locally built Docker image —— with your DAG code, dependencies, plugins, and so on —— that's then bundled, tagged, and pushed to your Docker Registry.
Included in that build is your
Dockerfile, a file that is automatically generated when you initialize an Airflow project on Astronomer via our CLI. Every successful build on Astronomer must include a
Dockerfile that references an Astronomer Certified Docker Image. Astronomer Certified (AC) is a production-ready distribution of Apache Airflow that mirrors the open source project and undergoes additional levels of rigorous testing conducted by our team.
To upgrade your Airflow Deployment to a higher version of Airflow, there are three steps:
- Initialize the upgrade by selecting a new Airflow version via the Software UI or CLI.
- Change the FROM statement in your project's
Dockerfileto reference an AC image that corresponds to the Airflow version indicated in Step 1.
- Deploy to Astronomer.
This guide provides more specific instructions for each of these steps. For information on Astronomer Certified versioning and support schedules, read Astronomer Certified Versioning and Support Policy.
Note: For more thorough guidelines on customizing your image, reference our "Customize Your Image" doc.
Available Astronomer Certified Versions
Starting with Astronomer Software v0.23, new versions of Astronomer Certified are automatically pulled into the Software UI and CLI within 24 hours of their publication via a cron job that pulls from Astronomer's update service. In other words, you don't have to upgrade Astronomer in order to upgrade Airflow.
Note: If you don't want to wait for new versions of Astronomer Certified to appear on their own, you can manually trigger the cron job with the following Kubernetes command:
kubectl create job --namespace astronomer --from=cronjob/astronomer-houston-update-airflow-check airflow-update-check-first-run
If you get a message indicating that a job already exists, delete the job and rerun the command.
Step 1. Initialize the Upgrade Process
The first step to upgrading your Deployment to a higher version of Apache Airflow is to indicate your intent to do so via the Software UI or CLI.
Note: The Software UI and CLI will only make available versions of Airflow that are higher than the version you're currently running in your
Dockerfile. For example, Airflow
1.10.7would not be available for an Airflow Deployment running
via the Software UI
To initialize the Airflow upgrade process via the Software UI, navigate to Deployment > Settings > Basics > Airflow Version. Next to Airflow Version,
- Select your desired version of Airflow
- Click Upgrade
This action will NOT interrupt or otherwise impact your Airflow Deployment or trigger a code change - it is simply a signal to our platform that you intend to upgrade such that we can guide your experience through the rest of the process.
Once you select a version, you can expect to see a banner next to Airflow Version indicating that the upgrade is in progress. For a user upgrading from 1.10.7 to 1.10.12, that banner would read
Upgrade from 1.10.7 to 1.10.12 in progress…
Note: If you'd like to change the version of Airflow you'd like to upgrade to, you can do so at anytime by clicking Cancel, re-selecting a new version and once again clicking Upgrade. More on that below.
via the Astronomer CLI
To initialize the Airflow upgrade process via the Astronomer CLI, first make sure you're authenticated by running
$ astro auth login <your-base-domain>.
Once authenticated, grab the
Deployment ID of the Airflow Deployment you'd like to upgrade by running:
astro deployment list
You can expect the following output:
astro deployment list
NAME DEPLOYMENT NAME DEPLOYMENT ID AIRFLOW VERSION
new-deployment-1-10-10-airflow-k8s-2 elementary-rotation-5522 ckgwdq8cs037169xtbt2rtu15 1.10.12
Deployment ID, run:
astro deployment airflow upgrade --deployment-id=<deployment-id>
This command will output a list of available versions of Airflow you can choose from and prompt you to pick one. For example, a user upgrading from Airflow 1.10.5 to Airflow 1.10.12 should see the following:
astro deployment airflow upgrade --deployment-id=ckguogf6x0685ewxtebr4v04x
# AIRFLOW VERSION
NAME DEPLOYMENT NAME ASTRO DEPLOYMENT ID AIRFLOW VERSION
Astronomer Stagings new-velocity-8501 v0.17.0 ckguogf6x0685ewxtebr4v04x 1.10.12
The upgrade from Airflow 1.10.5 to 1.10.12 has been started. To complete this process, add an Airflow 1.10.12 image to your Dockerfile and deploy to Astronomer.
As noted above, this action will NOT interrupt or otherwise impact your Airflow Deployment or trigger a code change - it is simply a signal to our platform that you intend to upgrade such that we can guide your experience through the rest of the process.
To complete the upgrade, all you have to do is add a corresponding AC image to your
Step 2: Deploy a New Astronomer Certified Image
1. Locate your Dockerfile in your Project Directory
First, open the
Dockerfile within your Astronomer directory. When you initialized an Airflow project via the Astronomer CLI, the following files should have been automatially generated:
├── dags # Where your DAGs go
│ └── example-dag.py # An example DAG that comes with the initialized project
├── Dockerfile # For Astronomer's Docker image and runtime overrides
├── include # For any other files you'd like to include
├── packages.txt # For OS-level packages
├── plugins # For any custom or community Airflow plugins
└── requirements.txt # For any Python packages
Depending on the OS distribution and version of Airflow you want to run, you'll want to reference the corresponding Astronomer Certified image in the FROM statement of your
2. Choose your new Astronomer Certified Image
Depending on the Airflow version you'd like to run or upgrade to, copy one of the images in the following table to your
Dockerfile and proceed to Step 3.
Once you upgrade Airflow versions, you CANNOT downgrade to an earlier version. The Airflow metadata database structurally changes with each release, making for backwards incompatibility across versions.
For our platform's full collection of Docker Images, reference Astronomer on Quay.io.
|Airflow Version||Debian-based Image|
Note: In November of 2020, Astronomer migrated its Docker Registry from Docker Hub to Quay.io due to a change in Docker Hub's rate limit policy. If you're using a legacy
astronomerinc/ap-airflowimage, replace it with a corresponding
quay.io/astronomer/ap-airflowimage to avoid rate limiting errors from DockerHub when you deploy to Astronomer (e.g.
toomanyrequests: You have reached your pull rate limit).
Step 3: Rebuild your Image
If you're developing locally, make sure to save your changes and issue the following from your command line:
$ astro dev stop
This will stop all 3 running Docker containers for each of the necessary Airflow components (Webserver, Scheduler, Postgres).
$ astro dev start
This will start those 3 Docker containers needed to run Airflow.
If you don't need to test this locally and just want to push to your Astronomer Software installation, you can run:
Step 4: Confirm your version in the Airflow UI
Once you've issued that command, navigate to your Airflow UI to confirm that you're now running the correct Airflow version.
If you're developing locally, you can:
- Head to http://localhost:8080/
- Navigate to
Once there, you should see your correct Airflow version listed.
Note: The URL listed above assumes your Webserver is at Port 8080 (default). To change that default, read this forum post.
If you're on Astronomer Software, navigate to your Airflow Deployment page on the Software UI.
Note: In Airflow 2.0, the Version page referenced above will be deprecated. Check the footer of the Airflow UI to validate Airflow version instead.
Migrate from Alpine to Debian
In an effort to standardize our offering and optimize for reliability, Debian Buster proved most suitable to handle complex dependencies and integrate well with Machine Learning Python Libraries that are commonly used with Airflow.
Aside from the initial Docker image build and deploy process, both base images offer the exact same Apache Airflow experience. For best practices on migrating from Alpine to Debian, read below.
Before you Begin
To avoid unexpected impact to your Airflow Deployment, we strongly recommend two things as you prepare to migrate from Alpine to Debian:
- Do not upgrade Airflow versions simultaneously.
- Test your changes locally before you push a new image to Astronomer.
If you're running an Alpine-based 1.10.12 image, for example, try the Debian-based AC 1.10.12 image locally before you push that image to Astronomer and before you upgrade to a higher version of Airflow.
Note: If your
packages.txtfile is empty, skip to step 3.
Step 1. Remove Packages
Debian Buster has many common packages installed by default, which means that you should be able to remove some dependencies.
If you have any packages installed primarily because they're native to another library (e.g.
sci-kit learn), we recommend that you remove those additional packages from your
packages.txt files and see if your image builds successfully.
If you test a Debian-based image and encounter an error, you can always add packages back as needed.
Note: If you need a particular version of any package, make sure to pin it in your
<package-name>==<version>). For Python packages that are pre-built into Astronomer's Debian image and listed in your
requirements.txtfile, the version of the package that's specified in
requirements.txtwill take precedence.
Step 2. Rename Existing Packages
For the dependencies you do have installed, a primary concern in migrating from Alpine to Debian is that Python and OS-level packages may be named differently.
packages.txt files as needed. If you include a package that does not exist or is named incorrectly, your image will fail to build.
Step 3. Modify your Dockerfile
Now, try to build your Debian-based image via the Astronomer CLI locally. To do so, replace the Alpine image in your
Dockerfile with an available Debian image.
For AC 1.10.12, you would replace:
For all available images, refer to the matrix above or Astronomer's Docker Registry.
Step 4. Test your Image Locally
Now, test your changes locally via the Astronomer CLI by running:
$ astro dev stop, then
$ astro dev start
If your image does not build successfully, it's likely that you're either missing additional dependencies or one of your packages is not named properly. Add and modify as needed.
For help from our team, reach out to Astronomer Support.
Step 5. Push to Astronomer
If your image does build successfully, you're ready to push it to your Airflow Deployment Astronomer.
To do so, trigger your CI/CD process or simply run:
For more information on deploying to Astronomer, refer to Deploy to Astronomer via the CLI.
Cancel Airflow Upgrade Initialization
If you begin the upgrade process for your Airflow Deployment and would like to cancel it, you can do so at any time either via the Software UI or CLI as long as you have NOT changed the Astronomer Certified Image in your
Dockerfile and deployed it.
Via the Software UI, select Cancel next to Airflow Version.
Via the Astronomer CLI, run:
astro deployment airflow upgrade --cancel --deployment-id=<deployment-id>
For example, if a user cancels an initialized upgrade from Airflow 1.10.7 to Airflow 1.10.12 via the CLI, they would see the following:
astro deployment airflow upgrade --cancel --deployment-id=ckguogf6x0685ewxtebr4v04x
Airflow upgrade process has been successfully canceled. Your Deployment was not interrupted and you are still running Airflow 1.10.7.
Canceling the Airflow upgrade process will NOT interrupt or otherwise impact your Airflow Deployment or code that's running with it. To re-initialize an upgrade, follow the steps above.
Upgrade to an AC Hotfix Version
To upgrade to the latest hotfix version of Astronomer Certified, replace the image referenced in your
Dockerfile with a pinned version that specifies a particular hotfix.
If you're looking for the latest Astronomer Certified 1.10.10, for example, you would:
- Check the AC 1.10.10 Changelog.
- Identify the latest patch (e.g.
- Pin the Astronomer Certified image in your Dockerfile to that patch version.
In this case, that would be:
FROM quay.io/astronomer/ap-airflow:1.10.10-5-buster-onbuild (Debian).
Note: If you're pushing code to an Airflow Deployment via the Astronomer CLI and install a new Astronomer Certified image for the first time without pinning a specific patch, the latest version available will automatically be pulled.
If a patch release becomes available after you've already built an Astronomer Certified image for the first time, subsequent code pushes will not automatically pull the latest corresponding patch. You must follow the process above to pin your image to a particular version.