Skip to main content

Astro Runtime Release Notes

Overview

Astro Runtime is a Docker image built and published by Astronomer that extends the Apache Airflow project to provide a differentiated data orchestration experience. This document provides a summary of all changes made to each available version of Astro Runtime.

For instructions on how to upgrade, read Upgrade Astro Runtime. For general product release notes, go to Astro Release Notes. If you have any questions or a bug to report, reach out to Astronomer Support.

Astro Runtime 5.0.1

  • Rlease date: May 9, 2022
  • Airflow version: 2.3.0

Astronomer Providers 1.2.0

Astro Runtime 5.0.1 includes v1.2.0 of the astronomer-providers package (CHANGELOG). This release includes 5 new deferrable operators:

- `DataprocSubmitJobOperatorAsync`
- `EmrContainerSensorAsync`
- `EmrStepSensorAsync`
- `EmrJobFlowSensorAsync`
- `LivyOperatorAsync`

To access the source code of this package, visit the Astronomer Providers GitHub repository.

Additional Improvements

Astro Runtime 5.0.0

  • Release date: April 30, 2022
  • Airflow version: 2.3.0

Support for Airflow 2.3 & Dynamic Task Mapping

Astro Runtime 5.0.0 provides support for Airflow 2.3.0, which is a significant open source release. The most notable new features in Airflow 2.3.0 are:

For more information on Airflow 2.3, see "Apache Airflow 2.3 — Everything You Need to Know" by Astronomer.

Astro Runtime 4.2.6

  • Release date: April 19, 2022
  • Airflow version: 2.2.5

Additional Improvements

  • Add initial support for Astro Runtime on Google Cloud Platform (GCP), including logging in Google Cloud Storage (GCS). Support for Astro on GCP is coming soon.

Astro Runtime 4.2.5

  • Release date: April 11, 2022
  • Airflow version: 2.2.5

Bug Fixes

  • Bug Fix: Apply a new constraints file to fix a version incompatibility error with apache-airflow-providers-elasticsearch that made task logs inaccessible to users in the Airflow UI. This change was required by Astronomer Software and did not impact users on Astro.

Astro Runtime 4.2.4

  • Release date: April 6, 2022
  • Airflow version: 2.2.5

Support for Airflow 2.2.5

Astro Runtime 4.2.2 includes support for Apache Airflow 2.2.5, which exclusively contains bug fixes and performance improvements. For details on the release, read the Airflow 2.2.5 changelog.

Astro Runtime 4.2.1

  • Release date: March 28, 2022
  • Airflow version: 2.2.4

New Deferrable Operators

Astro Runtime 4.2.1 upgrades the astronomer-providers package to v1.1.0 (CHANGELOG). In addition to bug fixes and performance improvements, this release includes 7 new deferrable operators:

- `S3KeySizeSensorAsync`
- `S3KeysUnchangedSensorAsync`
- `S3PrefixSensorAsync`
- `GCSObjectsWithPrefixExistenceSensorAsync`
- `GCSObjectUpdateSensorAsync`
- `GCSUploadSessionCompleteSensorAsync`
- `BigQueryTableExistenceSensorAsync`

For more information about deferrable operators and how to use them, see Deferrable Operators. To access the source code of this package, see the Astronomer Providers GitHub repository.

Additional Improvements

Astro Runtime 4.2.0

  • Release date: March 10, 2022
  • Airflow version: 2.2.4

New Astronomer Providers Package

The astronomer-providers package is now installed on Astro Runtime by default. This package is an open source collection of Apache Airflow providers and modules that is maintained by Astronomer. It includes deferrable versions of popular operators such as ExternalTaskSensor, DatabricksRunNowOperator, and SnowflakeOperator.

For more information, see Deferrable Operators. To access the source code of this package, see the Astronomer Providers GitHub repository.

Additional Improvements

Astro Runtime 4.1.0

  • Release date: February 22, 2022
  • Airflow version: 2.2.4

Support for Airflow 2.2.4

Astro Runtime 4.1.0 includes support for Apache Airflow 2.2.4, which exclusively contains bug fixes and performance improvements. For details on the release, read the Airflow 2.2.4 changelog.

Astro Runtime 4.0.11

  • Release date: February 14, 2022
  • Airflow version: 2.2.3

Additional Improvements

  • Upgraded the openlineage-airflow library to v0.5.2

Astro Runtime 4.0.10

  • Release date: February 9, 2022
  • Airflow version: 2.2.3

New Deferrable Operators Now Available

Astro Runtime now also includes the following operators:

  • KubernetesPodOperatorAsync
  • HttpSensorAsync
  • SnowflakeOperatorAsync
  • FileSensorAsync

These are all deferrable operators built by Astronomer and available exclusively on Astro Runtime. They are pre-installed into the Astro Runtime Docker image and ready to use.

Additional improvements

Additional Improvements

  • To support an enhanced logging experience on Astro, the apache-airflow-providers-elasticsearch provider package is now installed by default.

Astro Runtime 4.0.9

  • Release date: January 19, 2022
  • Airflow version: 2.2.3

Additional Improvements

Astro Runtime 4.0.8

  • Release date: December 21, 2021
  • Airflow version: 2.2.3

Support for Airflow 2.2.3

Astro Runtime 4.0.8 includes support for Airflow 2.2.3.

Airflow 2.2.3 exclusively contains bug fixes, including:

  • Fix for a broken link to task logs in the Gantt view of the Airflow UI (#20121)
  • Replace references to "Execution Date" in the Task Instance and DAG Run tables of the Airflow UI with "Logical Date" (#19063)
  • Fix problem whereby requests to the DAGRun endpoint of Airflow's REST API would return a 500 error if DAG run is in state skipped (#19898)
  • Fix problem where task logs in Airflow UI showed incorrect timezone (#19401)
  • Fix problem where the Connections form in the Airflow UI showed incorrect field names (#19411)

Bug fixes

  • Disabled the Pause button for astronomer_monitoring_dag, which cannot be disabled and helps the Astronomer team monitor the health of your Deployment.

Astro Runtime 4.0.7

  • Release date: December 15, 2021
  • Airflow version: 2.2.2

Astronomer Monitoring DAG

Astro Runtime 4.0.7 includes a monitoring DAG that is pre-installed in the Docker image and enabled for all customers. In addition to existing Deployment health and metrics functionality, this DAG allows the Astronomer team to better monitor the health of your Data Plane by enabling real-time visibility into whether your Workers are healthy and tasks are running.

The astronomer_monitoring_dag runs a simple bash task every 5 minutes to ensure that your Airflow Scheduler and Workers are functioning as expected. If the task fails twice in a row or is not scheduled within a 10-minute interval, Astronomer support receives an alert and will work with you to troubleshoot.

Because this DAG is essential to Astro's managed service, your organization will not be charged for its task runs. For the same reasons, this DAG can't be modified or disabled via the Airflow UI. To modify how frequently this DAG runs, you can specify an alternate schedule as a cron expression by setting AIRFLOW_MONITORING_DAG_SCHEDULE_INTERVAL as an environment variable.

Astro Runtime 4.0.6

  • Release date: December 2, 2021
  • Airflow version: 2.2.2

Additional Improvements

  • User-supplied airflow.cfg files are no longer valid in Astro projects. Environment variables are now the only valid method for setting Airflow configuration options.

Bug Fixes

  • Fixed an issue where the Browse menu of the Airflow UI was hidden in some versions of Astro Runtime

Astro Runtime 4.0.5

  • Release date: November 29, 2021
  • Airflow version: 2.2.2

Bug Fixes

  • Fixed an issue where Astro's S3 logging hook prevented users from setting up S3 as a custom XCom backend

Astro Runtime 4.0.4

  • Release date: November 19, 2021
  • Airflow version: 2.2.2

Bug Fixes

  • Fixed an issue where DAG run and task instance records didn't show up as expected in the Airflow UI

Astro Runtime 4.0.3

  • Release date: November 15, 2021
  • Airflow version: 2.2.2

Additional Improvements

  • Added support for Airflow 2.2.2, which includes a series of bug fixes for timetables, DAG scheduling, and database migrations. Most notably, it resolves an issue where some DAG runs would be missing in the Airflow UI if catchup=True was set.

Bug Fixes

  • Fixed an issue where the Astro-themed Airflow UI was not present in local development

Astro Runtime 4.0.2

  • Release date: October 29, 2021
  • Airflow version: 2.2.1

Additional Improvements

  • Added support for Airflow 2.2.1, which includes a series of bug fixes that address intermittent problems with database migrations from Airflow 2.1 to Airflow 2.2

Astro Runtime 4.0.1

  • Release date: October 26, 2021
  • Airflow version: 2.2.0

Bug Fixes

  • Fixed an issue where worker pods were stuck in a terminating state when scaling down
  • Fixed an issue where the Airflow UI navbar and footer did not show the correct running version of Astro Runtime

Astro Runtime 4.0.0

  • Release date: October 12, 2021
  • Airflow version: 2.2.0

Support for Airflow 2.2.0

Astro Runtime 4.0.0 is a significant release that supports and enhances Apache Airflow 2.2.0, an exciting milestone in the open source project. Most notably, this release introduces custom timetables and deferrable operators.

Custom Timetables

Timetables represent a powerful new framework that allows Airflow users to create custom schedules using Python. In an effort to provide more flexibility and address known limitations imposed by cron, timetables use an intuitive data_interval that, for example, allows you to schedule a DAG to run daily on Monday through Friday, but not on the weekend. Timetables can be easily plugged into existing DAGs, which means that it's easy to create your own or use community-developed timetables in your project.

In addition to supporting the timetables framework, the team at Astronomer has built a TradingHoursTimetable that's ready to use in Runtime 4.0.0. You can use this timetable to run a DAG based on whether or not a particular global market is open for trade.

For more information on using timetables, read the Apache Airflow Documentation.

Deferrable Operators

Deferrable operators are a new type of Airflow operator that promises improved performance and lower resource costs. While standard operators and sensors take up a Worker slot even when they are waiting for an external trigger, deferrable operators are designed to suspend themselves and free up that Worker slot while they wait. This is made possible by a new, lightweight Airflow component called the Triggerer.

Existing Airflow operators have to be re-written according to the deferrable operator framework. In addition to supporting those available in the open source project, Astronomer has built an exclusive collection of deferrable operators in Runtime 4.0.0. This collection includes the DatabricksSubmitRunOperator, the DatabricksRunNowOperator, and the ExternalTaskSensor. These are designed to be drop-in replacements for corresponding operators currently in use.

As part of supporting deferrable operators, the Triggerer is now available as a fully managed component on Astro. This means that you can start using deferrable operators in your DAGs as soon as you're ready. For more general information on deferrable operators, as well as how to use Astronomer's exclusive deferrable operators, read Deferrable Operators.

Astro Runtime 3.0.4

  • Release date: October 26, 2021
  • Airflow version: 2.1.1

Bug Fixes

  • Fixed an issue where worker pods were stuck in a terminating state when scaling down (backported from Runtime 4.0.1)

Astro Runtime 3.0.3

  • Release date: September 22, 2021
  • Airflow version: 2.1.1

Bug fixes

  • Fixed an issue where requests to Airflow's REST API with a temporary authentication token failed
  • Fixed an issue introduced in Runtime 3.0.2 where astro dev commands in the Astro CLI did not execute correctly

Astro Runtime 3.0.2

  • Release date: September 17, 2021
  • Airflow version: 2.1.1

Bug fixes

  • Fixed a series of issues that prevented task logs from appearing in the Airflow UI by implementing a custom task logging handler that does not interfere with AWS credentials or connections configured by users

Astro Runtime 3.0.1

  • Release date: September 1, 2021
  • Airflow version: 2.1.1

Additional improvements

  • Upgraded the default Python version to 3.9.6
  • Added a link to Astro documentation in the Airflow UI

Bug fixes

  • Removed nonfunctional security and user profile elements from the Airflow UI
  • The Airflow UI now shows the correct version of Astro Runtime in the footer

Astro Runtime 3.0.0

  • Release date: August 12, 2021
  • Airflow version: 2.1.1

Additional improvements

  • The Webserver is now the only Airflow component with access to logs, which reduces the risk of exposing sensitive information in logs (commit)
  • Added support for Python 3.9 (commit)
  • token keys in connections are now marked as masked by default (commit)

Bug fixes

  • Fixed module vulnerabilities exposed by yarn audit (commit)
  • Fixed an issue where tasks would fail when running with run_as_user (commit)
  • Fixed an issue where tasks would fail when running with CeleryKubernetesExecutor (commit)