Skip to main content
Version: 0.25

Astronomer Certified Image Architecture

Overview

The Astronomer Certified Docker image for Apache Airflow extends the community-developed Airflow image in a way that makes running Airflow more secure, reliable, and extensible. It is the default image for Airflow Deployments on Astronomer.

This guide provides reference information for the building blocks of Astronomer Certified, as well as information on its release and distribution.

Distribution

Astronomer Certified is distributed both as a Python wheel and a Debian-based Docker image. These distributions vary slightly in scope and dependencies.

The Python wheel and Docker image are functionally identical to open source Apache Airflow. Additionally, they both include performance and stability improvements, including critical bug fixes and security patches.

The Astronomer Certified Docker image is built from the Python wheel and incorporates additional logic that makes it easier for users to both get started and run Airflow at scale. This includes:

  • A robust testing suite that covers performance benchmarking and end-to-end functionality and upgrade testing for certified environments.
  • A built-in directory of example DAGs that leverage Airflow's most powerful features.
  • A collection of pre-installed Airflow provider packages.
  • Full compatibility with the Astronomer Platform.

Diagram of AC distribution scheme

Every supported version of the Astronomer Certified Python wheel is available at pip.astronomer.io. The Dockerfiles for all supported Astronomer Certified images can be found in Astronomer's ap-airflow GitHub repository:

Image Requirements

Running Airflow with the Astronomer Certified Docker image requires specific versions for key system components.

  • Python: 3.6, 3.7, 3.8, 3.9
  • Database: PostgreSQL (11, 12), MySQL (5.7, 8.0+)
  • System Distribution: Debian 10 (Buster)

Note: While the Astronomer Certified Python Wheel supports Python versions 3.6, 3.7, and 3.8, Astronomer Certified Docker images have been tested and built only with Python 3.7. To run Astronomer Certified on Docker with Python 3.6 or 3.8, you can create a custom image with a different Python version specified. For more information, read Change Python Versions.

These requirements are slightly different for running only the Python wheel. For the Python wheel, you can use:

  • Python: 3.6, 3.7, 3.8, 3.9

  • Database: PostgreSQL (9.6, 10, 11, 12, 13), MySQL (5.7, 8+), SQLite (3.15.0+)

  • System Distribution: Debian 10 (Buster)

    For more information on running a Python wheel installation of Astronomer Certified, read Install on a Virtual Machine.

Environment Variables

When an Airflow service is started, it checks a file for runtime environment variables. These are equivalent to values defined in Airflow's airflow.cfg file.

If you run the Astronomer Certified Docker image without the Astronomer platform, environment variables are defined in your Dockerfile. They can be overwritten with a runtime command, such as docker run.

If you're running the Astronomer Certified Docker image with the Astronomer platform, there are a few ways you can configure environment variables. For more information, read Environment Variables.

Astronomer Certified supports the same environment variables as Apache Airflow. For a list of all configurable environment variables, read the Apache Airflow documentation.

The following table lists the essential environment variables used when running Airflow with Astronomer Certified. Only environment variables explicitly defined in the Dockerfile are listed here; all other environment variables have the same default value as they have in OSS Airflow.

VariableDescriptionDefault Value
AIRFLOW_PIP_VERSIONThe version of pip that is used to install Python packages and Airflow itself.19.3.1
AIRFLOW_HOMEFilepath for your Airflow project directory.usr/local/airflow
AIRFLOW__WEBSERVER__BASE_URLThe URL used to access the Airflow UI.http://localhost:8080
ASTRONOMER_USERThe username for your Airflow user.astro
ASTRONOMER_UID The ID for your Airflow user.5000
PIP_NO_CACHE_DIRSpecifies whether to maintain copies of source files when installing via pip.True
PYTHON_MAJOR_MINOR_VERSIONThe version of Python to use for Airflow.3.9

Provider Packages

Starting in version 2.0.0, the Astronomer Certified image includes provider packages that are utilized in some background processes, as well as packages which are commonly used by the Airflow community. The following table contains version information for each provider package installed as part of Astronomer Certified:

Astronomer Certifiedamazonazurecelerycncf.kuberneteselasticsearchftpgooglehttpimapmysqlpostgresredisslacksqlitessh
2.0.01.0.01.1.01.0.01.0.11.0.41.0.01.0.01.0.01.0.01.0.01.0.01.0.01.0.01.0.01.0.0
2.0.21.3.01.3.01.0.11.2.01.0.41.0.12.2.01.1.11.0.11.1.01.0.11.0.13.0.01.0.21.3.0
2.1.01.4.02.0.01.0.11!1.2.11.0.41.1.03.0.02.0.01.0.11.1.01.0.21.0.13.0.01.0.21.3.0
2.1.11!2.0.01!3.0.01!2.0.01!2.0.01!2.0.11!2.0.01!4.0.01!2.0.01!2.0.01!2.0.01!2.0.01!2.0.01!4.0.01!2.0.01!2.0.0
2.1.31!2.1.01!3.1.01!2.0.01!2.0.21!2.0.21!2.0.01!5.0.01!2.0.01!2.0.01!2.1.01!2.0.01!2.0.01!4.0.01!2.0.01!2.1.0
2.1.41!2.2.01!3.1.11!2.1.01!2.0.21!2.0.31!2.0.11!5.1.01!2.0.11!2.0.11!2.1.11!2.2.01!2.0.11!4.0.11!2.0.11!2.1.1
2.2.01!2.2.01!3.2.01!2.1.01!2.0.31!2.0.31!2.0.11!6.0.01!2.0.11!2.0.11!2.1.11!2.3.01!2.0.11!4.1.01!2.0.11!2.2.0
2.2.11!2.2.01!3.2.01!2.1.01!2.0.31!2.0.31!2.0.11!6.0.01!2.0.11!2.0.11!2.1.11!2.3.01!2.0.11!4.1.01!2.0.11!2.2.0
2.2.21!2.3.01!3.3.01!2.1.01!2.1.01!2.1.01!2.0.11!6.1.01!2.0.11!2.0.11!2.1.11!2.3.01!2.0.11!4.1.01!2.0.11!2.3.0
2.2.31!2.3.01!3.3.01!2.1.01!2.1.01!2.1.01!2.0.11!6.1.01!2.0.11!2.0.11!2.1.11!2.3.01!2.0.11!4.1.01!2.0.11!2.3.0
2.2.41!3.0.01!3.6.01!2.1.01!3.0.21!2.2.01!2.0.11!6.4.01!2.0.31!2.2.01!2.2.01!3.0.01!2.0.11!4.2.01!2.1.01!2.4.0

System Dependencies

The Astronomer Certified Docker image includes a number of OS-level dependencies for running basic system processes. These dependencies can be installed in the Python Wheel as described in Install Packages.

Extras

Astronomer Certified includes a few packages that don't have a corresponding provider. These packages are used for basic system functions or optional Airflow functionality. The following list contains all extra packages built into Astronomer Certified by default:

  • async: Provides asynchronous workers for Gunicorn
  • password: Adds support for user password hashing
  • statsd: Adds support for sending metrics to StatsD
  • virtualenv: Adds support for running Python tasks in local virtual environments