Skip to main content

Deploy code to Astro

Using the Astro CLI to push your Astro project, including your DAG code, to a Deployment is the foundation for managing changes on Astro. Astro supports two types of deployment options:

  • astro deploy: This command pushes every file in your Astro project to all Airflow components in your Deployment. This includes your Dockerfile, DAGs, plugins, and all Python and OS-level packages.
  • astro deploy --dags: This command pushes only the code that exists in the /dags directory of your Astro project to a running Deployment on Astro. When you only need to push changes to your DAGs, running this command is a faster development experience than running astro deploy since it does not require installing your dependencies.

To run astro deploy --dags, you must first enable the DAG-only deploys feature for each Deployment.

Follow the steps in this document to manually push your Astro project to a Deployment. For production environments, Astronomer recommends automating all code deploys with CI/CD. See CI/CD.

To exclude specific files during a code deploy, see Ignore files with .airflowignore.

caution

If you're using Astro Runtime 6.0.5 or later on a Mac computer with an M1 chip, you must install Astro CLI 1.4.0 or later or your deploys will fail. See Install the CLI.

Prerequisites​

  • The Astro CLI is installed in an empty directory. If you're using an Apple M1 system with Astro Runtime 6.0.4 or later for local development, you must install Astro CLI 1.4.0 or later to deploy to Astro.
  • An Astro Workspace with at least one Deployment.
  • An Astro project.
  • Docker.

Step 1: Authenticate to Astro​

Run the following command to authenticate to Astro:

astro login

After running this command, you are prompted to open your web browser and log in to the Cloud UI. Once you complete this login, you are automatically authenticated to the CLI.

tip

If you have Deployment API key credentials set as OS-level environment variables on your local machine, you can deploy directly to Astro without needing to manually authenticate. This setup is required for automating code deploys with CI/CD.

Step 2: Push your Astro project to an Astro Deployment​

To deploy your Astro project, run:

astro deploy

This command returns a list of Deployments available in your Workspace and prompts you to pick one.

After you select a Deployment, the CLI parses your DAGs to ensure that they don't contain basic syntax and import errors. This test is equivalent to the one that runs during astro dev parse in a local Airflow environment. If any of your DAGs fail this parse, the deploy to Astro also fails.

If your code passes the parse, the Astro CLI builds all files in your Astro project directory into a new Docker image and then pushes the image to your Deployment on Astro. If the DAG-only deploy feature is enabled for your Deployment, the /dags directory is excluded from this Docker image and pushed separately. To force a deploy even if your project has DAG errors, you can run astro deploy --force.

tip

To validate your code before deploying it to Astro, you can run astro deploy --pytest. Adding the --pytest flag makes the CLI run all tests in your project's tests directory using pytest. If any of these tests fail, your code deploy also fails. This can help you prevent your team from deploying DAGs to Astro that contain errors.

For more information about using Pytest, see Test and troubleshoot locally.

Step 3: Validate your changes​

After the deploy completes, Docker image information for your Deployment is available in the Image tag field in the footer of the Airflow UI. Depending on how your organization deploys to Astro, the Image tag field displays a unique identifier generated by a Continuous Integration (CI) tool or a timestamp generated by the Astro CLI on astro deploy. The Image tag field in the Airflow UI footer identifies the Docker image running on the webserver of your Deployment. This information can help you determine if your deploy was successful, but it does not identify the Docker image running on your scheduler, triggerer, or workers. To confirm a code push with complete certainty, contact Astronomer support.

  1. In the Cloud UI, select a Workspace and then select the Deployment you pushed code to.

  2. Click Open Airflow.

  3. Scroll to the bottom of the page and view the Image tag information in the footer:

    Docker image information

info

Improved visibility into the health of your Deployment and the status of deploys is coming soon. If you're interested in early access to this feature, contact your Astronomer representative.

What happens during a code deploy​

When you deploy code to Astro, your Astro project is built into a Docker image. This includes system-level dependencies, Python-level dependencies, and your Dockerfile. It does not include any of the metadata associated with your local Airflow environment, including task history and Airflow connections or variables that were set locally. This Docker image is then pushed to all containers running Apache Airflow on Astro. If the DAG-only deploy feature is enabled for your Deployment, the /dags directory is excluded from this Docker image and is pushed separately.

Deploy code

With the exception of the Airflow webserver and some Celery workers, Kubernetes gracefully terminates all containers during this process. This forces them to restart and begin running your latest code.

If you deploy code to a Deployment that is running a previous version of your code, then the following happens:

  • Tasks that are running will continue to execute on existing Celery workers and will not be interrupted unless the task does not complete within 24 hours of the code deploy.

  • One or more new worker(s) will be created alongside your existing workers and immediately start executing scheduled tasks based on your latest code.

    These new workers will execute downstream tasks of DAG runs that are in progress. For example, if you deploy to Astronomer when Task A of your DAG is running, Task A will continue to run on an old Celery worker. If Task B and Task C are downstream of Task A, they will both be scheduled on new Celery workers running your latest code.

    This means that DAG runs could fail due to downstream tasks running code from a different source than their upstream tasks. DAG runs that fail this way need to be fully restarted from the Airflow UI so that all tasks are executed based on the same source code.

Astronomer sets a grace period of 24 hours for all workers to allow running tasks to continue executing. This grace period is not configurable. If a task does not complete within 24 hours, its worker will be terminated. Airflow will mark the task as a zombie and it will retry according to the task's retry policy. This is to ensure that our team can reliably upgrade and maintain Astro as a service.

tip

If you want to force long-running tasks to terminate sooner than 24 hours, specify an execution_timeout in your DAG's task definition.

Deploy DAGs only​

caution

This feature in Public Preview.

To push only DAGs to Astro for a faster development experience, you must enable the feature for each Deployment. You only need to enable the feature once. When it is enabled, you must still run astro deploy when you make a change to any file in your Astro project that is not in the dags directory.

Enabling DAG-only deploys on Astro has a few benefits:

  • DAG-only deploys are significantly faster than running astro deploy when you have only made changes to your DAGs.
  • When you run astro deploy --dags, the workers and schedulers in your Deployment will pick up your changes gracefully and will not restart. This results in a more efficient use of running workers and no downtime for your Deployment.
  • You can have different sets of users deploy project changes versus DAG changes. See DAG-based workflows for how you can set this up in your CI/CD pipelines.

Enable DAG-only deploys on a Deployment​

Before you complete this setup, ensure that you have access to the Deployment's Astro project and can trigger deploys from your current computer.

  1. Run the following command to enable the feature:

    astro deployment update --dag-deploy enable

  2. When the prompt appears in the Astro CLI, select the Deployment where you want to enable the feature. Running tasks will not be interrupted, but new tasks will not be scheduled until you trigger your first DAG-only deploy.

  3. Open your Deployment's Astro project.

  4. Run the following command finalize the setup and trigger a DAG-only deploy to your Deployment: 

    astro deploy --dags

    If you don't trigger a deploy after enabling the feature, your Deployment cannot schedule new tasks.

To disable the DAG-only deploy feature, contact Astronomer support.

Trigger a DAG-only deploy​

Triggering a DAG-only deploy pushes DAGs to Astro and mounts them to the workers and schedulers in your Deployment. DAG-only deploys do not disrupt running tasks and do not cause any components to restart when you push code. If you deploy changes to a DAG that is currently running, active task runs finish executing according to the code from before you triggered a deploy. New task runs are scheduled using the code from your latest deploy.

Run the following command to deploy only your dags directory to a Deployment:

astro deploy --dags

Disable DAG-only deploys on a Deployment​

If you have Workspace Admin permissions you can turn off DAG-only deploys for a Deployment at any time if your organization doesn't benefit from faster deploys or prefers a deployment method that is exclusively based on building and deploying your Astro project as a Docker image. When you turn off DAG-only deploys, the way Airflow and Astro read your DAGs changes, and all existing DAGs are removed from your Deployments and they might not appear in the Airflow UI.

To make sure you can continue to access to your data, Astronomer recommends deploying to Astro immediately after you turn off DAG-only deploy functionality. To determine if turning off DAG-only deploy functionality is the right choice for your organization, contact Astronomer support.

  1. Run the following command to turn off DAG-only deploys:

    astro deployment update --dag-deploy disable

  2. Run the following command to deploy all of the files in your Astro project as a Docker image:

    astro deploy

Deploy a prebuilt Docker image​

By default, running astro deploy with the Astro CLI builds your Astro project into a Docker image and deploys it to Astro. In some cases, you might want to skip the build step and deploy a prebuilt Docker image instead.

Deploying a prebuilt Docker image allows you to:

  • Test a single Docker image across Deployments instead of rebuilding it each time.
  • Reduce the time it takes to deploy. If your Astro project has a number of packages that take a long time to install, it can be more efficient to build it separately.
  • Specify additional mounts and arguments in your project, which is required for setups such as installing Python packages from private sources.

To deploy your Astro project as a prebuilt Docker image:

  1. Run docker build from an Astro project directory or specify the command in a CI/CD pipeline. This Docker image must be based on Astro Runtime and be available in a local Docker registry. If you run this command on an Apple M1 computer or on a computer with an ARM64 processor, you must specify --platform=linux/amd64 or else the deploy will fail. The Astro data plane requires an AMD64-based image and does not support ARM64 architecture.
  2. Optional. Test your Docker image in a local Airflow environment by adding the --image-name <image-name> flag to any of the following commands:
    • astro dev start
    • astro dev restart
    • astro dev parse
    • astro dev pytest
  3. Run astro deploy --image-name <image-name> or specify the command in a CI/CD pipeline.

For more information about this command, see the CLI command reference.

info

If you build an AMD64-based image and run astro deploy from an Apple M1 computer, you might see a warning in your terminal. You can ignore the warning.

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform 
(linux/arm64/v8) and no specific platform was requested