Deploy code to Astro

Using the Astro CLI to push your Astro project, including your DAG code, to a Deployment is the foundation for managing changes on Astro. Astro supports two types of deployment options:

• astro deploy: This command pushes every file in your Astro project to all Airflow components in your Deployment. This includes your Dockerfile, DAGs, plugins, and all Python and OS-level packages.
• astro deploy --dags: This command pushes only the code that exists in the /dags directory of your Astro project to a running Deployment on Astro. When you only need to push changes to your DAGs, running this command is a faster development experience than running astro deploy since it does not require installing your dependencies.

To run astro deploy --dags, you must first enable the DAG-only deploys feature for each Deployment.

Follow the steps in this document to manually push your Astro project to a Deployment. For production environments, Astronomer recommends automating all code deploys with CI/CD. See Choose a CI/CD strategy.

To exclude specific files during a code deploy, see Ignore files with .airflowignore.

caution

If you're using Astro Runtime 6.0.5 or later on a Mac computer with an M1 chip, you must install Astro CLI 1.4.0 or later or your deploys will fail. See Install the CLI.

Prerequisites

• The Astro CLI is installed in an empty directory. If you're using an Apple M1 system with Astro Runtime 6.0.4 or later for local development, you must install Astro CLI 1.4.0 or later to deploy to Astro.
• An Astro Workspace with at least one Deployment.
• An Astro project.
• Docker.

Step 1: Authenticate to Astro

Run the following command to authenticate to Astro:

astro login

After running this command, you are prompted to open your web browser and log in to the Cloud UI. Once you complete this login, you are automatically authenticated to the CLI.

tip

If you have Deployment API key credentials set as OS-level environment variables on your local machine, you can deploy directly to Astro without needing to manually authenticate. This setup is required for automating code deploys with CI/CD.

Step 2: Push your Astro project to an Astro Deployment

To deploy your Astro project, run:

astro deploy

This command returns a list of Deployments available in your Workspace and prompts you to pick one.

After you select a Deployment, the CLI parses your DAGs to ensure that they don't contain basic syntax and import errors. This test is equivalent to the one that runs during astro dev parse in a local Airflow environment. If any of your DAGs fail this parse, the deploy to Astro also fails.

If your code passes the parse, the Astro CLI builds all files in your Astro project directory into a new Docker image and then pushes the image to your Deployment on Astro. If the DAG-only deploy feature is enabled for your Deployment, the /dags directory is excluded from this Docker image and pushed separately. To force a deploy even if your project has DAG errors, you can run astro deploy --force.

If your internet connection has slow upload speeds, the deploy might fail with the error error parsing HTTP 408 response body: unexpected end of JSON input. If you're only deploying a change to your DAGs, set up DAG only deploys to deploy your changes over a slow connection. To make a non-DAG update over a slow connection, you can deploy with CI/CD.

tip

To validate your code before deploying it to Astro, you can run astro deploy --pytest. Adding the --pytest flag makes the CLI run all tests in your project's tests directory using pytest. If any of these tests fail, your code deploy also fails. This can help you prevent your team from deploying DAGs to Astro that contain errors.

For more information about using Pytest, see Test and troubleshoot locally.

Step 3: Validate your changes

When you start a code deploy to Astro, the status of the Deployment is DEPLOYING until it is determined that the underlying Airflow components are running the latest version of your code. During this time, you can hover over the status indicator to determine whether your entire Astro project or only DAGs were deployed .

When the deploy completes, the Docker Image and DAG Bundle Version fields in the Cloud UI are updated depending on the type of deploy you completed.

• The Docker Image field displays a unique identifier generated by a Continuous Integration (CI) tool or a timestamp generated by the Astro CLI after you complete an image deploy.
• The DAG Bundle Version field displays a unique timestamp generated by the Astro CLI after you complete an image deploy or a DAG-only deploy. If you do not have DAG-only deploys enabled, the DAG bundle version field isn't populated.

To confirm a deploy was successful, verify that the running versions of your Docker image and DAG bundle have been updated.

1. In the Cloud UI, select a Workspace and then select a Deployment.
2. Review the information in the Docker Image and DAG bundle version fields to determine the Deployment code version.

What happens during a code deploy

When you deploy code to Astro, your Astro project is built into a Docker image. This includes system-level dependencies, Python-level dependencies, and your Dockerfile. It does not include any of the metadata associated with your local Airflow environment, including task history and Airflow connections or variables that were set locally. This Docker image is then pushed to all containers running Apache Airflow on Astro. If the DAG-only deploy feature is enabled for your Deployment, the /dags directory is excluded from this Docker image and is pushed separately.

With the exception of the Airflow webserver and some Celery workers, Kubernetes gracefully terminates all containers during this process. This forces them to restart and begin running your latest code.

If you deploy code to a Deployment that is running a previous version of your code, then the following happens:

• Tasks that are running will continue to execute on existing Celery workers and will not be interrupted unless the task does not complete within 24 hours of the code deploy.

• One or more new worker(s) will be created alongside your existing workers and immediately start executing scheduled tasks based on your latest code.

  These new workers will execute downstream tasks of DAG runs that are in progress. For example, if you deploy to Astronomer when Task A of your DAG is running, Task A will continue to run on an old Celery worker. If Task B and Task C are downstream of Task A, they will both be scheduled on new Celery workers running your latest code.

  This means that DAG runs could fail due to downstream tasks running code from a different source than their upstream tasks. DAG runs that fail this way need to be fully restarted from the Airflow UI so that all tasks are executed based on the same source code.

Astronomer sets a grace period of 24 hours for all workers to allow running tasks to continue executing. This grace period is not configurable. If a task does not complete within 24 hours, its worker will be terminated. Airflow will mark the task as a zombie and it will retry according to the task's retry policy. This is to ensure that our team can reliably upgrade and maintain Astro as a service.

tip

If you want to force long-running tasks to terminate sooner than 24 hours, specify an execution_timeout in your DAG's task definition.

Deploy DAGs only

To push only DAGs to Astro for a faster development experience, you must enable the feature for each Deployment. You only need to enable the feature once. When it is enabled, you must still run astro deploy when you make a change to any file in your Astro project that is not in the dags directory.

Enabling DAG-only deploys on Astro has a few benefits:

• DAG-only deploys are significantly faster than running astro deploy when you have only made changes to your DAGs.
• When you run astro deploy --dags, the workers and schedulers in your Deployment will pick up your changes gracefully and will not restart. This results in a more efficient use of running workers and no downtime for your Deployment.
• You can have different sets of users deploy project changes versus DAG changes. See DAG-based templates for how you can set this up in your CI/CD pipelines.
• You can use it to update your DAGs when you have slow upload speed with your internet connection.

Enable DAG-only deploys on a Deployment

Before you enable DAG-only deploys on a Deployment, ensure that you have access to the Deployment's Astro project and can trigger deploys from your current computer with the Astro CLI. You cannot enable the DAG-only deploy feature in the Cloud UI.

1. Run the following command to enable the feature:

   astro deployment update --dag-deploy enable

2. When the prompt appears in the Astro CLI, select the Deployment where you want to enable the feature. Running tasks will not be interrupted, but new tasks will not be scheduled until you trigger your first DAG-only deploy.

3. Open your Deployment's Astro project.

4. Run the following command to finalize the setup and trigger a DAG-only deploy to your Deployment:

   astro deploy --dags

   If you don't trigger a deploy after enabling the feature, your Deployment cannot schedule new tasks.

Trigger a DAG-only deploy

Triggering a DAG-only deploy pushes DAGs to Astro and mounts them to the workers and schedulers in your Deployment. DAG-only deploys do not disrupt running tasks and do not cause any components to restart when you push code. If you deploy changes to a DAG that is currently running, active task runs finish executing according to the code from before you triggered a deploy. New task runs are scheduled using the code from your latest deploy.

Run the following command to deploy only your dags directory to a Deployment:

astro deploy --dags

Disable DAG-only deploys on a Deployment

If you have Workspace Admin permissions, you can turn off DAG-only deploys for a Deployment at any time if your organization doesn't benefit from faster deploys or prefers a deployment method that is exclusively based on building and deploying your Astro project as a Docker image. When you turn off DAG-only deploys, the way Airflow and Astro read your DAGs changes, and all existing DAGs are removed from your Deployments and they might not appear in the Airflow UI.

To make sure you can continue to access to your data, Astronomer recommends deploying to Astro immediately after you turn off the DAG-only deploy feature. To determine if turning off DAG-only deploy functionality is the right choice for your organization, contact Astronomer support.

1. Run the following command to turn off DAG-only deploys:

   astro deployment update --dag-deploy disable

2. Run the following command to deploy all of the files in your Astro project as a Docker image:

   astro deploy

Deploy a prebuilt Docker image

By default, running astro deploy with the Astro CLI builds your Astro project into a Docker image and deploys it to Astro. In some cases, you might want to skip the build step and deploy a prebuilt Docker image instead.

Deploying a prebuilt Docker image allows you to:

• Test a single Docker image across Deployments instead of rebuilding it each time.
• Reduce the time it takes to deploy. If your Astro project has a number of packages that take a long time to install, it can be more efficient to build it separately.
• Specify additional mounts and arguments in your project, which is required for setups such as installing Python packages from private sources.

To deploy your Astro project as a prebuilt Docker image:

1. Run docker build from an Astro project directory or specify the command in a CI/CD pipeline. This Docker image must be based on Astro Runtime and be available in a local Docker registry. If you run this command on an Apple M1 computer or on a computer with an ARM64 processor, you must specify --platform=linux/amd64 or else the deploy will fail. Astro Deployments require an AMD64-based image and do not support ARM64 architecture.
2. Optional. Test your Docker image in a local Airflow environment by adding the --image-name <image-name> flag to any of the following commands:
   • astro dev start
   • astro dev restart
   • astro dev parse
   • astro dev pytest
3. Run astro deploy --image-name <image-name> or specify the command in a CI/CD pipeline.

For more information about this command, see the CLI command reference.

info

If you build an AMD64-based image and run astro deploy from an Apple M1 computer, you might see a warning in your terminal. You can ignore the warning.

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform 
(linux/arm64/v8) and no specific platform was requested

Related documentation

• Choose a CI/CD Strategy for deploying code to Astro
• Develop your project
• Set environment variables