Skip to main content

deployments

A Deployment is an Astro Runtime environment that is powered by the core components of Apache Airflow. In a Deployment, you can deploy and run DAGs, configure worker resources, and view metrics.

Before you create your Deployment, copy the following information from your source Airflow environment:

  • Environment name
  • Airflow version
  • Environment class or size
  • Number of schedulers
  • Minimum number of workers
  • Maximum number of workers
  • Execution role permissions
  • Airflow configurations
  • Environment variables
  1. In the Cloud UI, select a Workspace.

  2. On the Deployments page, click Deployment.

  3. Complete the following fields:

    • Name: Enter the name of your source Airflow environment.
    • Astro Runtime: Select the Runtime version that's based on the Airflow version in your source Airflow environment. See the following table to determine which version of Runtime to use. Where exact version matches are not available, the nearest Runtime version is provided with its supported Airflow version in parentheses.
    Airflow VersionRuntime Version
    2.03.0.4 (Airflow 2.1.1)*
    2.24.2.9 (Airflow 2.2.5)
    2.46.3.0 (Airflow 2.4.3)

    *The earliest available Airflow version on Astro Runtime is 2.1.1. There are no known risks for upgrading directly from Airflow 2.0 to Airflow 2.1.1 during migration. For a complete list of supported Airflow versions, see Astro Runtime release and lifecycle schedule.

    • Description: (Optional) Enter a description for your Deployment.
    • Cluster: Select the Astro cluster where you want to create this Deployment.
    • Worker Type: Select the worker type for your default worker queue. See Worker queues.
    • Max Tasks Per Worker: (Optional) Set maximum number of tasks that a single worker can run at a time.
    • Worker Count: Set the same minimum and maximum worker count as in source Airflow environment.
    • Scheduler Count: Set to the same number of schedulers as in your source Airflow environment.
    • Scheduler Size: Set your scheduler size in Astronomer Units (AU). An AU is a unit of CPU and memory allocated to each scheduler in a Deployment. Use the following table to determine how many AUs you need based on the size of your source Airflow environment.
    Environment sizeAUsCPU / memory
    Small (Up to ~50 DAGs)5.5vCPU, 1.88GiB
    Medium (Up to ~250 DAGs)101vCPU, 3.75GiB*
    Large (Up to ~1000 DAGs)151.5vCPU, 5.64GiB*

*Some of the following recommendations for CPU and memory might be less than what you currently allocate to Airflow components in your source environment. If you notice significant performance differences or your Deployment on Astro parses DAGs more slowly than your source Airflow environment, adjust your resource usage on Astro. See Configure Deployment

  1. Click Create Deployment.
  2. Specify any system-level environment variables as Astro environment variables. See Environment variables.
  3. Set an email to receive alerts from your Deployment. See Add Or Delete A Deployment Alert Email.

Sign up for Developer Updates

Get a summary of new Astro features once a month.

You can unsubscribe at any time.
By proceeding you agree to our Privacy Policy, our Website Terms and to receive emails from Astronomer.