Skip to main content

deployments

A Deployment is an Astro Runtime environment that is powered by the core components of Apache Airflow. In a Deployment, you can deploy and run DAGs, configure worker resources, and view metrics.

You can choose to use an existing Deployment, or create a new one. However, you must have at least one Deployment to complete your migration.

Before you create your Deployment, copy the following information from your source Airflow environment:

  • Environment name
  • Airflow version
  • Environment class or size
  • Number of schedulers
  • Minimum number of workers
  • Maximum number of workers
  • Execution role permissions
  • Airflow configurations
  • Environment variables
Alternative setup for Astro Hybrid

This setup varies slightly for Astro Hybrid users. See Deployment settings for all configurations related to Astro Hybrid Deployments.

  1. In the Cloud UI, select a Workspace.

  2. On the Deployments page, click Deployment.

  3. Complete the following fields:

    • Name: Enter the name of your source Airflow environment.

    • Astro Runtime: Select the Runtime version that's based on the Airflow version in your source Airflow environment. See the following table to determine which version of Runtime to use. Where exact version matches are not available, the nearest Runtime version is provided with its supported Airflow version in parentheses.

      Airflow VersionRuntime Version
      2.03.0.4 (Airflow 2.1.1)¹
      2.24.2.9 (Airflow 2.2.5)
      2.46.3.0 (Airflow 2.4.3)
    info

    ¹The earliest available Airflow version on Astro Runtime is 2.1.1. There are no known risks for upgrading directly from Airflow 2.0 to Airflow 2.1.1 during migration. For a complete list of supported Airflow versions, see Astro Runtime release and lifecycle schedule.

    • Description: (Optional) Enter a description for your Deployment.

    • Cluster: Choose whether you want to run your Deployment in a Standard cluster or Dedicated cluster. If you don't have specific networking or cloud requirements, Astronomer recommends using the default Standard cluster configurations.

      To configure and use dedicated clusters, see Create a dedicated cluster. If you don't have the option of choosing between standard or dedicated, that means you are an Astro Hybrid user and must choose a cluster that has been configured for your Organization. See Manage Hybrid clusters.

    • Executor: Choose the same executor as in your source Airflow environment.

    • Scheduler: Set your scheduler size in Astronomer Units (AU). An AU is a unit of CPU and memory allocated to each scheduler in a Deployment. Use the following table to determine how many AUs you need based on the size of your source Airflow environment.

      Environment sizeScheduler sizeCPU / memory
      Small (Up to ~50 DAGs)Small1vCPU, 2GiB
      Medium (Up to ~250 DAGs)Medium2vCPU, 4GiB²
      Large (Up to ~1000 DAGs)Large4vCPU, 8GiB²
    info

    ²Some of the following recommendations for CPU and memory might be less than what you currently allocate to Airflow components in your source environment. If you notice significant performance differences or your Deployment on Astro parses DAGs more slowly than your source Airflow environment, adjust your resource use on Astro. See Deployment settings

    • Worker Type: Select the worker type for your default worker queue. See Worker queues.
    • Min / Max # Workers: Set the same minimum and maximum worker count as in source Airflow environment.
    • KPO Pods: (Optional) If you use the KubernetesPodOperator or Kubernetes Executor, set limits on how many resources your tasks can request.
  4. Click Create Deployment.

  5. Specify any system-level environment variables as Astro environment variables. See Environment variables.

  6. Set an email to receive alerts from Astronomer support about your Deployments. See Add Or Delete A Deployment Alert Email.

Was this page helpful?

Sign up for Developer Updates

Get a summary of new Astro features once a month.

You can unsubscribe at any time.
By proceeding you agree to our Privacy Policy, our Website Terms and to receive emails from Astronomer.