deployments

A Deployment is an Astro Runtime environment that is powered by the core components of Apache Airflow. In a Deployment, you can deploy and run DAGs, configure worker resources, and view metrics.

You can choose to use an existing Deployment, or create a new one. However, you must have at least one Deployment to complete your migration.

Before you create your Deployment, copy the following information from your source Airflow environment:

• Environment name
• Airflow version
• Environment class or size
• Number of schedulers
• Minimum number of workers
• Maximum number of workers
• Execution role permissions
• Airflow configurations
• Environment variables

Option 1: Cloud UI

Alternative setup for Astro Hybrid

This setup varies slightly for Astro Hybrid users. See Deployment settings for all configurations related to Astro Hybrid Deployments.

1. In the Cloud UI, select a Workspace.

2. On the Deployments page, click Deployment.

3. Complete the following fields:

   • Name: Enter the name of your source Airflow environment.

   • Astro Runtime: Select the Runtime version that's based on the Airflow version in your source Airflow environment. See the following table to determine which version of Runtime to use. Where exact version matches are not available, the nearest Runtime version is provided with its supported Airflow version in parentheses.

     Airflow Version	Runtime Version
     2.0	3.0.4 (Airflow 2.1.1)¹
     2.2	4.2.9 (Airflow 2.2.5)
     2.4	6.3.0 (Airflow 2.4.3)

   info

   ¹The earliest available Airflow version on Astro Runtime is 2.1.1. There are no known risks for upgrading directly from Airflow 2.0 to Airflow 2.1.1 during migration. For a complete list of supported Airflow versions, see Astro Runtime release and lifecycle schedule.

   • Description: (Optional) Enter a description for your Deployment.

   • Cluster: Choose whether you want to run your Deployment in a Standard cluster or Dedicated cluster. If you don't have specific networking or cloud requirements, Astronomer recommends using the default Standard cluster configurations.

     To configure and use dedicated clusters, see Create a dedicated cluster. If you don't have the option of choosing between standard or dedicated, that means you are an Astro Hybrid user and must choose a cluster that has been configured for your Organization. See Manage Hybrid clusters.

   • Executor: Choose the same executor as in your source Airflow environment.

   • Scheduler: Set your scheduler size in Astronomer Units (AU). An AU is a unit of CPU and memory allocated to each scheduler in a Deployment. Use the following table to determine how many AUs you need based on the size of your source Airflow environment.

     Environment size	Scheduler size	CPU / memory
     Small (Up to ~50 DAGs)	Small	1vCPU, 2GiB
     Medium (Up to ~250 DAGs)	Medium	2vCPU, 4GiB²
     Large (Up to ~1000 DAGs)	Large	4vCPU, 8GiB²

   info

   ²Some of the following recommendations for CPU and memory might be less than what you currently allocate to Airflow components in your source environment. If you notice significant performance differences or your Deployment on Astro parses DAGs more slowly than your source Airflow environment, adjust your resource use on Astro. See Deployment settings

   • Worker Type: Select the worker type for your default worker queue. See Worker queues.
   • Min / Max # Workers: Set the same minimum and maximum worker count as in source Airflow environment.
   • KPO Pods: (Optional) If you use the KubernetesPodOperator or Kubernetes Executor, set limits on how many resources your tasks can request.

4. Click Create Deployment.

5. Specify any system-level environment variables as Astro environment variables. See Environment variables.

6. Set an email to receive alerts from Astronomer support about your Deployments. See Add Or Delete A Deployment Alert Email.

Option 2: Astro CLI & Deployments-as-Code

info

This option is available only on Astro Hybrid.

1. On your local machine, create a directory with the name of the source Airflow environment. In this directory, create a file called config.yaml.

2. Open config.yaml and add the following:

   deployment:
     environment_variables:
       - is_secret: <true-or-false>
         key: <variable-name>
         value: <variable-value>
     configuration:
       name: <deployment-name>
       description: <deployment-description>
       runtime_version: <runtime-version>
       dag_deploy_enabled: false
       scheduler_au: <scheduler-au>
       scheduler_count: <scheduler-count>
       cluster_name: <cluster-name>
       workspace_name: <workspace-name>
     worker_queues:
       - name: default
         max_worker_count: <max-worker-count>
         min_worker_count: <min-worker-count>
         worker_concurrency: 16
         worker_type: <worker-type>
     alert_emails:
       - <alert-email>

3. Replace the placeholder values in the configuration:

   • <true-or-false> / <variable-name> / <variable-value>: Set system-level environment variables for your Deployment and specify whether they should be secret. Repeat this configuration in the file for any additional variables you need to set.

   • <deployment-name>: Enter the same name as your source Airflow environment.

   • <deployment-description>: (Optional) Enter a description for your Deployment.

   • <runtime-version>: Select the Runtime version that's based on the Airflow version in your source Airflow environment. See the following table to determine which version of Runtime to use. Where exact version matches are not available, the nearest Runtime version is provided with its supported Airflow version in parentheses.

     Airflow Version	Runtime Version
     2.0	3.0.4 (Airflow 2.1.1)¹
     2.2	4.2.9 (Airflow 2.2.5)
     2.4	6.3.0 (Airflow 2.4.3)

   info

   ¹The earliest available Airflow version on Astro Runtime is 2.1.1. There are no known risks for upgrading directly from Airflow 2.0 to Airflow 2.1.1 during migration. For a complete list of supported Airflow versions, see Astro Runtime release and lifecycle schedule.

   • <scheduler-au>: Set your scheduler size in Astronomer Units (AU). An AU is a unit of CPU and memory allocated to each scheduler in a Deployment. Use the following table to determine how many AUs you need based on the size of your source Airflow environment.

     Environment size	AUs	CPU / memory
     Small (Up to ~50 DAGs)	5	.5vCPU, 1.88GiB
     Medium (Up to ~250 DAGs)	10	1vCPU, 3.75GiB²
     Large (Up to ~1000 DAGs)	15	1.5vCPU, 5.64GiB²

   info

   ² Some of the following recommendations for CPU and memory are smaller than what you have in the equivalent source environment. Although you might not need more CPU or memory than what's recommended, some environments might parse DAGs slower than in your source Airflow environment to start. Use these recommendations as a starting point, then adjust your resource usage after tracking your performance on Astro.

   • <scheduler-count>: Specify the same number of schedulers as in your source Airflow environment.

   • <cluster-name>: Specify name of the Astro cluster in which you want to create this Deployment.

   • <workspace-name>: The name of the Workspace you created.

   • <max-worker-count>/ <min-worker-count>: Specify the same minimum and maximum worker count as in source Airflow environment.

   • <worker-type>: Specify the worker type for your default worker queue. You can see which worker types are available for your cluster in the Clusters menu of the Cloud UI. If you did not customize the available worker types for your cluster, the default available worker types are:

     AWS	GCP	Azure
     M5.XLARGE	E2-STANDARD-4	STANDARD_D4D_V5

   • <alert-email>: Set an email to receive alerts from Astronomer support about your Deployments. See Add Or Delete A Deployment Alert Email.

     After you finish entering these values, your config.yaml file should look something like the following:

     deployment:
       environment_variables:
         - is_secret: true
           key: MY_VARIABLE_KEY
           value: MY_VARIABLE_VALUE
         - is_secret: false
           key: MY_VARIABLE_KEY_2
           value: MY_VARIABLE_VALUE_2
       configuration:
         name: My Deployment
         description: The Deployment I'm using for migration.
         runtime_version: 6.3.0
         dag_deploy_enabled: false
         scheduler_au: 5
         scheduler_count: 2
         cluster_name: My Cluster
         workspace_name: My Workspace
       worker_queues:
         - name: default
           max_worker_count: 10
           min_worker_count: 1
           worker_concurrency: 16
           worker_type: E2-STANDARD-4
       alert_emails:
         - myalertemail@cosmicenergy.org

4. Run the following command to push your configuration to Astro and create your Deployment:

   astro deployment create --deployment-file config.yaml