final-step
After you successfully deploy your code to Astro, you need to migrate your workloads from your source Airflow environment to Astro on a DAG-by-DAG basis. Depending on how your workloads are set up, Astronomer recommends letting DAG owners determine the order to migrate and test DAGs.
You can complete the following steps in the few days or weeks following your migration set up. Provide updates to your Astronomer Data Engineer as they continue to assist you through the process and any solve any difficulties that arise.
Continue to validate and move your DAGs until you have fully cut over your source Airflow instance. After you finish migrating from your source Airflow environment, repeat the complete migration process for any other Airflow instances in your source Airflow environment.
Confirm connections and variables
In the Airflow UI for your Deployment, test all connections that you migrated from your source Airflow environment.
Additionally, check Airflow variable values in Admin > Variables.
Test and validate DAGs in Astro
To create a strategy for testing DAGs, determine which DAGs need the most care when running and testing them.
If your DAG workflow is idempotent and can run twice or more without negative effects, you can run and test these DAGs with minimal risk. If your DAG workflow is non-idempotent and can become invalid when you rerun it, you should test the DAG with more caution and downtime.
Cut over DAGs to Astro using Starship
Starship includes features for simultaneously pausing DAGs in your source Airflow environment and starting them on Astro. This allows you to cut over your production workflows without downtime.
- Starship Plugin
- Starship Operator
For each DAG in your Astro Deployment:
Confirm that the DAG ID in your Deployment is the same as the DAG ID in your source Airflow environment.
In the Airflow UI for your source Airflow environment, go to Astronomer > Migration Tool 🚀.
Click DAGs cutover. In the table that appears, click the Pause icon in the Local column for the DAG you're cutting over.
Click the Start icon in the Remote column for the DAG you're cutting over.
After completing this cutover, the Start and Pause icons switch. If there's an issue after cutting over, click the Remote pause button and then the Local start button to move your workflow back to your source Airflow environment.
The Starship operator does not contain cut-over functionality.
To cut over a DAG, pause the DAG in the source Airflow and unpause the DAG in Astro. Keep both Airflow environments open as you test and ensure that the cutover was successful.
Optimize Deployment resource usage
Monitor analytics
As you cut over DAGs, view Deployment metrics to get a sense of how many resources your Deployment is using. Use this information to adjust your worker queues and resource usage accordingly, or to tell when a DAG isn't running as expected.
Modify instance types or use worker queues
If your current worker type doesn't have the right amount of resources for your workflows, see Deployment settings to learn about configuring worker types on your Deployments.
You can additionally configure worker queues to assign each of your tasks to different worker instance types. View your Deployment metrics to help you determine what changes are required.
Enable DAG-only deploys
Deploying to Astro with DAG-only deploys enabled can make deploys faster in cases where you've only modified your dags directory. To enable the DAG-only deploy feature, see Deploy DAGs only.