Develop Airflow DAGs locally with PyCharm
This example shows how to set up PyCharm for local development with Airflow and the Astro CLI. Setting up a local development environment allows you to iterate more quickly when developing DAGs by taking advantage of IDE features.
Before you start
Before trying this example, make sure you have:
- PyCharm
- The Astro CLI
- An Astro project running locally on your computer. See Getting started with the Astro CLI
Configure the Python interpreter
To develop Airflow DAGs in PyCharm, you need to configure at least one Python interpreter. In this example you configure an interpreter using Docker, which allows you to write Python code and interact with running DAGs from within PyCharm.
-
In your PyCharm preferences go to Build, Execution, Deployment >> Docker and specify how to connect to the Docker daemon. For more, see Connect to Docker.
-
Go to Project: <your-project-name> >> Python Interpreter. Click the Add Interpreter button on the right, and then click On Docker:
-
In the window that appears, select Pull or use existing for the Image option. In the Image tag field, select the Docker image that your Airflow environment is running, then click Next.
-
Wait for PyCharm to finish pulling the Docker image and then click Next.
-
Ensure the System Interpreter is set to Python 3. By default this path will point to the location of Python 3 on the machine and should not need to be changed. Then, click Create.
-
On the original Python Interpreter setup screen, ensure the Python interpreter is set to the docker image and click Apply, followed by OK.
Write Airflow code with PyCharm
After you configure PyCharm to use the correct interpreter for Airflow, you get multiple advantages like code autocompletion, identifying deprecated or unused imports, and error syntax highlighting.
PyCharm will show you when there are deprecated imports in your project:
It can also alert you when an import is unused:
Like with other Python projects, PyCharm will highlight syntax errors in your Airflow code:
Lastly, here is an example of PyCharm autocompleting code and showing built-in definitions:
Manage Airflow Docker containers with PyCharm
With PyCharm configured to use the Python interpreter from Docker, you can connect to your Docker containers directly from PyCharm using the Services pane. If you do not see this pane, try pressing cmd+8.
From the Services pane, start Docker by clicking the green play button. You’ll see the same containers appear as when you run docker ps after starting your Astro project locally:
View logs for the containers by clicking on /scheduler, /triggerer, /webserver, or /airflow-dev_2cd823-postgres. Note that these strings might differ based on where the parent directory for your project is located:
Run Airflow CLI commands by right clicking on /scheduler and selecting Create Terminal to bash into the container:
