Run the Astronomer CLI in Podman Containers

Overview

By default, the Astronomer CLI uses Docker to execute a few specific commands:

• astro dev [...]: For running an Airflow environment on your local machine
• astro auth login: For authenticating to Astronomer Software
• astro deploy: For pushing code to a Deployment

Alternatively, you can use Podman to execute these same commands.

Prerequisites

To complete this setup, you need:

• Podman 3.1.0+ installed on your local machine.
• The Astronomer CLI.

Linux Setup

1. Run the following command to start the Podman API service:

   podman system service -t 0 &

   warning

   Avoid running this command from a directory containing a Dockerfile.

2. Run the following command to create a new Astronomer project:

   astro dev init

3. Run the following command to specify Podman as the CLI's primary container engine:

   astro config set container.engine podman

4. Run astro dev start to confirm that Podman is running the containers for your local Airflow environment.

Mac Setup

To set up Podman for an Astronomer project:

1. Run the following commands to start Podman:

   $ podman machine init
   $ podman machine start

2. Run the following command to pick up the Identity and connection URI for your podman-machine-default:

   podman system connection ls

   The output should look like the following:

   podman-machine-default*      /Users/user/.ssh/podman-machine-default  ssh://core@localhost:54523/run/user/1000/podman/podman.sock
   podman-machine-default-root  /Users/user/.ssh/podman-machine-default  ssh://root@localhost:54523/run/podman/podman.sock

   Copy the Identity and URI from podman-machine-default* for the next two steps.

3. Run the following commands to set system environment variables for Podman:

   # Store the Identity for your Podman instance
   $ export CONTAINER_SSHKEY=<your-podman-identity>
   # Ensure that all images created via `podman build` are readable by Software Deployments. 
   # Primarily for use in CI/CD pipelines which require use of the Podman CLI.
   $ export BUILDAH_FORMAT=docker

4. Run the following command to set the connection URI from the Astronomer CLI:

   astro config set podman.connection_uri <your-podman-uri>

5. Enable Remote Login on your Mac.

6. In a separate terminal window, complete the following set of commands and configurations to mount your local Airflow project directory to the Podman machine:

   $ podman machine --log-level=debug ssh -- exit 2>&1 | grep Executing
   # copy ssh command from above output for the next command, for example:
   # 49671 core@localhost
   
   $ ssh -i /Users/user/.ssh/podman-machine-default -R 10000:$(hostname):22 -p <ssh-command>
   $ ssh-keygen -t rsa
   $ ssh-copy-id -p 10000 <user>@127.0.0.1
   $ sudo mkdir -p airflow-dir
   $ sudo chown core:core airflow-dir
   $ sudo vi /etc/fuse.conf
   # uncomment the user_allow_other line in /etc/fuse.conf and save the file
   $ sshfs -p 10000 -o allow_other <user>@127.0.0.1:<local_airflow_dir_path> airflow-dir
   
   # check if sshfs is working fine or not
   $ cd airflow-dir
   $ ls # you should be able to see all the files present in your local airflow directory
   $ pwd
   
   #keep the session running

   Copy the output of pwd for step 7.

7. Open a new terminal window. In an empty directory, run the following commands to create a new Astronomer project, set Podman as your primary container engine, and generate a pod-config.yml file for your project:

   $ astro dev init
   $ astro config set container.engine podman
   $ astro dev start

8. In the pod-config.yml file, replace the default configuration with the following values:

   volumes:
     - hostPath:
         path: <pwd-output>/dags
         type: Directory
       name: airflow-dags-dir
     - hostPath:
         path: <pwd-output>/plugins
         type: Directory
       name: airflow-plugins-dir
     - hostPath:
         path: <pwd-output>/include
         type: Directory
       name: airflow-include-dir

You can now run the Astronomer CLI in Podman containers for this Astronomer project.

Switch Between Using Docker and Podman

Once you set up the Astronomer CLI to use Podman on your local machine, the CLI will automatically run Podman containers whenever you run a command that requires them. To revert back to default behavior and run CLI commands in Docker containers, run the following command:

astro config set container.engine docker

If you need to switch back to using Podman again, run the following command:

astro config set container.engine podman