Run the Astro CLI in Podman containers

By default, the Astro CLI uses Docker to execute a few specific commands:

• astro dev [...]: For running an Airflow environment on your local machine
• astro login: For authenticating to Astronomer Software
• astro deploy: For pushing code to a Deployment

Alternatively, you can use Podman to execute these same commands.

Prerequisites

To complete this setup, you need:

• Podman 3.1.0+ installed on your local machine.
• A pre-1.0 version of the Astro CLI, such as version 0.29.1.

Linux setup

1. Run the following command to start the Podman API service:

   podman system service -t 0 &
   danger

   Avoid running this command from a directory containing a Dockerfile.

2. Run the following command to create a new Astro project:

   astro dev init

3. Run the following command to specify Podman as the CLI's primary container engine:

   astro config set container.engine podman

4. Run astro dev start to confirm that Podman is running the containers for your local Airflow environment.

Mac setup

To set up Podman for an Astro project:

1. Run the following commands to start Podman:

   $ podman machine init
   $ podman machine start

2. Run the following command to pick up the Identity and connection URI for your podman-machine-default:

   podman system connection ls

   The output should look like the following:

   podman-machine-default*      /Users/user/.ssh/podman-machine-default  ssh://core@localhost:54523/run/user/1000/podman/podman.sock
   podman-machine-default-root  /Users/user/.ssh/podman-machine-default  ssh://root@localhost:54523/run/podman/podman.sock

   Copy the Identity and URI from podman-machine-default* for the next two steps.

3. Run the following commands to set system environment variables for Podman:

   # Store the Identity for your Podman instance
   $ export CONTAINER_SSHKEY=<your-podman-identity>
   # Ensure that all images created using `podman build` are readable by Software Deployments.
   # Primarily for use in CI/CD pipelines which require use of the Podman CLI.
   $ export BUILDAH_FORMAT=docker

4. Run the following command to set the connection URI from the Astro CLI:

   astro config set podman.connection_uri <your-podman-uri>

5. Enable Remote Login on your Mac.

6. In a separate terminal window, complete the following set of commands and configurations to mount your local Astro project directory to the Podman machine:

   $ podman machine --log-level=debug ssh -- exit 2>&1 | grep Executing
   # copy ssh command from above output for the next command, for example:
   # 49671 core@localhost
   
   $ ssh -i /Users/user/.ssh/podman-machine-default -R 10000:$(hostname):22 -p <ssh-command>
   $ ssh-keygen -t rsa
   $ ssh-copy-id -p 10000 <user>@127.0.0.1
   $ sudo mkdir -p airflow-dir
   $ sudo chown core:core airflow-dir
   $ sudo vi /etc/fuse.conf
   # uncomment the user_allow_other line in /etc/fuse.conf and save the file
   $ sshfs -p 10000 -o allow_other <user>@127.0.0.1:<local_airflow_dir_path> airflow-dir
   
   # check if sshfs is working fine or not
   $ cd airflow-dir
   $ ls # you should be able to see all the files present in your local airflow directory
   $ pwd
   
   #keep the session running

   Copy the output of pwd for step 7.

7. Open a new terminal window. In an empty directory, run the following commands to create a new Astro project, set Podman as your primary container engine, and generate a pod-config.yml file for your project:

   $ astro dev init --use-astronomer-certified
   $ astro config set container.engine podman
   $ astro dev start

8. In the pod-config.yml file, replace the default configuration with the following values:

   volumes:
     - hostPath:
         path: <pwd-output>/dags
         type: Directory
       name: airflow-dags-dir
     - hostPath:
         path: <pwd-output>/plugins
         type: Directory
       name: airflow-plugins-dir
     - hostPath:
         path: <pwd-output>/include
         type: Directory
       name: airflow-include-dir

You can now run the Astro CLI in Podman containers for this Astro project.

Switch between using Docker and Podman

Once you set up the Astro CLI to use Podman on your local machine, the CLI automatically runs Podman containers whenever you run a command that requires them. To revert to the default behavior and run CLI commands in Docker containers, run the following command:

astro config set container.engine docker

If you need to switch back to using Podman again, run the following command:

astro config set container.engine podman