View Deployment logs
View task and Airflow component logs to troubleshoot your data pipelines and better understand the health of both your tasks and their execution environment.
Airflow has two different log types:
- Component logs record the performance of your Airflow components.
- Task logs records events emitted by your DAG as each task executes.
Task logs can additionally be exported to third-party observability platforms. See:
Airflow Component Logs
Airflow has four core components: the scheduler, triggerer, worker, and webserver. Each component records its process in component logs. These logs can be used to monitor overall performance, troubleshoot errors, and optimize resources.
-
Scheduler logs describe the performance of the scheduler, which is responsible for scheduling and queueing task runs. These logs can help you understand scheduler performance and indicate if a task failed due to an issue with the scheduler. For more information on configuring the scheduler on Astro, see Scheduler resources.
-
Triggerer logs describe the performance of the triggerer, the Airflow component responsible for running triggers and signaling tasks to resume when their conditions have been met. The triggerer is used exclusively for tasks that are run with deferrable operators.
-
Worker logs are generated by Celery Workers and can help you monitor task execution to optimize performance. This type of log is not available when using the Kubernetes Exeuctor.
-
Webserver logs relate to the health and performance of the Airflow UI. If the Airflow UI at any point does not load, for example, webserver logs might indicate why.
Airflow component log levels
Logs and messages might also be associated with one of the following log levels:
- Error: Emitted when a process fails or does not complete. For example, these logs might indicate a missing DAG file, an issue with your scheduler's connection to the Airflow database, or an irregularity with your scheduler's heartbeat.
- Warn: Emitted when Airflow detects an issue that may or may not be of concern but does not require immediate action. This often includes deprecation notices marked as
DeprecationWarning. For example, Airflow might recommend that you upgrade your Deployment if there was a change to the Airflow database or task execution logic. - Info: Emitted frequently by Airflow to show that a standard scheduler process, such as DAG parsing, has started. These logs are frequent and can contain useful information. If you run dynamically generated DAGs, for example, these logs will show how many DAGs were created per DAG file and how long it took the scheduler to parse each of them.
View Airflow component logs in the Astro UI
You can access scheduler, triggerer, and task logs in the Astro UI to find the past 24 hours of logs for any Deployment on its Logs page.
-
In the Astro UI, select a Workspace and then a Deployment.
-
Click the Logs tab.
The maximum number of lines returned is 10,000, with 50 results displayed per page. If there are no logs available for a given Deployment, the following message appears:
No matching events have been recorded in the past 24 hours.
Typically, this indicates that the Deployment you selected does not currently have any DAGs running.
Filter options
You can use the following options to specify the types of logs or messages that you want to view.
-
String search: Enter a string, keyword, or phrase to find in your logs. You can also search with suffix wildcards by adding a
*to your search query. For example,acti*returns results that includeactionandacting. The string search does not include fuzzy matching, so misspelled strings or incomplete strings without a wildcard,*, return zero results. -
Time range: Filter the logs displayed based on time.
-
Log type: Filter based on whether the log message is from a scheduler, worker, webserver, or trigger.
View Airflow component logs locally
To show logs for your Airflow scheduler, webserver, or triggerer locally, run the following Astro CLI command:
astro dev logs
After you run this command, the most recent logs for these components appear in your terminal window.
By default, running astro dev logs shows logs for all Airflow components. To see logs only for a specific component, add any of the following flags to your command:
--scheduler--webserver--triggerer
To continue monitoring logs, run astro dev logs --follow. The --follow flag ensures that the latest logs continue to appear in your terminal window. For more information about this command, see CLI Command Reference.
Airflow task logs
Airflow task logs can help you troubleshoot a specific task instance that failed or retried. Based on your preference, you can choose to use to access task logs in the Astro UI or the Airflow UI. Both provide filters, search, and download options for task logs and share other information about your DAG performance on the same page.
Task logs for Astro Deployments are retained for 90 days. The task log retention policy is not currently configurable.
You can also access local Airflow task logs in your local Airflow UI or printed to the terminal.
Airflow task log levels
Similar to the Airflow component log levels, task logs might also be associated with one of the following log levels, that you can search or filter with:
- Error
- Warn
- Info
- Debug
- Critical
View task logs on the Astro UI
To access task logs from the Astro UI:
- In the Astro UI, select a Workspace.
- Click DAGs.
- Click the DAG you want to view task logs for.
- Click a task run in the DAG run grid.
- Click the Logs tab to switch from Graph view.
View task logs in the Airflow UI
- Access the Airflow UI.
- To access the Airflow UI for a Deployment, open the Deployment in the Astro UI and click Open Airflow.
- To access the Airflow UI in a local environment, open a browser and go to
http://localhost:8080.
- Click a DAG.
- Click Graph.
- Click a task run.
- Click Instance Details.
- Click Log.