Warm tip: This article is reproduced from serverfault.com, please click

Debugging airflow tasks using airflow test vs using DebugExecutor

发布于 2020-11-29 10:46:50

I'm searching for best way to run/debug tasks and dags in my IDE. I see that there are two ways of doing this. I can run airflow test command in debug mode for particular dag and optionally task. Other way is to use DebugExecutor and run particular dag. I see that both ways require that Airflow database is up and running and that all pools are configured (probably queues as well). My questions are:

  1. What is the main difference between these two?
  2. Does airflow test uses DebugExecutor under the hood?
  3. Is there a way to run/debug dags and tasks without running Airflow database and creating dependent pools and queues?
Questioner
partlov
Viewed
0
12.9k 2020-11-29 19:50:53

What is the main difference between these two?

Debug Executor runs a full DAG Run so you can test the trigger rules. airflow test command run only one task.

This is even clearer in Airflow 2.0. We have separate commands:

  • airflow dags test - starts one DAG Run with DebugExecutor.
  • airflow tasks test - starts one tasks.

Does airflow test uses DebugExecutor under the hood?

No. If you use DebugExecutor then you need to run full scheduler. If you use the airflow task command, only code that is being executed by worker is executed.

Is there a way to run/debug dags and tasks without running Airflow database and creating dependent pools and queues?

You can load a DAG with DagBag and then call the execute method of your task.

from airflow.model.dagbag import DagBag
dag_file_path = "/home/test-user/dags/dag-file.py"
dagbag = DagBag(dag_folder=dag_file_path)
dagbag.dags['test-dag-id'].task_dict['task-id'].execute({})