I'm searching for best way to run/debug tasks and dags in my IDE. I see that there are two ways of doing this. I can run airflow test
command in debug mode for particular dag and optionally task. Other way is to use DebugExecutor
and run particular dag. I see that both ways require that Airflow database is up and running and that all pools are configured (probably queues as well). My questions are:
airflow test
uses DebugExecutor
under the hood?What is the main difference between these two?
Debug Executor runs a full DAG Run so you can test the trigger rules.
airflow test
command run only one task.
This is even clearer in Airflow 2.0. We have separate commands:
airflow dags test
- starts one DAG Run with DebugExecutor
.airflow tasks test
- starts one tasks.Does airflow test uses
DebugExecutor
under the hood?
No. If you use DebugExecutor
then you need to run full scheduler. If you use the airflow task
command, only code that is being executed by worker is executed.
Is there a way to run/debug dags and tasks without running Airflow database and creating dependent pools and queues?
You can load a DAG with DagBag
and then call the execute method of your task.
from airflow.model.dagbag import DagBag
dag_file_path = "/home/test-user/dags/dag-file.py"
dagbag = DagBag(dag_folder=dag_file_path)
dagbag.dags['test-dag-id'].task_dict['task-id'].execute({})
Great, thanks! So, this way of running tasks is something similar we would use for unit tests.