Warm tip: This article is reproduced from serverfault.com, please click

python: How to trace function execution order in large project

发布于 2018-05-28 03:47:05

I want to trace function/class executive order in scrapy framework. There are multiple *.py files across the default project, and I want to know which py file and class has been executed in order. It sound silly to put logger line in every class and function. How to visualize this order?

cprofile is mainly used for measuring total time. I could also visualize the execution order inside one module, which is common question, but visualizing multiple modules are difficult.

In terms of trace package, I did not find appropriate examples to work with large project like scrapy or django. Trace usage tutorial is about a single python file.

I want to trace multiple *.py files in multiple modules in a large project, eg scrapy, instead of just one module.

I am aware of debug tools like pdb, but I find it cumbersome to put break point across the whole project. More importantly, it is not easy to summarize the execution order.

Finally I solved by using Hunter, which is better than build-in trace module. Trace module did not offer include_dir attribute.

For those who are curiosity about how to trace all lines of scrapy.

$PYTHONHUNTER='Q(module_startswith=["scrapy", "your_project"])' scrapy list 


In terms of django, tracing execution codes of rest_framework and save to test.log, for example:

$PYTHONHUNTER='Q(module_startswith=["rest_framework", "your_project"]), action=CallPrinter(stream=open("test.log", "w"))' python manage.py runserver --noreload --nothreading
Questioner
anonymous
Viewed
1
H.Tibat 2018-05-28 12:07:07

trace

The trace module allows you to trace program execution, generate annotated statement coverage listings, print caller/callee relationships and list functions executed during a program run. It can be used in another program or from the command line.

python -m trace --count -C . somefile.py ...

The above will execute somefile.py and generate annotated listings of all Python modules imported during the execution into the current directory.

PDB

The module pdb defines an interactive source code debugger for Python programs. It supports setting (conditional) breakpoints and single stepping at the source line level, inspection of stack frames, source code listing, and evaluation of arbitrary Python code in the context of any stack frame. It also supports post-mortem debugging and can be called under program control.

Most Common Used Command:

w(here)

  • Print a stack trace, with the most recent frame at the bottom. An arrow indicates the current frame, which determines the context of most commands.

d(own)

  • Move the current frame one level down in the stack trace (to a newer frame).

u(p)

  • Move the current frame one level up in the stack trace (to an older frame).

You can also check this question Python debugging tips

Coverage

Coverage.py measures code coverage, typically during test execution. It uses the code analysis tools and tracing hooks provided in the Python standard library to determine which lines are executable, and which have been executed.

Hunter

Hunter is a flexible code tracing toolkit, not for measuring coverage, but for debugging, logging, inspection and other nefarious purposes.

The default action is to just print the code being executed. Example:

import hunter
hunter.trace(module='posixpath')

import os
os.path.join('a', 'b')

Result in terminal: Hunter Result in terminal