9.7. Consider Interactive Debugging with pdb¶
Everyone encounters bugs in code while developing programs. Using the print function can help you track down the sources of many issues (see Item 75: “Use repr Strings for Debugging Output”). Writing tests for specific cases that cause trouble is another great way to isolate problems (see Item 76: “Verify Related Behaviors in TestCase Subclasses”).
But these tools aren’t enough to find every root cause. When you need something more powerful, it’s time to try Python’s built-in interactive debugger. The debugger lets you inspect program state, print local variables, and step through a Python program one statement at a time.
In most other programming languages, you use a debugger by specifying what line of a source file you’d like to stop on, and then execute the program. In contrast, with Python, the easiest way to use the debugger is by modifying your program to directly initiate the debugger just before you think you’ll have an issue worth investigating. This means that there is no difference between starting a Python program in order to run the debugger and starting it normally.
To initiate the debugger, all you have to do is call the breakpoint built-in function. This is equivalent to importing the pdb built-in module and running its set_trace function:
>>> # always_breakpoint.py
>>> import math
>>>
>>> def compute_rmse(observed, ideal):
>>> total_err_2 = 0
>>> count = 0
>>> for got, wanted in zip(observed, ideal):
>>> err_2 = (got - wanted) ** 2
>>> breakpoint() # Start the debugger here
>>> total_err_2 += err_2
>>> count += 1
>>>
>>> mean_err = total_err_2 / count
>>> rmse = math.sqrt(mean_err)
>>> return rmse
>>>
>>> result = compute_rmse(
>>> [1.8, 1.7, 3.2, 6],
>>> [2, 1.5, 3, 5])
>>> print(result)
0.5291502622129182
As soon as the breakpoint function runs, the program pauses its execution before the line of code immediately following the breakpoint call. The terminal that started the program turns into an interactive Python shell:
$ python3 always_breakpoint.py > always_breakpoint.py(12)compute_rmse() -> total_err_2 += err_2 (Pdb)
At the (Pdb) prompt, you can type in the names of local variables to see their values printed out (or use p ). You can see a list of all local variables by calling the locals built-in function. You can import modules, inspect global state, construct new objects, run the help built-in function, and even modify parts of the running program—whatever you need to do to aid in your debugging.
In addition, the debugger has a variety of special commands to control and understand program execution; type help to see the full list.
Three very useful commands make inspecting the running program easier:
where: Print the current execution call stack. This lets you figure out where you are in your program and how you arrived at the breakpoint trigger.
up: Move your scope up the execution call stack to the caller of the current function. This allows you to inspect the local variables in higher levels of the program that led to the breakpoint.
down: Move your scope back down the execution call stack one level.
When you’re done inspecting the current state, you can use these five debugger commands to control the program’s execution in different ways:
step: Run the program until the next line of execution in the program, and then return control back to the debugger prompt. If the next line of execution includes calling a function, the debugger stops within the function that was called.
next: Run the program until the next line of execution in the current function, and then return control back to the debugger prompt. If the next line of execution includes calling a function, the debugger will not stop until the called function has returned.
return: Run the program until the current function returns, and then return control back to the debugger prompt.
continue: Continue running the program until the next breakpoint is hit (either through the breakpoint call or one added by a debugger command).
quit: Exit the debugger and end the program. Run this command if you’ve found the problem, gone too far, or need to make program modifications and try again.
The breakpoint function can be called anywhere in a program. If you know that the problem you’re trying to debug happens only under special circumstances, then you can just write plain old Python code to call breakpoint after a specific condition is met. For example, here I start the debugger only if the squared error for a datapoint is more than 1:
# conditional_breakpoint.py def compute_rmse(observed, ideal):
...
- for got, wanted in zip(observed, ideal):
err_2 = (got - wanted) ** 2 if err_2 >= 1: # Start the debugger if True
breakpoint()
total_err_2 += err_2 count += 1
...
- result = compute_rmse(
[1.8, 1.7, 3.2, 7], [2, 1.5, 3, 5])
print(result)
When I run the program and it enters the debugger, I can confirm that the condition was true by inspecting local variables:
$ python3 conditional_breakpoint.py > conditional_breakpoint.py(14)compute_rmse() -> total_err_2 += err_2 (Pdb) wanted 5 (Pdb) got 7 (Pdb) err_2 4
Another useful way to reach the debugger prompt is by using post-mortem debugging. This enables you to debug a program after it’s already raised an exception and crashed. This is especially helpful when you’re not quite sure where to put the breakpoint function call.
Here, I have a script that will crash due to the 7j complex number being present in one of the function’s arguments:
>>> # postmortem_breakpoint.py
>>> import math
>>>
>>> def compute_rmse(observed, ideal):
>>> ...
>>>
>>> result = compute_rmse(
>>> [1.8, 1.7, 3.2, 7j], # Bad input
>>> [2, 1.5, 3, 5])
>>> print(result)
None
I use the command line python3 -m pdb -c continue to run the program under control of the pdb module. The continue command tells pdb to get the program started immediately. Once it’s running, the program hits a problem and automatically enters the interactive debugger, at which point I can inspect the program state:
$ python3 -m pdb -c continue postmortem_breakpoint.py Traceback (most recent call last):
- File ".../pdb.py", line 1697, in main
pdb._runscript(mainpyfile)
- File ".../pdb.py", line 1566, in _runscript
self.run(statement)
- File ".../bdb.py", line 585, in run
exec(cmd, globals, locals)
File "<string>", line 1, in <module> File "postmortem_breakpoint.py", line 4, in <module>
import math
- File "postmortem_breakpoint.py", line 16, in compute_rmse
rmse = math.sqrt(mean_err)
TypeError: can't convert complex to float Uncaught exception. Entering post mortem debugging Running 'cont' or 'step' will restart the program > postmortem_breakpoint.py(16)compute_rmse() -> rmse = math.sqrt(mean_err) (Pdb) mean_err (-5.97-17.5j)
You can also use post-mortem debugging after hitting an uncaught exception in the interactive Python interpreter by calling the pm function of the pdb module (which is often done in a single line as import pdb; pdb.pm()):
$ python3 >>> import my_module >>> my_module.compute_stddev([5]) Traceback (most recent call last):
File "<stdin>", line 1, in <module> File "my_module.py", line 17, in compute_stddev
variance = compute_variance(data)
- File "my_module.py", line 13, in compute_variance
variance = err_2_sum / (len(data) - 1)
ZeroDivisionError: float division by zero >>> import pdb; pdb.pm() > my_module.py(13)compute_variance() -> variance = err_2_sum / (len(data) - 1) (Pdb) err_2_sum 0.0 (Pdb) len(data) 1
9.7.1. Things to Remember¶
✦ You can initiate the Python interactive debugger at a point of interest directly in your program by calling the breakpoint built-in function.
✦ The Python debugger prompt is a full Python shell that lets you inspect and modify the state of a running program.
✦ pdb shell commands let you precisely control program execution and allow you to alternate between inspecting program state and progressing program execution.
✦ The pdb module can be used for debug exceptions after they happen in independent Python programs (using python -m pdb -c continue ) or the interactive Python interpreter (using import pdb; pdb.pm()).