Python has great exception-handling with nice traceback messages that can help debug issues with your code. Here’s an example of a typical traceback message:
Traceback (most recent call last): File "/Users/kbrazil/Library/Python/3.7/bin/jc", line 11, in <module> load_entry_point('jc', 'console_scripts', 'jc')() File "/Users/kbrazil/git/jc/jc/cli.py", line 396, in main result = parser.parse(data, raw=raw, quiet=quiet) File "/Users/kbrazil/git/jc/jc/parsers/uname.py", line 108, in parse raw_output['kernel_release'] = parsed_line.pop(0) IndexError: pop from empty list
I usually read these from the bottom-up to zero-in on the issue. Here I can see that my program is trying to pop the last item off a list called parsed_line
, but the list is empty and Python doesn’t know what to do, so it quits with an IndexError
exception.
The traceback conveniently includes the line number and snippet of the offending code. This is usually correct, but the line numbering can be off depending on the type of error or exception. This might be enough information for me to dig into the code and figure out why parsed_line
is empty. But what about a more complex example?
Traceback (most recent call last): File "/Users/kbrazil/Library/Python/3.7/bin/jc", line 11, in <module> load_entry_point('jc', 'console_scripts', 'jc')() File "/Users/kbrazil/git/jc/jc/cli.py", line 396, in main result = parser.parse(data, raw=raw, quiet=quiet) File "/Users/kbrazil/git/jc/jc/parsers/arp.py", line 226, in parse 'hwtype': line[4].lstrip('[').rstrip(']'), IndexError: list index out of range
In this traceback I can see that the program is trying to pull the fifth item from the line
list but Python can’t grab it – probably because the list doesn’t have that many items. This traceback doesn’t show me the state of the variables, so I can’t tell what input the function took or what the line
variable looks like when causing this issue.
What I’d really like is to see more context (the code lines before and after the error) along with the variable state when the error occurred. Many times this is done with a debugger or with print()
statements. But there is another way!
cgitb (deprecated)
Back in 1995 when CGI scripts were all the rage, Python added the cgi
library along with its helper module, cgitb
. This module would print more verbose traceback messages to the browser to help with troubleshooting. Conveniently, its traceback messages would include surrounding code context and variable state! cgitb
is poorly named since it can drop-in replace standard tracebacks on any type of program. The name might be why it never really gained traction. Unfortunately, cgitb
is set to be deprecated in Python 3.10, but let’s see how it works and then check out how to replace it:
IndexError Python 3.7.6: /usr/local/opt/python/bin/python3.7 Mon Jul 6 12:09:08 2020 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /Users/kbrazil/Library/Python/3.7/bin/jc in <module>() 2 # EASY-INSTALL-ENTRY-SCRIPT: 'jc','console_scripts','jc' 3 __requires__ = 'jc' 4 import re 5 import sys 6 from pkg_resources import load_entry_point 7 8 if __name__ == '__main__': 9 sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) 10 sys.exit( 11 load_entry_point('jc', 'console_scripts', 'jc')() 12 ) load_entry_point = <function load_entry_point> /Users/kbrazil/git/jc/jc/cli.py in main() 391 392 if parser_name in parsers: 393 # load parser module just in time so we don't need to load all modules 394 parser = parser_module(arg) 395 try: 396 result = parser.parse(data, raw=raw, quiet=quiet) 397 found = True 398 break 399 400 except Exception: 401 if debug: result undefined parser = <module 'jc.parsers.arp' from '/Users/kbrazil/git/jc/jc/parsers/arp.py'> parser.parse = <function parse> data = "#!/usr/bin/env python3\n\nimport jc.parsers.ls\nimp...print(tabulate.tabulate(parsed, headers='keys'))\n" raw = False quiet = False /Users/kbrazil/git/jc/jc/parsers/arp.py in parse(data="#!/usr/bin/env python3\n\nimport jc.parsers.ls\nimp...print(tabulate.tabulate(parsed, headers='keys'))\n", raw=False, quiet=False) 221 for line in cleandata: 222 line = line.split() 223 output_line = { 224 'name': line[0], 225 'address': line[1].lstrip('(').rstrip(')'), 226 'hwtype': line[4].lstrip('[').rstrip(']'), 227 'hwaddress': line[3], 228 'iface': line[6], 229 } 230 raw_output.append(output_line) 231 line = ['#!/usr/bin/env', 'python3'] ].lstrip undefined IndexError: list index out of range __cause__ = None __class__ = <class 'IndexError'> __context__ = None __delattr__ = <method-wrapper '__delattr__' of IndexError object> __dict__ = {} __dir__ = <built-in method __dir__ of IndexError object> __doc__ = 'Sequence index out of range.' __eq__ = <method-wrapper '__eq__' of IndexError object> __format__ = <built-in method __format__ of IndexError object> __ge__ = <method-wrapper '__ge__' of IndexError object> __getattribute__ = <method-wrapper '__getattribute__' of IndexError object> __gt__ = <method-wrapper '__gt__' of IndexError object> __hash__ = <method-wrapper '__hash__' of IndexError object> __init__ = <method-wrapper '__init__' of IndexError object> __init_subclass__ = <built-in method __init_subclass__ of type object> __le__ = <method-wrapper '__le__' of IndexError object> __lt__ = <method-wrapper '__lt__' of IndexError object> __ne__ = <method-wrapper '__ne__' of IndexError object> __new__ = <built-in method __new__ of type object> __reduce__ = <built-in method __reduce__ of IndexError object> __reduce_ex__ = <built-in method __reduce_ex__ of IndexError object> __repr__ = <method-wrapper '__repr__' of IndexError object> __setattr__ = <method-wrapper '__setattr__' of IndexError object> __setstate__ = <built-in method __setstate__ of IndexError object> __sizeof__ = <built-in method __sizeof__ of IndexError object> __str__ = <method-wrapper '__str__' of IndexError object> __subclasshook__ = <built-in method __subclasshook__ of type object> __suppress_context__ = False __traceback__ = <traceback object> args = ('list index out of range',) with_traceback = <built-in method with_traceback of IndexError object> The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/Users/kbrazil/Library/Python/3.7/bin/jc", line 11, in <module> load_entry_point('jc', 'console_scripts', 'jc')() File "/Users/kbrazil/git/jc/jc/cli.py", line 396, in main result = parser.parse(data, raw=raw, quiet=quiet) File "/Users/kbrazil/git/jc/jc/parsers/arp.py", line 226, in parse 'hwtype': line[4].lstrip('[').rstrip(']'), IndexError: list index out of range
This verbose traceback gives me just what I’m looking for! Though the default is 5, I told cgitb
to print out 11 lines of context. Now I can see the two variables I’m particularly interested in to troubleshoot this issue: data
and line
.
data = "#!/usr/bin/env python3\n\nimport jc.parsers.ls\nimp...print(tabulate.tabulate(parsed, headers='keys'))\n"
(Notice how it snips the value if it’s too long. Pretty cool!)
line = ['#!/usr/bin/env', 'python3']
Now I can easily see that the data that was input into the function does not look like the type of data expected at all. (it is expecting text output from the arp
command and instead it was fed in another Python script file) I can also see that the line
list only has two items.
I included cgitb
in jc
to provide a verbose debug command option (-dd
) to help speed up troubleshooting of parsing issues – typically during development of a new parser or to quickly identify an issue a user is having over email. It seemed perfect for my needs and aside from the weird name it worked well.
Then I noticed that cgitb
was to be deprecated along with the cgi
module with no replacement.
tracebackplus
I decided to vendorize the builtin cgitb
library so it wouldn’t be orphaned in later versions of Python. After looking at the code I found it would be pretty easy to simplify the module by taking out all of the HTML rendering cruft. And why not rename it to something more descriptive while we’re at it? After not too much thought, I settled on tracebackplus
.
Like cgitb
, tracebackplus
doesn’t require any external libraries and can easily replace standard tracebacks with the following code:
import tracebackplus tracebackplus.enable(context=11)
Here is the code for tracebackplus
along with the permissive MIT license. Feel free to use this code in your projects.
Here’s an example of how it is being used in jc
to provide different levels of debugging using the -d
(standard traceback) or -dd
(tracebackplus
) command line arguments:
try: result = parser.parse(data, raw=raw, quiet=quiet) found = True break except Exception: if debug: if verbose_debug: import jc.tracebackplus jc.tracebackplus.enable(context=11) raise else: import jc.utils jc.utils.error_message( f'{parser_name} parser could not parse the input data. Did you use the correct parser?\n' ' For details use the -d or -dd option.') sys.exit(1)
Happy debugging!
Here is another nice verbose traceback option I found: https://gist.github.com/cknd/9f769b09483e42eb00701a22765be0dc
Here are some more fancy traceback options:
https://github.com/cknd/stackprinter
https://github.com/Qix-/better-exceptions
https://github.com/skorokithakis/tbvaccine
https://github.com/willmcgugan/rich
https://github.com/alexmojaki/stack_data