12.2. The cProfile Profiler¶
Although the timeit module is useful for measuring small code snippets, the cProfile module is more effective for analyzing entire functions or pro- grams. Profiling analyzes your program’s speed, memory usage, and other aspects systematically. The cProfile module is Python’s profiler, or software that can measure a program’s runtime as well as build a profile of runtimes for the program’s individual function calls. This information provides sub- stantially more granular measurements of your code.
To use the cProfile profiler, pass a string of the code you want to mea- sure to cProfile.run() . Let’s look at how cProfiler measures and reports the execution of a short function that sums all the numbers from 1 to 1,000,000:
>>> import time, cProfile
>>> def addUpNumbers():
>>> total = 0
>>> for i in range(1, 1000001):
>>> total += i
>>> cProfile.run('addUpNumbers()')
4 function calls in 0.049 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.049 0.049 0.049 0.049 <ipython-input-4-8bd977dfa834>:2(addUpNumbers)
1 0.000 0.000 0.049 0.049 <string>:1(<module>)
1 0.000 0.000 0.049 0.049 {built-in method builtins.exec}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
When you run this program, the output will look something like this:
Each line represents a different function and the amount of time spent in that function. The columns in cProfile.run() ’s output are:
ncalls The number of calls made to the function
tottime The total time spent in the function, excluding time in subfunctions
percall The total time divided by the number of calls
cumtime The cumulative time spent in the function and all subfunctions
percall The cumulative time divided by the number of calls
filename:lineno(function) The file the function is in and at which line number
For example, download the rsaCipher.py and al_sweigart_pubkey.txt files from https://nostarch.com/crackingcodes/. This RSA Cipher program was fea- tured in Cracking Codes with Python (No Starch Press, 2018). Enter the follow- ing into the interactive shell to profile the encryptAndWriteToFile() function as it encrypts a 300,000-character message created by the ‘abc’ * 100000 expression:
import cProfile, rsaCipher cProfile.run("rsaCipher.encryptAndWriteToFile('encrypted_file.txt', 'al_sweigart_pubkey.txt', 'abc'*100000)")
You can see that the code passed to cProfile.run() took 28.9 seconds to complete. Pay attention to the functions with the highest total times; in this case, Python’s built-in pow() function takes up 28.617 seconds. That’s nearly the entire code’s runtime! We can’t change this code (it’s part of Python), but perhaps we could change our code to rely on it less.
This isn’t possible in this case, because rsaCipher.py is already fairly optimized. Even so, profiling this code has provided us insight that pow() is the main bottleneck. So there’s little sense in trying to improve, say, the readKeyFile() function (which takes so little time to run that cProfile reports its runtime as 0).
This idea is captured by Amdahl’s Law, a formula that calculates how much the overall program speeds up given an improvement to one of its components. The formula is speed-up of whole task = 1 / ((1 – p) + (p / s)) where s is the speed-up made to a component and p is the portion of that component of the overall program. So if you double the speed of a compo- nent that makes up 90 percent of the program’s total runtime, you’ll get 1 / ((1 – 0.9) + (0.9 / 2)) = 1.818, or an 82 percent speed-up of the overall program. This is better than, say, tripling the speed of a component that only makes up 25 percent of the total runtime, which would only be a 1 / ((1 – 0.25) + (0.25 / 2)) = 1.143, or 14 percent overall speed-up. You don’t need to memorize the formula. Just remember that doubling the speed of your code’s slow or lengthy parts is more productive than dou- bling the speed of an already quick or short part. This is common sense: a 10 percent price discount on an expensive house is better than a 10 per- cent discount on a cheap pair of shoes.