10.8. Consider warnings to Refactor and Migrate Usage¶
It’s natural for APIs to change in order to satisfy new requirements that meet formerly unanticipated needs. When an API is small and has few upstream or downstream dependencies, making such changes is straightforward. One programmer can often update a small API and all of its callers in a single commit.
However, as a codebase grows, the number of callers of an API can be so large or fragmented across source repositories that it’s infeasible or impractical to make API changes in lockstep with updating callers to match. Instead, you need a way to notify and encourage the people that you collaborate with to refactor their code and migrate their API usage to the latest forms.
For example, say that I want to provide a module for calculating how far a car will travel at a given average speed and duration. Here, I define such a function and assume that speed is in miles per hour and duration is in hours:
>>> def print_distance(speed, duration):
>>> distance = speed * duration
>>> print(f'{distance} miles')
>>>
>>> print_distance(5, 2.5)
12.5 miles
Imagine that this works so well that I quickly gather a large number of dependencies on this function. Other programmers that I collaborate with need to calculate and print distances like this all across our shared codebase.
Despite its success, this implementation is error prone because the units for the arguments are implicit. For example, if I wanted to see how far a bullet travels in 3 seconds at 1000 meters per second, I would get the wrong result:
>>> print_distance(1000, 3)
3000 miles
I can address this problem by expanding the API of print_distance to include optional keyword arguments (see Item 23: “Provide Optional Behavior with Keyword Arguments” and Item 25: “Enforce Clarity with Keyword-Only and Positional-Only Arguments”) for the units of speed, duration, and the computed distance to print out:
>>> CONVERSIONS = {
>>> 'mph': 1.60934 / 3600 * 1000, # m/s
>>> 'hours': 3600, # seconds
>>> 'miles': 1.60934 * 1000, # m
>>> 'meters': 1, # m
>>> 'm/s': 1, # m
>>> 'seconds': 1, # s
>>> }
>>>
>>> def convert(value, units):
>>> rate = CONVERSIONS[units]
>>> return rate * value
>>>
>>> def localize(value, units):
>>> rate = CONVERSIONS[units]
>>> return value / rate
>>>
>>> def print_distance(speed, duration, *,
>>> speed_units='mph',
>>> time_units='hours',
>>> distance_units='miles'):
>>> norm_speed = convert(speed, speed_units)
>>> norm_duration = convert(duration, time_units)
>>> norm_distance = norm_speed * norm_duration
>>> distance = localize(norm_distance, distance_units)
>>> print(f'{distance} {distance_units}')
Now, I can modify the speeding bullet call to produce an accurate result with a unit conversion to miles:
>>> print_distance(1000, 3,
>>> speed_units='meters',
>>> time_units='seconds')
1.8641182099494205 miles
It seems like requiring units to be specified for this function is a much better way to go. Making them explicit reduces the likelihood of errors and is easier for new readers of the code to understand. But how can I migrate all callers of the API over to always specifying units? How do I minimize breakage of any code that’s dependent on print_distance while also encouraging callers to adopt the new units arguments as soon as possible?
For this purpose, Python provides the built-in warnings module. Using warnings is a programmatic way to inform other programmers that their code needs to be modified due to a change to an underlying library that they depend on. While exceptions are primarily for automated error handling by machines (see Item 87: “Define a Root Exception to Insulate Callers from APIs”), warnings are all about communication between humans about what to expect in their collaboration with each other.
I can modify print_distance to issue warnings when the optional keyword arguments for specifying units are not supplied. This way, the arguments can continue being optional temporarily (see Item 24: “Use None and Docstrings to Specify Dynamic Default Arguments” for background), while providing an explicit notice to people running dependent programs that they should expect breakage in the future if they fail to take action:
>>> import warnings
>>>
>>> def print_distance(speed, duration, *,
>>> speed_units=None,
>>> time_units=None,
>>> distance_units=None):
>>> if speed_units is None:
>>> warnings.warn(
>>> 'speed_units required', DeprecationWarning)
>>> speed_units = 'mph'
>>>
>>> if time_units is None:
>>> warnings.warn(
>>> 'time_units required', DeprecationWarning)
>>> time_units = 'hours'
>>>
>>> if distance_units is None:
>>> warnings.warn( 'distance_units required', DeprecationWarning)
>>> distance_units = 'miles'
>>>
>>> norm_speed = convert(speed, speed_units)
>>> norm_duration = convert(duration, time_units)
>>> norm_distance = norm_speed * norm_duration
>>> distance = localize(norm_distance, distance_units)
>>> print(f'{distance} {distance_units}')
I can verify that this code issues a warning by calling the function with the same arguments as before and capturing the sys.stderr output from the warnings module:
import contextlib import io
fake_stderr = io.StringIO() with contextlib.redirect_stderr(fake_stderr):
- print_distance(1000, 3,
speed_units='meters', time_units='seconds')
print(fake_stderr.getvalue())
>>>
1.8641182099494205 miles
.../example.py:97: DeprecationWarning: distance_units required
warnings.warn(
Adding warnings to this function required quite a lot of repetitive boilerplate that’s hard to read and maintain. Also, the warning message indicates the line where warning.warn was called, but what I really want to point out is where the call to print_distance was made without soon-to-be-required keyword arguments.
Luckily, the warnings.warn function supports the stacklevel parameter, which makes it possible to report the correct place in the stack as the cause of the warning. stacklevel also makes it easy to write functions that can issue warnings on behalf of other code, reducing boilerplate. Here, I define a helper function that warns if an optional argument wasn’t supplied and then provides a default value for it:
>>> def require(name, value, default):
>>> if value is not None:
>>> return value
>>> warnings.warn(
>>> f'{name} will be required soon, update your code',
>>> DeprecationWarning,
>>> stacklevel=3)
>>> return default
>>>
>>> def print_distance(speed, duration, *,
>>> speed_units=None,
>>> time_units=None,
>>> distance_units=None):
>>> speed_units = require('speed_units', speed_units, 'mph')
>>> time_units = require('time_units', time_units, 'hours')
>>> distance_units = require(
>>> 'distance_units', distance_units, 'miles')
>>>
>>> norm_speed = convert(speed, speed_units)
>>> norm_duration = convert(duration, time_units)
>>> norm_distance = norm_speed * norm_duration
>>> distance = localize(norm_distance, distance_units)
>>> print(f'{distance} {distance_units}')
I can verify that this propagates the proper offending line by inspecting the captured output:
import contextlib import io
fake_stderr = io.StringIO() with contextlib.redirect_stderr(fake_stderr):
- print_distance(1000, 3,
speed_units='meters', time_units='seconds')
print(fake_stderr.getvalue())
>>>
1.8641182099494205 miles
.../example.py:174: DeprecationWarning: distance_units will be
➥ required soon, update your code
print_distance(1000, 3,
The warnings module also lets me configure what should happen when a warning is encountered. One option is to make all warnings become errors, which raises the warning as an exception instead of printing it out to sys.stderr:
>>> warnings.simplefilter('error')
>>> try:
>>> warnings.warn('This usage is deprecated',
>>> DeprecationWarning)
>>> except DeprecationWarning:
>>> pass # Expected
This exception-raising behavior is especially useful for automated tests in order to detect changes in upstream dependencies and fail tests accordingly. Using such test failures is a great way to make it clear to the people you collaborate with that they will need to update their code. You can use the -W error command-line argument to the Python interpreter or the PYTHONWARNINGS environment variable to apply this policy:
$ python -W error example_test.py Traceback (most recent call last):
- File ".../example_test.py", line 6, in <module>
warnings.warn('This might raise an exception!')
UserWarning: This might raise an exception!
Once the people responsible for code that depends on a deprecated API are aware that they’ll need to do a migration, they can tell the warnings module to ignore the error by using the simplefilter and filterwarnings functions (see https://docs.python.org/3/library/ warnings for all the details):
>>> warnings.simplefilter('ignore')
>>> warnings.warn('This will not be printed to stderr')
After a program is deployed into production, it doesn’t make sense for warnings to cause errors because they might crash the program at a critical time. Instead, a better approach is to replicate warnings into the logging built-in module. Here, I accomplish this by calling the logging.captureWarnings function and configuring the corresponding ‘py.warnings’ logger:
import logging
fake_stderr = io.StringIO() handler = logging.StreamHandler(fake_stderr) formatter = logging.Formatter(
'%(asctime)-15s WARNING] %(message)s')
handler.setFormatter(formatter)
logging.captureWarnings(True) logger = logging.getLogger('py.warnings') logger.addHandler(handler) logger.setLevel(logging.DEBUG)
warnings.resetwarnings() warnings.simplefilter('default') warnings.warn('This will go to the logs output')
print(fake_stderr.getvalue())
>>>
2019-06-11 19:48:19,132 WARNING] .../example.py:227:
➥ UserWarning: This will go to the logs output
warnings.warn('This will go to the logs output')
Using logging to capture warnings ensures that any error reporting systems that my program already has in place will also receive notice of important warnings in production. This can be especially useful if my tests don’t cover every edge case that I might see when the program is undergoing real usage.
API library maintainers should also write unit tests to verify that warnings are generated under the correct circumstances with clear and actionable messages (see Item 76: “Verify Related Behaviors in TestCase Subclasses”). Here, I use the warnings.catch_warnings function as a context manager (see Item 66: “Consider contextlib and with Statements for Reusable try/finally Behavior” for background) to wrap a call to the require function that I defined above:
>>> with warnings.catch_warnings(record=True) as found_warnings:
>>> found = require('my_arg', None, 'fake units')
>>> expected = 'fake units'
>>> assert found == expected
Once I’ve collected the warning messages, I can verify that their number, detail messages, and categories match my expectations:
assert len(found_warnings) == 1 single_warning = found_warnings[0] assert str(single_warning.message) == ( ‘my_arg will be required soon, update your code’) assert single_warning.category == DeprecationWarning
10.8.1. Things to Remember¶
✦ The warnings module can be used to notify callers of your API about deprecated usage. Warning messages encourage such callers to fix their code before later changes break their programs.
✦ Raise warnings as errors by using the -W error command-line argument to the Python interpreter. This is especially useful in automated tests to catch potential regressions of dependencies.
✦ In production, you can replicate warnings into the logging module to ensure that your existing error reporting systems will capture warnings at runtime.
✦ It’s useful to write tests for the warnings that your code generates to make sure that they’ll be triggered at the right time in any of your downstream dependencies.