7.1. Don’t Add or Delete Items from a List While Looping Over It¶
Adding or deleting items from a list while looping (that is, iterating) over it with a for or while loop will most likely cause bugs. Consider this scenario: you want to iterate over a list of strings that describe items of clothing and ensure that there is an even number of socks by inserting a matching sock each time a sock is found in the list. The task seems straightforward: iter- ate over the list’s strings, and when you find ‘sock’ in a string, such as ‘red sock’ , append another ‘red sock’ string to the list. But this code won’t work. It gets caught in an infinite loop, and you’ll have to press CTRL-C to interrupt it:
clothes = ['skirt', 'red sock'] for clothing in clothes: # Iterate over the list.
if 'sock' in clothing: # Find strings with 'sock'.
clothes.append(clothing) # Add the sock's pair.
print('Added a sock:', clothing) # Inform the user.
Added a sock: red sock Added a sock: red sock Added a sock: red sock --snip-- Added a sock: red sock Traceback (most recent call last): File "<stdin>", line 3, in <module> KeyboardInterrupt
You’ll find a visualization of the execution of this code at https://autbor .com/addingloop/.
The problem is that when you append ‘red sock’ to the clothes list, the list now has a new, third item that it must iterate over: [‘skirt’, ‘red sock’, ‘red sock’] . The for loop reaches the second ‘red sock’ on the next iteration, so it appends another ‘red sock’ string. This makes the list [‘skirt’, ‘red sock’, ‘red sock’, ‘red sock’] , giving the list another string for Python to iterate over. This will continue happening, as shown in Figure 8-1, which is why we see the never-ending stream of ‘Added a sock.’ messages. The loop only stops once the computer runs out of memory and crashes the Python program or until you interrupt it by pressing CTRL-C.
Figure 8-1: On each iteration of the for loop, a new ‘red sock’ is appended to the list, which clothing refers to on the next iteration. This cycle repeats forever.
The takeaway is don’t add items to a list while you’re iterating over that list. Instead, use a separate list for the contents of the new, modified list, such as newClothes in this example:
>>> clothes = ['skirt', 'red sock', 'blue sock']
>>> newClothes = []
>>> for clothing in clothes:
>>>
>>> if 'sock' in clothing:
>>>
>>> print('Appending:', clothing)
>>>
>>> newClothes.append(clothing) # We change the newClothes list, notclothes.
>>>
>>> print(newClothes)
Appending: red sock
Appending: blue sock
['red sock', 'blue sock']
>>> clothes.extend(newClothes) # Appends the items in newClothes to clothes.
>>> print(clothes)
['skirt', 'red sock', 'blue sock', 'red sock', 'blue sock']
A visualization of the execution of this code is at https://autbor.com/ addingloopfixed/.
Our for loop iterated over the items in the clothes list but didn’t mod- ify clothes inside the loop. Instead, it changed a separate list, newClothes . Then, after the loop, we modify clothes by extending it with the contents of newClothes . You now have a clothes list with matching socks.
Similarly, you shouldn’t delete items from a list while iterating over it. Consider code in which we want to remove any string that isn’t ‘hello’ from a list. The naive approach is to iterate over the list, deleting the items that don’t match ‘hello’ :
>>> greetings = ['hello', 'hello', 'mello', 'yello', 'hello']
>>> for i, word in enumerate(greetings):
>>>
>>> if word != 'hello': # Remove everything that isn't 'hello'.
>>>
>>> del greetings[i]
>>>
>>> print(greetings)
['hello', 'hello', 'yello', 'hello']
A visualization of the execution of this code is at https://autbor.com/ deletingloop/.
It seems that ‘yello’ is left in the list. The reason is that when the for loop was examining index 2 , it deleted ‘mello’ from the list. But this shifted all the remaining items in the list down one index, moving ‘yello’ from index 3 to index 2 . The next iteration of the loop examines index 3 , which is now the last ‘hello’ , as in Figure 8-2. The ‘yello’ string slipped by unexam- ined! Don’t remove items from a list while you’re iterating over that list.
Figure 8-2: When the loop removes ‘mello’ , the items in the list shift down one index, causing i to skip over ‘yello’ .
Instead, create a new list that copies all the items except the ones you want to delete, and then replace the original list. For a bug-free equivalent of the previous example, enter the following code into the interactive shell.
>>> greetings = ['hello', 'hello', 'mello', 'yello', 'hello']
>>> newGreetings = []
>>> for word in greetings:
>>>
>>> if word == 'hello': # Copy everything that is 'hello'.
>>>
>>> newGreetings.append(word)
>>> greetings = newGreetings # Replace the original list.
>>> print(greetings)
['hello', 'hello', 'hello']
A visualization of the execution of this code is at https://autbor.com/ deletingloopfixed/.
Remember that because this code is just a simple loop that creates a list, you can replace it with a list comprehension. The list comprehension doesn’t run faster or use less memory, but it’s shorter to type without los- ing much readability. Enter the following into the interactive shell, which is equivalent to the code in the previous example:
>>> greetings = ['hello', 'hello', 'mello', 'yello', 'hello']
>>> greetings = [word for word in greetings if word == 'hello']
>>> print(greetings)
['hello', 'hello', 'hello']
Not only is the list comprehension more succinct, it also avoids the got- cha that occurs when changing a list while iterating over it.
RE FE RE NCE S, ME MORY US AGE, A ND SYS.GE T SIZEOF( )
It might seem like creating a new list instead of modifying the original one wastes memory. But remember that, just as variables technically contain refer- ences to values instead of the actual values, lists also contain references to values. The newGreetings.append(word) line shown earlier isn’t making a copy of the string in the word variable, just a copy of the reference to the string, which is much smaller.
You can see this by using the sys.getsizeof() function, which returns the number of bytes that the object passed to it takes up in memory. In this interac- tive shell example, we can see that the short string ‘cat’ takes up 52 bytes, whereas a longer string takes up 85 bytes:
>>> import sys
>>> sys.getsizeof('cat')
52
>>> sys.getsizeof('a much longer string than just "cat"')
85
(In the Python version I use, the overhead for the string object takes up 49 bytes, whereas each actual character in the string takes up 1 byte.) But a list containing either of these strings takes up 72 bytes, no matter how long the string is:
>>> sys.getsizeof(['cat'])
64
>>> sys.getsizeof(['a much longer string than just "cat"'])
64
The reason is that a list technically doesn’t contain the strings, but rather just a reference to the strings, and a reference is the same size no matter the size of the referred data. Code like newGreetings.append(word) isn’t copying the string in word , but the reference to the string. If you want to find out how much memory an object, and all the objects it refers to, take up, Python core devel- oper Raymond Hettinger has written a function for this, which you can access at https://code.activestate.com/recipes/577504-compute-memory-footprint -of-an-object-and-its-cont/.
So you shouldn’t feel like you’re wasting memory by creating a new list rather than modifying the original list while iterating over it. Even if your list- modifying code seemingly works, it can be the source of subtle bugs that take a long time to discover and fix. Wasting a programmer’s time is far more expen- sive than wasting a computer’s memory.
Although you shouldn’t add or remove items from a list (or any iter- able object) while iterating over it, it’s fine to modify the list’s contents. For example, say we have a list of numbers as strings: [‘1’, ‘2’, ‘3’, ‘4’, ‘5’] . We can convert this list of strings into a list of integers [1, 2, 3, 4, 5] while iterating over the list:
>>> numbers = ['1', '2', '3', '4', '5']
>>> for i, number in enumerate(numbers):
>>> numbers[i] = int(number)
>>> numbers
[1, 2, 3, 4, 5]
A visualization of the execution of this code is at https://autbor.com/ covertstringnumbers. Modifying the items in the list is fine; it’s changing the number of items in the list that is bug prone.
Another possible way to add or delete items in a list safely is by iterating backward from the end of the list to the beginning. This way, you can delete items from the list as you iterate over it, or add items to the list as long as you add them to the end of the list. For example, enter the following code, which removes even integers from the someInts list.
someInts = [1, 7, 4, 5] for i in range(len(someInts)):
if someInts[i] % 2 == 0:
del someInts[i]
Traceback (most recent call last): File "<stdin>", line 2, in <module> IndexError: list index out of range
>>> someInts = [1, 7, 4, 5]
>>> for i in range(len(someInts) - 1, -1, -1):
>>>
>>> if someInts[i] % 2 == 0:
>>>
>>> del someInts[i]
>>>
>>> someInts
[1, 7, 5]
This code works because none of the items that the loop will iterate over in the future ever have their index changed. But the repeated shift- ing up of values after the deleted value makes this technique inefficient for long lists. A visualization of the execution of this code is at https://autbor.com/ iteratebackwards1. You can see the difference between iterating forward and backward in Figure 8-3.
Figure 8-3: Removing even numbers from a list while iterating forward (left) and backward (right)
Similarly, you can add items to the end of the list as you iterate back- ward over it. Enter the following into the interactive shell, which appends a copy of any even integers in the someInts list to the end of the list:
>>> someInts = [1, 7, 4, 5]
>>> for i in range(len(someInts) - 1, -1, -1):
>>>
>>> if someInts[i] % 2 == 0:
>>>
>>> someInts.append(someInts[i])
>>>
>>> someInts
[1, 7, 4, 5, 4]
A visualization of the execution of this code is at https://autbor.com/ iteratebackwards2. By iterating backward, we can append items to or remove items from the list. But this can be tricky to do correctly because slight changes to this basic technique could end up introducing bugs. It’s much simpler to create a new list rather than modifying the original list. As Python core developer Raymond Hettinger put it:
Q. What are the best practices for modifying a list while looping over it?
Don’t.