Views and copies

Intermediate

Introduction

Python list slices copy: b = a[1:3] gives a new list, and changing it leaves a alone. NumPy does the opposite. A slice of an array is a view: a new array object that looks at the same data. That makes slicing instant, even of huge arrays, but it means a change made through the slice shows up in the original. Knowing which operations return views and which return copies prevents some of the most confusing bugs in NumPy code.

How it works

A view stores a reference to the base data plus a description of which elements it sees: where it starts and how far it steps along each axis. Writing to a view writes to that shared data:

a = np.array([10, 20, 30, 40, 50, 60])
b = a[1:5:2]        # a view: [20 40]
b[0] = 99
print(a)            # [10 99 30 40 50 60]
print(b.base is a)  # True

These return views: basic slices (a[1:3], a[:, 0], a[::2]), reshape (when the data allows it), the transpose a.T, and ravel of an array laid out in order.

These return copies: fancy indexing with a list of positions (a[[1, 3]]), boolean masks (a[a > 30]), astype, flatten, and copy() itself:

c = a[[1, 3]]       # a copy
c[0] = -1
print(a)            # [10 99 30 40 50 60]  unchanged
print(np.shares_memory(a, b), np.shares_memory(a, c))   # True False

The rule of thumb: if NumPy can describe the result as "the same data, read with a different start and step", it gives a view; if the result picks arbitrary elements, it must gather them into new memory, so it copies.

Note that assigning through fancy indexing or a mask on the left of =, as in a[[1, 3]] = 0 or a[a > 30] = 0, does change a: that is a direct assignment into a, not a write to a temporary copy.

When a function receives an array and changes a slice of it, the caller's data changes too. If you need to modify a part independently, take it with .copy().

Example

# A model of a NumPy view: it keeps the base data and a list of positions, so writes go to the base.
base = [10, 20, 30, 40, 50, 60]

class View:
    def __init__(self, data, start, stop, step):
        self.data = data
        self.positions = list(range(start, stop, step))

    def __setitem__(self, i, value):
        self.data[self.positions[i]] = value  # writes go to the shared data

    def values(self):
        return [self.data[p] for p in self.positions]

b = View(base, 1, 5, 2)  # like b = a[1:5:2]
print("view:", b.values())
b[0] = 99  # like b[0] = 99
print("base after writing through the view:", base)

c = base[1:5:2]  # a Python list slice copies, like a[[1, 3]] or a.copy() in NumPy
c[0] = -1
print("copy:", c, "base unchanged:", base)
Open in compiler

Line by line

  1. View keeps a reference to the base list and the positions it sees, like a NumPy view's start and step.
  2. Writing b[0] writes into base itself, so the base changes.
  3. A Python list slice is a new list, which behaves like a NumPy copy.
  4. Changing the copy leaves the base as it was.

Expected result

view: [20, 40]
base after writing through the view: [10, 99, 30, 40, 50, 60]
copy: [-1, 40] base unchanged: [10, 99, 30, 40, 50, 60]

In the real world

Views make NumPy fast and memory-light: cropping a large image, taking every other video frame or a column of a big table costs nothing. They also cause classic bugs, such as normalising "a copy" of a column that was really a view and finding the original data changed. pandas warns about the same issue with its SettingWithCopyWarning.

Common mistakes

  • Assuming a slice is a copy, as with lists, and changing the original by accident.
  • Assuming fancy indexing gives a view, and expecting writes to reach the original.
  • Keeping a small view of a huge array, which keeps the whole huge array in memory.
  • Calling copy() everywhere out of fear, which wastes memory.

Best practices

  • Copy explicitly with .copy() when you will modify a part independently.
  • Check with np.shares_memory(a, b) when unsure.
  • Copy a small slice of a large array if you keep only the slice, so the large one can be freed.

Summary

NumPy slices, reshapes and transposes are views that share data with their base, so writing to them changes the original. Fancy indexing, masks, astype, flatten and copy return independent copies. Use .copy() when you need independence, and np.shares_memory to check.

Quick check

1. a = np.arange(5); b = a[1:3]; b[0] = 9. What is a?
2. Which returns a copy?
3. How do you take a slice that you can change without changing a?
4. Does a[a > 0] = 0 change a?

Key takeaways

  • Basic slices and reshapes return views: writing to them writes to the original.
  • Fancy indexing, boolean masks, astype and copy() return copies.
  • Call .copy() when you need to change a part without changing the whole.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.