Boolean masks and where

Intermediate

Introduction

Filtering data is one of the most common things you do with it: the exam scores above 60, the days hotter than 30 degrees, the readings that are not missing. With lists you write a loop with an if. With NumPy you write the condition once, as an expression on the whole array, and get a boolean mask that selects, counts and changes the matching elements in one step.

How it works

Comparison operators on an array work element-wise and give an array of booleans:

a = np.array([4, 9, 1, 7])
print(a > 5)          # [False  True False  True]
print(a[a > 5])       # [9 7]
print(np.count_nonzero(a > 5))   # 2

Indexing with a boolean array of the same shape keeps the elements where it is True. The result is always 1-D and always a copy. Because True counts as 1, np.count_nonzero(mask) or mask.sum() counts the matches.

To combine conditions, use the element-wise operators & (and), | (or) and ~ (not), with parentheses around each comparison:

print((a > 2) & (a < 8))   # [ True False False  True]

The parentheses are required: & binds more tightly than >, so a > 2 & a < 8 means something else and fails. Python's own and and or do not work on arrays at all; they raise "The truth value of an array with more than one element is ambiguous".

np.where(condition, x, y) builds a new array taking x where the condition holds and y elsewhere; x and y may be arrays or single values:

print(np.where(a > 5, a, 0))     # [0 9 0 7]

Finally, masked assignment changes the original array in place, only where the mask is true: a[a > 8] = 8 caps every value at 8.

Example

# A model of boolean masks: a comparison gives one True or False per element.
scores = [55, 82, 47, 91, 68]
mask = [s >= 60 for s in scores]  # scores >= 60
print("mask:", mask)
print("passed:", [s for s, m in zip(scores, mask) if m])  # scores[mask]
print("count:", sum(mask))  # np.count_nonzero(mask)

middle = [(s >= 60) and (s < 90) for s in scores]  # (scores >= 60) & (scores < 90)
print("60 to 89:", [s for s, m in zip(scores, middle) if m])
print("where:", [s if m else 0 for s, m in zip(scores, mask)])  # np.where(mask, scores, 0)

capped = [85 if s > 85 else s for s in scores]  # scores[scores > 85] = 85
print("capped:", capped)
Open in compiler

Line by line

  1. The mask holds one True or False per score, like scores >= 60.
  2. Selecting with the mask keeps the passing scores; summing the mask counts them.
  3. Two conditions combine element by element, like & in NumPy.
  4. The where line keeps passing scores and puts 0 elsewhere; the last line caps high scores, like a masked assignment.

Expected result

mask: [False, True, False, True, True]
passed: [82, 91, 68]
count: 3
60 to 89: [82, 68]
where: [0, 82, 0, 91, 68]
capped: [55, 82, 47, 85, 68]

In the real world

Masks clean data (data[~np.isnan(data)] drops missing values), select pixels by colour in images, and pick out events above a threshold in signals. pandas uses exactly the same syntax for filtering rows: df[(df.age > 30) & (df.city == "Dhaka")].

Common mistakes

  • Using and / or instead of & and |, which raises the "ambiguous" error.
  • Leaving out parentheses around each comparison.
  • Expecting a[mask] to keep the shape: it is always 1-D.
  • Changing the result of a[mask] and expecting the original to change; assign with a[mask] = ... instead.

Best practices

  • Name masks that you use more than once: passed = scores >= 60.
  • Use np.where to build a new array and masked assignment to change one in place.
  • Count with np.count_nonzero or mask.sum(), never a loop.

Summary

Comparisons on arrays give boolean masks. Indexing with a mask selects the matching elements as a new 1-D array, &, | and ~ combine conditions (each in parentheses), np.where chooses between two values per element, and a[mask] = value changes the matching elements in place.

Quick check

1. a = np.array([3, 8, 1]). What is a[a > 2]?
2. Which correctly selects values between 2 and 8?
3. What does np.where(a > 0, a, 0) do?
4. What is the shape of a[mask] when a has shape (3, 4)?

Key takeaways

  • a > 5 gives a boolean array; a[a > 5] selects the elements where it is True.
  • Combine conditions with &, | and ~, each condition in parentheses.
  • np.where picks between two values per element; a[mask] = v changes elements in place.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.