pandas: Totals from a file read in chunks

A file is too large to load at once, so it is read with chunksize=k and the group totals are combined chunk by chunk. Model the loop.

What to build

A program that aggregates category totals over chunks.

Requirements

  • For each chunk of k rows (the last may be shorter), print chunk I: rows R, total T, counting chunks from 1, with T to two decimals.
  • Then print CATEGORY TOTAL for every category over the whole file, sorted by category name, totals to two decimals.

Input

Line 1: k. Line 2: n. Then n lines CATEGORY,AMOUNT.

Output

One line per chunk, then one line per category.

Constraints

1 ≤ k ≤ 1000; 1 ≤ n ≤ 10⁴.

Example

Input:

2
5
a,1
b,2
a,3
c,4
b,5

Output:

chunk 1: rows 2, total 3.00
chunk 2: rows 2, total 7.00
chunk 3: rows 1, total 5.00
a 4.00
b 7.00
c 4.00

Why it works

Sums can be built up piece by piece: adding each chunk's amounts into running totals gives exactly the totals of the whole file, while only one chunk is in memory at a time.

Examples

Example 1 · Input
2
5
a,1
b,2
a,3
c,4
b,5
Expected output
chunk 1: rows 2, total 3.00
chunk 2: rows 2, total 7.00
chunk 3: rows 1, total 5.00
a 4.00
b 7.00
c 4.00
Example 2 · Input
10
2
x,1.5
x,2.5
Expected output
chunk 1: rows 2, total 4.00
x 4.00

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Process the rows k at a time with a slice rows[i:i + k].

Hint 2

Keep one dictionary of category totals for the whole file, and add each chunk's rows to it.

Solution

Show one correct solution (try it yourself first)
k = int(input())
n = int(input())
rows = [input().split(",") for _ in range(n)]
totals = {}
for c, i in enumerate(range(0, n, k), 1):
    chunk = rows[i:i + k]
    chunk_total = 0
    for cat, amount in chunk:
        totals[cat] = totals.get(cat, 0) + float(amount)
        chunk_total += float(amount)
    print(f"chunk {c}: rows {len(chunk)}, total {chunk_total:.2f}")
for cat in sorted(totals):
    print(f"{cat} {totals[cat]:.2f}")

Lesson: Reading and writing files →