Dask, for logs over four million rows

clvkit stays small by default: the four runtime dependencies and nothing else. CustomerBase.from_transactions is the one step whose cost scales with transactions rather than customers, so it’s the only place a second engine is offered:

uv add "clvkit[dask]"

Pass engine="dask" to CustomerBase.from_transactions with a dask.dataframe.DataFrame in place of the pandas one. The summary it returns is an ordinary small pandas-backed CustomerBase all the same.

Where it starts to win

Synthetic logs in the input contract, read from Parquet by both engines and summarised at time_unit="W", collapse="D" on a 10-core M-series MacBook Pro. Three transactions per customer, a CDNOW-shaped log:

rows	pandas	Dask
100,000	0.06 s · 192 MB	0.22 s · 216 MB	pandas 3.7× faster
1,000,000	0.35 s · 474 MB	0.50 s · 556 MB	pandas 1.4× faster
4,000,000	1.99 s · 1,349 MB	1.68 s · 1,698 MB	Dask 1.2× faster
16,000,000	17.40 s · 4,973 MB	9.59 s · 5,589 MB	Dask 1.8× faster

When to reach for it

The crossover is at about four million transactions. Below it, Dask’s graph construction and shuffle cost more than the work they distribute, and pandas wins by up to 3.7×. That’s the default, and it’s the right call for most logs. Above 4M, engine="dask" roughly halves the wall-clock, as long as the log is already on disk in a format Dask reads in parallel, like Parquet or partitioned CSV.

It won’t fix running out of memory on one machine. Dask’s peak memory runs 12 to 25% higher than pandas’ at every size measured. The default scheduler runs its workers as threads in this same process, so the shuffle buffers and the finished summary share one address space, and the raw log never being held whole doesn’t help when the buffers replace it. If pandas runs out of RAM, the fix is dask.distributed with workers in separate processes, which this library doesn’t configure for you.

Absolute times are machine-specific; the crossover is the part worth comparing. Reproduce the table with uv run --extra dask python benchmarks/dask_crossover.py.