Four streaming primitives — TopK, HeavyHitters,
Cardinality (HLL), Filter (Galois) — composed
around a shared mergeable-summary property. Dual Python/native backend,
Arrow bridge to DuckDB/Polars/DataFusion.
Six normalized axes covering the dimensions that actually distinguish streaming primitives. Stratum's strength: tail latency, mergeability, SQL interop. Trade-offs: bulk-build time (slower than heapq's tight C loop at small N), Python FFI per-call overhead.
Wall-clock throughput building the Top-K from a fixed array. Higher is
better. The Python backend's win comes from numpy.argpartition
(O(N), SIMD-vectorised). Stratum native pays Python→C FFI overhead
in this Python-driven bulk test; in a pure C++/Rust hot loop the
membrane runs at 3.8 B ops/s.
| Approach | Throughput | State | Notes |
|---|---|---|---|
Python heapq | 12.0 M ev/s | 3,720 B | GIL-bound; GC tail |
| Stratum (python) ★ | 191.3 M ev/s | 800 B | numpy argpartition |
| Stratum (native) ★ | 25.5 M ev/s | 3,200 B | Python→C FFI bulk path |
| Redis ZADD (localhost) | ~0.1 M ev/s | — | TCP round-trip dominated (estimated) |
The story Stratum's native backend is built around. Python's
heapq has a smooth low p50 but a long tail from GC
pauses and dict resizes. Stratum's hot path cannot spike: it's one
atomic-relaxed load (0.26 ns) plus a branchless 8-popcount filter
probe (5.94 ns). No allocation, no branch the predictor doesn't know,
no syscall.
Where each approach sits in the size/speed trade-off. Stratum primitives sit on the frontier because their state size is bounded by K (Top-K, heavy hitters) or by a fixed register array (HyperLogLog), independent of stream size.
The honest weakness, exposed: the real Gaussian-elimination Cuckoo builder is slower than building a Bloom filter. We solve 8 GF(2) linear systems per block — costs more than 7 hash + 7 SetBit ops. We win the query side, not the build side.
Every primitive sits on the same architectural skeleton: shared-nothing
per-shard workers, lock-free SPSC ring buffers, mmap snapshots, and a
deterministic merge() operation. The math comes from
four different 1980s–2010s papers; the fit is the contribution.
Seven versions shipped over one week. Each builds on the previous; nothing was thrown away. Honest failure modes documented in CHANGELOG.md.
bench_micro output
Run code/build/bench_micro on your machine and paste the
output below. The dashboard re-renders with your measured numbers
overlaid against the baseline. Everything runs client-side.
contains: X ns/op, membrane: X ns/op,
build: ... (X Mkeys/s), memory: X bits/key.