Follow-up to #279 / #374.
Window sizing uses a hardcoded TRACK_BYTES_PER_CELL = 12 * 64 (768 B) per cell per
track (python/genvarloader/_dataset/_streaming.py). It is added to cell_bytes and
therefore drives _window_samples directly, so a bad estimate silently mis-sizes the
read window — too high wastes memory and fragments the sweep, too low risks
over-committing.
It is also what makes the documented cohort threshold drop roughly 49x per track:
at the default max_mem="512MB", max_cells goes from 8,388,608 with no tracks to
335,544 with one and 171,196 with two. That is a big user-visible cliff resting on a
guess.
Replace it with a cheap up-front sample: count_intervals over a small number of
regions gives real interval density for the actual sources, which is what the constant
is standing in for. Keep the hardcoded value as the fallback when sampling is not
possible, and log which one was used.
🤖 Generated with Claude Code
https://claude.ai/code/session_01EBwY8GiRjf1zJBaEZ2CiEa
Follow-up to #279 / #374.
Window sizing uses a hardcoded
TRACK_BYTES_PER_CELL = 12 * 64(768 B) per cell pertrack (
python/genvarloader/_dataset/_streaming.py). It is added tocell_bytesandtherefore drives
_window_samplesdirectly, so a bad estimate silently mis-sizes theread window — too high wastes memory and fragments the sweep, too low risks
over-committing.
It is also what makes the documented cohort threshold drop roughly 49x per track:
at the default
max_mem="512MB",max_cellsgoes from 8,388,608 with no tracks to335,544 with one and 171,196 with two. That is a big user-visible cliff resting on a
guess.
Replace it with a cheap up-front sample:
count_intervalsover a small number ofregions gives real interval density for the actual sources, which is what the constant
is standing in for. Keep the hardcoded value as the fallback when sampling is not
possible, and log which one was used.
🤖 Generated with Claude Code
https://claude.ai/code/session_01EBwY8GiRjf1zJBaEZ2CiEa