The #375 Track B work added RecordStreamEngine.window_realign_inputs, which decodes a window
synchronously in the calling thread so Python can size the deletion-extended track query
(t_ends_ext) before pulling that window's first batch.
A mixed VCF/PGEN stream therefore decodes each window twice: once here, once in the producer
thread that fills the ping-pong slots.
How it is made safe today. _mixed_engine() (python/genvarloader/_dataset/_streaming.py)
gives the mixed path its own RecordStreamEngine, built plan-less via build_engine([], 1, 1).
That engine has its own filler, its own reader and its own lock, so the synchronous decode and the
drive's producer never actually contend — they share no mutable state. PgenWindowFiller's
reader_lock remains in place regardless, because it is what would make a shared-filler
arrangement safe: apply_sample_subset mutates a single pgenlib reader and the GIL is released
between that mutation and the read, so two concurrent fills on one filler would silently swap
sample columns. VCF is exempt either way — VcfWindowFiller::fill opens a fresh
VcfRecordSource per call.
The cost. Roughly 2x decode on the mixed path, plus a second engine object per backend.
Folding the variant table + CSR into what the producer already hands the consumer would remove the
second decode and collapse the two engines into one. That consolidation is precisely the case
reader_lock exists to protect, so it should be done with the lock, not by removing it.
Not required for v1 parity, which is why it was deferred.
Relates to #375.
The #375 Track B work added
RecordStreamEngine.window_realign_inputs, which decodes a windowsynchronously in the calling thread so Python can size the deletion-extended track query
(
t_ends_ext) before pulling that window's first batch.A mixed VCF/PGEN stream therefore decodes each window twice: once here, once in the producer
thread that fills the ping-pong slots.
How it is made safe today.
_mixed_engine()(python/genvarloader/_dataset/_streaming.py)gives the mixed path its own
RecordStreamEngine, built plan-less viabuild_engine([], 1, 1).That engine has its own filler, its own reader and its own lock, so the synchronous decode and the
drive's producer never actually contend — they share no mutable state.
PgenWindowFiller'sreader_lockremains in place regardless, because it is what would make a shared-fillerarrangement safe:
apply_sample_subsetmutates a single pgenlib reader and the GIL is releasedbetween that mutation and the read, so two concurrent
fills on one filler would silently swapsample columns. VCF is exempt either way —
VcfWindowFiller::fillopens a freshVcfRecordSourceper call.The cost. Roughly 2x decode on the mixed path, plus a second engine object per backend.
Folding the variant table + CSR into what the producer already hands the consumer would remove the
second decode and collapse the two engines into one. That consolidation is precisely the case
reader_lockexists to protect, so it should be done with the lock, not by removing it.Not required for v1 parity, which is why it was deferred.
Relates to #375.