Skip to content

feat(extractors)!: take any reference that has extract and chrom_sizes - #14

Open
Hoeze wants to merge 11 commits into
feat/interval-resize-rulefrom
feat/extractor-use-strand
Open

Hoeze wants to merge 11 commits into
feat/interval-resize-rulefrom
feat/extractor-use-strand

Conversation

@Hoeze

@Hoeze Hoeze commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

tl;dr: VariantSeqExtractor reads any reference with extract and chrom_sizes, checks intervals, variants and REFs against it, and takes extract(interval, variants, *, anchor_pos=None, pad_with_n=False). FastaStringExtractor returns uppercase sequences by default.

Strand and arguments:

  • use_strand is a required keyword-only argument of FastaStringExtractor and VariantSeqExtractor. Before, the defaults differed (False and True), so a caller who relied on a default could get reverse-complemented sequences or not without noticing.
  • The per-call use_strand of both extract methods is gone. The extractor has one strand handling.
  • Both extract methods raise a TypeError for unknown arguments. Before, they swallowed them, so a typo like fixed_length=False passed silently.

Reference:

  • kipoiseq2.extractors.base.SequenceExtractor is a protocol with extract(interval) and chrom_sizes. It replaces BaseExtractor, so a reference needs no base class.
  • VariantSeqExtractor(reference, *, use_strand) takes the reference as its first argument and raises a TypeError for other objects. It replaces fasta_file and reference_sequence. For a FASTA file, pass FastaStringExtractor(path, use_strand=False).
  • VariantSeqExtractor opens no file, so it has no close(). The caller owns the reference and closes it. Before, BaseExtractor closed the files in __del__.
  • ref_seq_extractor and the deprecated alias fasta are gone.
  • FastaStringExtractor is a context manager.
  • FastaStringExtractor.chrom_sizes maps each chromosome to its length, from the index. The pyfaidx object fasta is private.
  • FastaStringExtractor uppercases the sequences by default. The keyword-only uppercase=True replaces force_upper=False. uppercase=False keeps the case of the file, e.g. to find soft-masked repeats.
  • Encoder and Tokenizer of kipoiseq2.encoders are case-sensitive, so they take the extracted sequences as they are, also in soft-masked repeats. For a lowercase letter, their error now points to FastaStringExtractor, which uppercases by default.
  • FastaStringExtractor raises a ValueError for a gzip file that is not bgzip, or a bgzip file without a .gz or .bgz suffix. It raises an ImportError for a bgzip file if Biopython is missing. Before, pyfaidx raised a misleading ImportError or a UnicodeDecodeError.

Checks:

  • FastaStringExtractor.extract raises a ValueError for an unknown chromosome and for an interval outside the chromosome. Before, an interval past the chromosome end gave a shorter sequence.
  • VariantSeqExtractor.extract reads the chromosome length from the chrom_sizes of the reference, so chrom_len is gone. With pad_with_n=True, it fills the positions outside the chromosome with N. Otherwise it raises a ValueError.
  • VariantSeqExtractor.extract raises a ValueError for a variant on another chromosome, a variant outside the chromosome, and a REF that differs from the reference. Before, it applied such variants without a sign. The REF check ignores case, so it also accepts the lowercase bases of a reference that was read with uppercase=False.

anchor_pos:

  • One keyword replaces anchor and fixed_len. anchor_pos=None lets the length follow the variants, and an int keeps the width of the interval, anchored at that position. Before, fixed_len=False still needed an anchor without effect.
  • anchor_pos must lie from interval.start to interval.end. Before, an anchor outside the interval was moved to the nearest end.
  • The docstring states the rule for a variant at the anchor. A variant lies on the side of its REF, and a REF that spans the anchor is split there. So an insertion whose REF ends at the anchor lies left of it, while GenomeKit normalizes it and puts it right.
  • is_padding is renamed to pad_with_n.

Internals: the variant sequences are built from (start, end, alt) tuples and strings instead of the classes Subsequence and IntervalSeqBuilder.

Verified:

  • 4000 random cases give the same output as 0.1, byte for byte. They cover SNVs, insertions, deletions and MNVs, all strands, anchors in the interval, both length modes, and padding on and off. 10 cases raise a ValueError in both versions.
  • New tests compare VariantSeqExtractor with GenomeKit outputs for the same reference and variants. 16 of the 18 cases agree. In the other two, the insertion lies at the anchor, so the tests keep kipoiseq2's value and name GenomeKit's.

BREAKING CHANGE: use_strand is a required keyword-only argument of FastaStringExtractor and VariantSeqExtractor. To keep the 0.1 behavior, pass use_strand=False to FastaStringExtractor and use_strand=True to VariantSeqExtractor. The extract methods no longer take use_strand, and raise a TypeError for unknown arguments. BaseExtractor is removed: a reference is any object with extract(interval) and chrom_sizes. VariantSeqExtractor(fasta_file=path, use_strand=u) becomes VariantSeqExtractor(FastaStringExtractor(path, use_strand=False), use_strand=u), and reference_sequence=extractor becomes the first argument. VariantSeqExtractor.close, ref_seq_extractor and fasta are removed. FastaStringExtractor reads uppercase sequences by default. The keyword-only uppercase=True replaces force_upper=False, and uppercase=False keeps the case of the file. FastaStringExtractor.fasta is private: use chrom_sizes. FastaStringExtractor.extract raises a ValueError for intervals outside the chromosome. VariantSeqExtractor.extract takes (interval, variants, *, anchor_pos=None, pad_with_n=False): anchor=a becomes anchor_pos=a, anchor=a, fixed_len=False becomes anchor_pos=None, and is_padding becomes pad_with_n. It no longer takes chrom_len, and raises a ValueError for an anchor_pos outside the interval, for positions outside the chromosome without pad_with_n=True, and for variants on another chromosome, outside the chromosome or with a REF that differs from the reference.

feat(extractors)!: require use_strand and reject unknown arguments
fix(extractors): raise a clear error for unreadable FASTA files
feat(extractors)!: uppercase FASTA sequences by default
feat(extractors)!: check intervals against the chromosome sizes
feat(extractors)!: check the chrom and REF of variants
feat(extractors)!: merge anchor and fixed_len into anchor_pos

@Hoeze
Hoeze added this pull request to stack #16 October 15, 2026 22:32
@Hoeze
Hoeze force-pushed the feat/extractor-use-strand branch from 78fb5e8 to 32305da Compare October 16, 2026 19:34
@Hoeze
Hoeze force-pushed the feat/extractor-use-strand branch 2 times, most recently from 359eb52 to e653064 Compare October 16, 2026 23:51
@Hoeze Hoeze changed the title feat(extractors)!: require use_strand and reject unknown arguments feat(extractors)!: take any reference that has extract and chrom_sizes Oct 16, 2026
@Hoeze
Hoeze force-pushed the feat/extractor-use-strand branch from e653064 to 9e25d96 Compare October 17, 2026 14:08
@Hoeze
Hoeze force-pushed the feat/extractor-use-strand branch from 9e25d96 to fd2a421 Compare October 17, 2026 16:22
Hoeze added 11 commits October 17, 2026 18:36
FastaStringExtractor defaulted to use_strand=False and VariantSeqExtractor
to use_strand=True, so a caller who relied on a default could get
reverse-complemented sequences or not without noticing. Both extract
methods also swallowed unknown keyword arguments, so a typo like
fixed_length=False passed silently.

BREAKING CHANGE: use_strand is a required keyword-only argument of
FastaStringExtractor and VariantSeqExtractor. To keep the 0.1 behavior,
pass use_strand=False to FastaStringExtractor and use_strand=True to
VariantSeqExtractor. force_upper of FastaStringExtractor is keyword-only
too. FastaStringExtractor.extract and VariantSeqExtractor.extract raise a
TypeError for unknown arguments.
VariantSeqExtractor wrapped the REF and ALT of each variant in a
Subsequence, a str with coordinates, and collected the parts of the
result in IntervalSeqBuilder, a list subclass whose restore() swapped
reference ranges for slices of the fetched sequence. No code read the
coordinates of the ALT parts.

extract now splits the variants as (start, end, alt) tuples, collects
(start, end) reference ranges and ALT strings, and joins them after
the fetch. The results stay the same: 4000 random cases with SNVs,
insertions, deletions and MNVs on both strands give equal sequences
before and after.

Subsequence and IntervalSeqBuilder are gone. They were not in __all__.
pyfaidx reads uncompressed FASTA files, and bgzip files whose name ends
in .gz or .bgz if Biopython is installed. For other compressed files it
failed with a misleading error: a plain gzip file ending in .gz raised
an ImportError about Biopython, and one with another name raised a
UnicodeDecodeError while it built the index.

FastaStringExtractor now reads the gzip header before pyfaidx opens the
file. It raises ValueError for a gzip file that is not a bgzip file,
and for a bgzip file whose name does not end in .gz or .bgz. It raises
ImportError for a bgzip file if Biopython is missing. The docstrings
no longer say that the file can be gzipped.
FastaStringExtractor(fasta_file, *, use_strand, uppercase=True) replaces
force_upper=False. It passes sequence_always_upper=uppercase to
pyfaidx.Fasta, so pyfaidx uppercases each sequence when it reads it.

Encoder and Tokenizer are case-sensitive, so they take the extracted
sequences as they are, also in soft-masked repeats.
uppercase=False keeps the case of the file, e.g. to find soft-masked
bases. The error of the encoders for a lowercase letter now points to
FastaStringExtractor, which uppercases by default.

BREAKING CHANGE: FastaStringExtractor no longer takes force_upper, and it
returns uppercase sequences by default. Pass uppercase=False to keep the
lowercase letters of the file.
VariantSeqExtractor took the mutually exclusive pair fasta_file and
reference_sequence, and kept the deprecated alias fasta. Its reference
had to be a BaseExtractor, whose extract took (interval, *args,
**kwargs). So a VariantSeqExtractor counted as a reference too, and
failed only when called without variants. BaseExtractor also closed the
files in __del__.

- SequenceExtractor is a typing.Protocol with extract(interval) -> str.
  Any object with this method is a reference, without subclassing.
- VariantSeqExtractor(reference, *, use_strand) raises TypeError for
  a reference that is not a SequenceExtractor. The caller owns the
  reference and closes it.
- FastaStringExtractor is a context manager.

VariantSeqExtractor passes the reference intervals with the strand ".",
so a FastaStringExtractor with use_strand=True as the reference does
not reverse complement the sequence twice. A test covers this case.

BREAKING CHANGE: BaseExtractor is removed. VariantSeqExtractor takes the
reference extractor as its first argument: VariantSeqExtractor(
fasta_file=path, use_strand=u) becomes VariantSeqExtractor(
FastaStringExtractor(path, use_strand=False), use_strand=u), and
VariantSeqExtractor(reference_sequence=ex, use_strand=u) becomes
VariantSeqExtractor(ex, use_strand=u). VariantSeqExtractor.close,
ref_seq_extractor and the deprecated alias fasta are removed.
FastaStringExtractor.extract returned 5 bases for a 15 bp interval that
ran 10 bases past the chromosome end, and raised a pyfaidx FetchError
for a negative start. VariantSeqExtractor.extract took the chromosome
length as chrom_len: float = math.inf, so without it the reference
silently returned fewer bases at the chromosome end. For an anchor
outside the chromosome, it put the N at the wrong end.

- FastaStringExtractor.chrom_sizes maps each chromosome in the index to
  its length. The pyfaidx object fasta is private.
- FastaStringExtractor.extract raises ValueError for an interval
  outside [0, chromosome length], and for an unknown chromosome. It
  raises TypeError if the interval is not an Interval.
- SequenceExtractor has chrom_sizes too.
- VariantSeqExtractor.extract reads the chromosome length from
  reference.chrom_sizes. It fetches only bases within the chromosome,
  marks the others, and turns them into N with is_padding=True. Without
  is_padding, it raises ValueError if the result needs such a base.

4000 random cases give the same sequences and errors as before with
chrom_len set to the chromosome length.

BREAKING CHANGE: FastaStringExtractor.extract raises ValueError for an
interval outside its chromosome or on an unknown chromosome, and
FastaStringExtractor.fasta is private. Use chrom_sizes for the
chromosome names and lengths. VariantSeqExtractor.extract no longer
takes chrom_len, and a reference needs chrom_sizes.
VariantSeqExtractor.extract applied a variant at its position also if
it lay on another chromosome or if its REF did not match the reference.
Such a variant comes from a wrong VCF or a wrong genome build, so the
sequence would be wrong without a sign.

BREAKING CHANGE: VariantSeqExtractor.extract raises a ValueError for a
variant on another chromosome than the interval, for a variant outside
the chromosome, and for a REF that differs from the reference, ignoring
case. It raises a TypeError if the interval is not an Interval or a
variant is not a Variant.
VariantSeqExtractor.extract needed an anchor also with fixed_len=False,
where the anchor had no effect. One keyword now does both jobs:
anchor_pos=None lets the length follow the variants, and an int keeps
the width of the interval, anchored at that position. The docstring
states the rule for a variant at the anchor, which differs from
GenomeKit for an insertion whose REF ends at the anchor.

An anchor outside the interval was moved to the nearest end without a
sign. The per-call use_strand gave a second way to set the strand
handling of an extractor.

BREAKING CHANGE: VariantSeqExtractor.extract takes
(interval, variants, *, anchor_pos=None, pad_with_n=False).
anchor=a becomes anchor_pos=a, anchor=a with fixed_len=False becomes
anchor_pos=None, and is_padding becomes pad_with_n. It raises a
ValueError unless interval.start <= anchor_pos <= interval.end.
FastaStringExtractor.extract and VariantSeqExtractor.extract no longer
take use_strand.
The expected values are the outputs of GenomeKit 7.7.0 for the same
reference and variants: 11 bp windows anchored at the variant start,
windows around the variant without an anchor, and 60 bp haplotypes
with the anchors "center", "5p" and "start" on both strands.

The two "all" rows with the anchor "center" keep the value of
kipoiseq2 on purpose. The REF of the insertion ends at the anchor, so
kipoiseq2 puts the insertion left of it. GenomeKit normalizes the
insertion and puts it right.
FastaStringExtractor.extract and VariantSeqExtractor.extract repeated
the strand rule of Interval. They now call _is_reverse.
The isort guard kept dataclasses first, because the extractors imported
Interval from kipoiseq2. They import kipoiseq2.dataclasses now, so the
imports of the package follow the isort order.
@Hoeze
Hoeze force-pushed the feat/extractor-use-strand branch from fd2a421 to c7d3b7a Compare October 17, 2026 17:19

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant