[Hoeze]

tl;dr: the new module kipoiseq2.encoders has the frozen classes Encoder and Tokenizer, which replace one_hot, tokenize and their helpers. An Alphabet holds the letters and the letters that each ambiguity code stands for. The rule ambiguous of Encoder sets the rows of the codes: "zeros", "fractional" or "ones". Tokenizer gives each code its own token. The lookup is case-sensitive, and a letter that is not in the Alphabet raises a ValueError that names it.

Encoder(alphabet=DNA, *, ambiguous="zeros", dtype=np.uint8) one-hot encodes with one_hot(seq). It returns an array of shape (L, len(alphabet.letters)). A letter gives 1 in its column, and a code of k letters gives the row of the rule ambiguous:

• "zeros" gives a row of zeros.
• "fractional" gives 1/k in the columns of the letters of the code, so N gives 0.25 in every DNA column and R gives [0.5, 0, 0.5, 0]. It needs a floating-point dtype.
• "ones" gives 1 in the columns of the letters of the code, in any dtype, so N gives [1, 1, 1, 1] and R gives [1, 0, 1, 0]. It keeps the letters of each code in integer dtypes. Its rows do not sum to 1, so a model trained on one-hot input sees N as a strong signal.

decode(x) converts the array back to a str. It maps each row to the letter or code with this row, and the first code wins if several codes share a row. With "zeros", all codes share the row of zeros, so this row becomes the first code, e.g. N. With "fractional" and "ones", each code has its own row, so decode(one_hot(seq)) equals seq. decode raises a ValueError for any other row.

Tokenizer(alphabet=DNA, *, token_map=None, dtype=np.uint8) converts a sequence to integer tokens with tokenize(seq). By default, letters[i] becomes the token i, and each code becomes a token of its own after the letters, in the order of the codes. For DNA, N is 4, R is 5 and V is 14. So the token of a letter or code is the index of its row in the table of Encoder. token_map sets other tokens, e.g. the vocabulary of a pretrained model, and several letters and codes can share one. Its keys are the letters and the codes of the alphabet, so 15 keys for DNA.

Both constructors check the configuration once. They raise a TypeError for a wrong type, also for dtype=None, which numpy would silently read as float64. They raise a ValueError for a rule other than "zeros", "fractional" or "ones", for "fractional" with a dtype that is not a floating-point type, or if the keys of token_map are not the letters and the codes. Both classes compare by their configuration, hash, have a repr and pickle, e.g. for the worker processes of a data loader. They are safe as default arguments.

The new module kipoiseq2.alphabets has the frozen dataclass Alphabet(letters, ambiguous={code: letters}) and the presets DNA, RNA and AMINO_ACIDS. DNA and RNA have N and the other IUPAC ambiguity codes, because GRCh37 and GRCh38 contain a few of them. AMINO_ACIDS has the 20 standard amino acids as letters and the codes X, B and Z. For a vocabulary without the IUPAC codes, e.g. the one of HyenaDNA, use the strict Alphabet(["A", "C", "G", "T"], ambiguous={"N": ["A", "C", "G", "T"]}). Alphabet raises a TypeError for a str, which the old functions read in two different ways. It raises a ValueError for letters that are empty, differ in length or repeat. It also raises a ValueError for a code that is a letter, differs in length from the letters, or stands for fewer than 2 letters or for letters outside the alphabet. Two alphabets are equal only with their codes in the same order, because the order sets the tokens. The package kipoiseq2 exports Alphabet, the presets, Encoder and Tokenizer.

The lookup is case-sensitive, so a lowercase letter of a soft-masked repeat is unknown unless the alphabet contains it. A letter that is not in the alphabet raises a ValueError that names it. For ASCII lowercase letters whose uppercase form is in the alphabet, the message points to seq.upper() and to FastaStringExtractor. Before, one_hot_dna and OneHot() encoded unknown and lowercase letters as rows of zeros, and the other code paths raised a KeyError. Now DNA and RNA encode the IUPAC ambiguity codes by the rule ambiguous, and all other unknown letters raise.

For letters of one ASCII character, the lookup translates the bytes of the sequence with a table of 256 bytes and takes the rows of the result. decode builds the str from the bytes of these letters instead of joining them. Other letters, e.g. codons, use a dict and a join. decode checks the rows with a loop over the columns, because numpy reduces an axis of a few values slowly. With "fractional" and "ones", it compares only the rows that are not the row of a letter with the rows of the codes. The times per call for random ACGTN are for 1 kb and for 196,608 bases, on Python 3.14 and numpy 2.5. Each case ran in its own process, 3 times. Each time is the median of these 3 processes, and each process reports the median of 21 runs:

• Tokens: 1.14 us and 58 us.
• One-hot in the default uint8: 2.58 us and 197 us. "ones" takes the same time.
• One-hot in float32 with "fractional": 2.66 us and 229 us.
• decode in uint8: 26 us and 2.2 ms with "zeros", and 35 us and 2.6 ms with "ones".
• decode in float32 with "fractional": 36 us and 0.96 ms.

ReorderedOneHot is gone, because np.expand_dims and np.swapaxes give each of its layouts from the Encoder.one_hot array. tokenize, token2one_hot, one_hot2token and one_hot2string are gone, because Tokenizer and Encoder.decode replace them. Letters of different lengths and a len(seq) that is not a multiple of the letter length raise a ValueError instead of an AssertionError.

With -1, numpy indexing took the last row, so np.eye(4)[tokens] silently encoded N as T. Now a table with one row per letter raises an IndexError for the token of a code. An embedding with len(alphabet.letters) + len(alphabet.ambiguous) rows takes the tokens as they are.

README.md shows the encoders and states the case rule. MIGRATING.md lists each removed name with its replacement.

BREAKING CHANGE: one_hot_dna, one_hot, tokenize, token2one_hot, one_hot2token, one_hot2string, OneHot and ReorderedOneHot are removed. Use Encoder(alphabet, ...).one_hot(seq), Encoder.decode and Tokenizer(alphabet, ...).tokenize(seq) instead, and np.expand_dims and np.swapaxes instead of ReorderedOneHot. They take an Alphabet instead of alphabet, neutral_alphabet or alphabet_size. A neutral letter becomes an ambiguity code for all letters. The presets of kipoiseq2.alphabets replace kipoiseq2.utils.DNA, RNA, AMINO_ACIDS and alphabets. AMINO_ACIDS.letters holds 20 amino acids instead of 22, because B and Z are codes now. Encoder returns uint8 and encodes the codes as rows of zeros by default. Encoder(DNA, ambiguous="fractional", dtype=np.float32) gives the old result of one_hot_dna for N. neutral_value is gone: 0 becomes ambiguous="zeros", 0.25 becomes ambiguous="fractional" with a float dtype, and 1 becomes ambiguous="ones". The encoders are case-sensitive, so letters that are not in the alphabet, also lowercase ones, raise a ValueError. Uppercase the sequence first, e.g. with seq.upper(). With DNA and RNA, the IUPAC ambiguity codes encode by the rule ambiguous. With "fractional", they give fractions of their bases, where one_hot_dna gave rows of zeros. A token_map needs a key for each letter and code, so 15 keys for DNA. Tokenizer encodes each code as a token of its own after the letters instead of -1, and it returns uint8 by default. decode takes the array of one sequence and returns a str.

feat!: add kipoiseq2.alphabets with Alphabet and its presets
feat(transforms)!: remove ReorderedOneHot
fix(alphabets): raise ValueError for letters of different lengths

[Hoeze]

remove the name parameter and give the function a better name