Issues

• Support for python 3.13 and 3.14

  OpenNMT/Tokenizer#344 · Filter by author racinmat opened on Mar 26, 2026
  1

• ICU_INCLUDE_DIRS for cli projects

  OpenNMT/Tokenizer#331 · Filter by author emabiz opened on Jul 15, 2024

• How to return token ranges when tokenizing text?

  OpenNMT/Tokenizer#328 · Filter by author anderleich opened on Apr 16, 2024

• tokenize by script boundaries - only

  OpenNMT/Tokenizer#327 · Filter by author mediabuff opened on Mar 8, 2024

• SentencePiece 0.1.97 changed API to take std::vector<string_view> "everywhere", breaking SetVocabulary here

  OpenNMT/Tokenizer#323 · Filter by author Micket opened on Mar 16, 2023
  2

• space/none mode potentiel issue with case_markup

  enhancement
  OpenNMT/Tokenizer#176 · Filter by author Zenglinxiao opened on Oct 9, 2020
  42