Repository navigation

Fix loading tokenizers from s3 - #71

Merged

meilame-tayebjee merged 2 commits into

mainfrom

Feb 4, 2026

Merged

Fix loading tokenizers from s3#71
meilame-tayebjee merged 2 commits into
mainfrom
dev

Commits on Feb 4, 2026
- fix: fix Tokenizer has no len / no attribute token_to_id at loading
  meilame-tayebjee
  committed
  44b3249
- chore: force all tokenizers to have a load_from_s3 method
  meilame-tayebjee
  committed
  9bccd0c