A byte-level BPE tokenizer of 200,000 tokens for English, Swahili, Afrikaans, German, French, Spanish and Chinese, whose vocabulary is grouped by translation with grouptok: a group gathers tokens that translate each other, across languages (e.g. water, Wasser, eau, agua, maji). Its tokenizer.json is exactly that of the baseline bpe-en-sw-af-de-fr-es-zh-200k, so the two split every text identically and differ only in groups.json. No pad, bos or eos token is declared; set them for your model if you need them. Each sentence pair is split by the BPE, and its language's aligner links the English subwords to the other side's. The links of all pairs and all languages are pooled and turned into…
Independent publisher
Henry Hale
henryhale
Models in Library2
Datasets in Library0
Models on Hugging Face9
Followers—
Compare the published evidence
This profile connects the 2 model and dataset records shown below to their upstream cards and sources. The library counts describe SAVRN's coverage, rather than this publisher's entire output.
Reported licenses: apache-2.0. Open a record to review its specific license and conditions.
Compare model specifications, source dates, and hardware estimates on the individual records. Publisher identity verification is separate from a model's quality, deployment readiness, or suitability for your work.
Models
A byte-level BPE tokenizer of 151,665 tokens for English, Swahili, Afrikaans, German, French, Spanish and Chinese, whose vocabulary is grouped by translation with grouptok: a group gathers tokens that translate each other, across languages (e.g. water, Wasser, eau, agua, maji). Its tokenizer.json is exactly that of the baseline bpe-en-sw-af-de-fr-es-zh-152k, so the two split every text identically and differ only in groups.json. No pad, bos or eos token is declared; set them for your model if you need them. Each sentence pair is split by the BPE, and its language's aligner links the English subwords to the other side's. The links of all pairs and all languages are pooled and turned into…