grouptok-en-sw-af-de-fr-es-zh-200k

A byte-level BPE tokenizer of 200,000 tokens for English, Swahili, Afrikaans, German, French, Spanish and Chinese, whose vocabulary is grouped by translation with grouptok: a group gathers tokens that translate each other, across languages (e.g. water, Wasser, eau, agua, maji). Its tokenizer.json is exactly that of the baseline bpe-en-sw-af-de-fr-es-zh-200k, so the two split every text identically and differ only in groups.json.

Details

No pad, bos or eos token is declared; set them for your model if you need them.

How the groups are made

Each sentence pair is split by the BPE, and its language's aligner links the English subwords to the other side's. The links of all pairs and all languages are pooled and turned into groups, so one group can hold the word in several languages, although each aligner only ever saw English with one of them. English is the first side of every pair, so a group's member 0 is its English word.

Usage

Any library that reads tokenizer.json tokenizes text exactly as the BPE baseline:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("henryhale/grouptok-en-sw-af-de-fr-es-zh-200k")
tok("The water is cold").input_ids

The groups, with grouptok (pip install grouptok):

from huggingface_hub import snapshot_download
from grouptok import GroupedTokenizer

tok = GroupedTokenizer.from_pretrained(snapshot_download("henryhale/grouptok-en-sw-af-de-fr-es-zh-200k"))
groups = tok.readable_groups()
next(g for g in groups if " water" in g)

or from groups.json alone, each token's group and member slot:

import json
from huggingface_hub import hf_hub_download

groups = json.load(open(hf_hub_download("henryhale/grouptok-en-sw-af-de-fr-es-zh-200k", "groups.json")))
[(groups["token_group"][i], groups["token_member"][i]) for i in tok("The water is cold").input_ids]

Files

• tokenizer.json, tokenizer_config.json: the tokenizer (and a chat template), identical to the BPE baseline's
• groups.json: the translation groups, each token's group id and member slot

Training data

The BPE baseline's: 1,000,000 English–X sentence pairs per language. Where a language has two corpora, all of the smaller one and the first rows of the larger one.

Limitations

• The groups inherit the aligners' errors, and the aligners differ by language, so group quality differs by language too.
• Groups link single subwords one to one: multi-word expressions and their one-word translations are never grouped.
• The training text is mostly parliamentary, news, UN and web-mined text, and the languages are balanced by number of sentence pairs, not by amount of text.