sinlib

sinlib: Pretrained Sinhala NLP and Spellchecking Models

This repository hosts the official pretrained model checkpoints, vocabularies, and statistical datasets for the sinlib library—a comprehensive Sinhala NLP toolkit.

Models & Artifacts Hosted

This repository contains the following files loaded dynamically by sinlib.spellcheck.TypoDetector:

• bigru_detector.pt: A bidirectional GRU sequence labeling model (BiGRUSequenceLabeler) trained to detect spelling errors and character substitutions at the akshara level.
• bigru_corrector.pt: A sequence-to-sequence bidirectional GRU encoder-decoder model with Attention (BiGRUSeq2Seq) that performs generative character/akshara corrections.
• akshara_vocab.json: Vocabulary mappings mapping Sinhala phonological units (aksharas) and basic punctuation to token IDs for the neural models.
• akshara_ngram.json: Stored counts and vocabularies used by the statistical Trigram model (AksharaNGram) for likelihood evaluation.
• news_unigrams.json & news_bigrams.json: Pre-calculated unigram and bigram word-level frequencies extracted from Sinhala news corpora, used for context-aware candidate re-ranking (Stupid Backoff).
• dictionary.npy: The canonical baseline dictionary containing ~61K valid Sinhala words.
• ngram_probs.npy: N-gram probabilities used by the default PreTrainedTokenizer fallback checks.

Additional models, configuration files, and vocabulary resources required by the sinlib package are also included in this repository.

Intended Use

These weights are designed to be loaded directly through the sinlib Python package.

Installation

pip install sinlib

Example Inference (Spellcheck)

from sinlib.spellcheck import TypoDetector

# Automatically downloads and caches the model files from this HF repository
detector = TypoDetector.from_pretrained("Ransaka/sinlib")

# Run spellcheck (with punctuation preservation & unicode normalization)
sentence = "කොළඹ වරායේ සිට බස්නහිර දෙසින් නාවික සැතපුම් 17ක් පමණ දුරන්."
corrected = detector(sentence)

print(corrected)
# Output: "කොළඹ වරායේ සිට බස්නාහිර දෙසින් නාවික සැතපුම් 17ක් පමණ දුරින්."

Typo Detection Fallback

If PyTorch is not installed in the target environment, sinlib falls back automatically to the statistical N-Gram model (akshara_ngram.json) to perform spelling checks.

# Check word suspicion level
is_typo = detector.is_word_suspicious("පසලට") # True

For more details, usage examples, and API references, please refer to the official documentation:

https://sinlib.readthedocs.io/en/latest/