PicoPolite240M is a small-scale language model created using extremely limited computational resources (a single GPU with 32 GB memory) to research training methods for LLMs and SLMs. We will publish a book detailing the entire learning methodology. Furthermore, we will release the complete synthetic dataset created for this purpose. Using the data below, I trained the model for one epoch with a token window of 2000 and fp32 precision; the RoPE θ was set to 3200. Subsequently, using the data below, I trained it for one epoch with a token window of 6400 and bf16 precision; the RoPE θ was set to 10000. Afterward, I applied LongRoPE to extrapolate the token window. This model utilizes google…
Organization
Reiwa-AI K.K.
Reiwa-AI
Models in Library1
Datasets in Library1
Models on Hugging Face1
Followers1
Compare the published evidence
This profile connects the 2 model and dataset records shown below to their upstream cards and sources. The library counts describe SAVRN's coverage, rather than this publisher's entire output.
Reported licenses: gemma, other. Open a record to review its specific license and conditions.
Compare model specifications, source dates, and hardware estimates on the individual records. Publisher identity verification is separate from a model's quality, deployment readiness, or suitability for your work.
Models
Datasets
This dataset is a collection of datasets intended for various language models, with response formats standardized using an LLM. The reformatting process preserves the writing style and persona of the original responses; the responses were not generated by the LLM from scratch. The aim is to have the model acquire basic knowledge and common sense during the initial phase of SFT (Supervised Fine-Tuning). This dataset contains LLM-transformed records derived from multiple source datasets with different licenses. - original-license: the license applicable to the original source - source: the URL from which the source data was obtained The original response text was stylistically rewritten by an…