Model Card for Model ID
Model Details
Lightning is a small, autoregressive transformer which utilizes FlashAttention and MHA. This model is trained on a variety of books from
a dataset(300 MB). This is the expanded version of Lightning-30m.
Model Description
Lightning utilizes FlashAttention and AdamW for performance and capability. Lightning is designed to provide quick, coherent outputs, improved with
a larger size and weight.
- Developed by: AobanZ
- Model type: Transformer
- Language(s) (NLP): English
- License: MIT
- Finetuned from model [optional]: lightning-30m
Model Sources [optional]
- Repository: https://huggingface.co/Aobangaming/lightning-60m
Uses
Lightning is intended to be used for research, analysis and fine-tuning, stories and other.
It is not intended to be used for professional advice, real writing or any kind of heavy work as
generated outputs may be incorrect.
Direct Use
Lightning can be used directly for text generation, experimentation, and conversational interactions. Users can provide text prompts and generate responses using the model's built-in language modeling capabilities.
Direct use is primarily intended for research and experimentation. Outputs may be incomplete, inaccurate, repetitive, or unrelated to the input, and should be evaluated before being used for other purposes.
Downstream Use
Lightning may be fined-tuned for an AI Story makers, Research, and small continuation models. However, please note that
generated outputs may be corrupted and/or incorrect.
Out-of-Scope Use
Heavy Work may overload the model, which will cause corrupted outputs and/or misinformation if implemented
into a larger-app/ecosystem.
Bias, Risks, and Limitations
Lightning is designed to process english and conversational text ONLY and cannot be fined-tuned for any other uses(eg. Robotics)
Recommendations
We recommend users of Lightning to finetune the model on new text,
and add necessary guardrails and precautions to prevent misuse.
How to Get Started with the Model
Use the code below to get started with the model.
import torch
from transformers import AutoModelForCausalLM
from tokenizers import Tokenizer
model_id = "Aobangaming/lightning-60m"
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True
)
tokenizer = Tokenizer.from_pretrained(model_id)
prompt = "The"
encoded = tokenizer.encode(prompt, add_special_tokens=False)
input_ids = torch.tensor([encoded.ids])
outputs = model.generate(
input_ids,
max_new_tokens=100,
do_sample=True,
temperature=0.8,
top_k=40,
top_p=0.6,
)
print(tokenizer.decode(outputs[0].tolist()))
Training Details
Training Data
Lightning was trained on the full Booksum dataset.
Training Procedure
Lightning was trained on an RTX 3050 GPU, using FlashAttention and MHA.
The model was trained on a large dataset. It was not trained on fine-tuning datasets since memory issues.
Training Results
Training Hyperparameters
| Hyperparameter |
Value |
Comment |
| Precision |
FP32 |
| Optimizer |
AdamW |
Better weight decay |
| Learning rate |
5e-4 |
| Batch size |
32 |
Adapted for larger dataset |
Environmental Impact
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
- Hardware Type: RTX 3050 6GB
- Hours used: 1.5
- Cloud Provider: My Computer
- Compute Region: Asia
- Carbon Emitted: ~0.17 kg CO₂e
Technical Specifications
Model Architecture and Objective
Lightning-60m uses a 8-layer causal Transformer with 384-dimensional hidden states and 6 attention heads. Each attention head has a dimension of 64.
The architecture uses pre-layer normalization, causal scaled dot-product attention, a 4× expansion GELU feed-forward network, sinusoidal positional encoding, and untied input/output embeddings.
| Hyperparameter |
Value |
Comment |
| Layers |
8 |
| D_MODEL |
384 |
Optimized for 64dim/head |
| Attention Heads |
6 |
Improved from lightning-30m |
| Vocabulary |
~65830 |
w/ 210 Sequence length |
Compute Infrastructure
Hardware
RTX 3050 6GB
Software
Windows 11, Intel i5-10400
