Kiel-3-Omni is an advanced, staged multimodal conversational model developed by KielTech. Built directly upon Kiel-2.1-Omni, Kiel-3-Omni is engineered with a modular training pipeline that natively integrates voice capabilities, robust text instruction-following, and upcoming vision understanding.
This repository hosts the Phase 2 Unified Preview, combining core voice foundations with high-capacity text Supervised Fine-Tuning (SFT).
Current Training Roadmap
| Phase |
Capability |
Status |
| Phase 1 |
Voice Adaptation & Token Expansion (+2048 EnCodec Audio Tokens) |
YesCompleted & Merged |
| Phase 2 |
Text Supervised Fine-Tuning (ultrachat_200k conversational alignment) |
YesCompleted & Merged |
| Phase 3 |
Vision Understanding (Visual encoders & chart/image reasoning) |
In Progress / Next |
| Phase 4 |
End-to-End Multimodal Unification & Alignment |
Planned |
Architecture & Technical Highlights
- Base Foundation: Built upon Kiel-2.1-Omni (
kiel2/Kiel-2.1-Omni), maintaining seamless continuity with KielTech's multimodal architecture.
- Token-Space Expansion: Native embedding expansion of +2048 audio codec tokens, allowing seamless integration of voice-to-text token spaces.
- Staged LoRA & Unification: Adapters trained via QLoRA/PEFT across dedicated modalities and losslessly merged into the primary model weights for zero-overhead inference.
- Developed by: KielTech
Quick Start: Loading the Model
You can load and run inference with the unified weights using the following snippet:
import torch
from transformers import GenerationConfig
from janus.models import MultiModalityCausalLM, VLChatProcessor
# Load model and processor from Hugging Face Hub
repo_id = "kiel2/kiel-3-Omni-preview"
processor = VLChatProcessor.from_pretrained(repo_id, trust_remote_code=True)
tokenizer = processor.tokenizer
tokenizer.pad_token = tokenizer.eos_token
model = MultiModalityCausalLM.from_pretrained(
repo_id,
torch_dtype=torch.float16,
device_map="auto",
trust_remote_code=True
)
# Format a multi-turn conversation
conversation = [
{"role": "<|User|>", "content": "Hello! Introduce yourself."},
{"role": "<|Assistant|>", "content": ""}
]
prompt = processor.apply_sft_template_for_multi_turn_prompts(
conversations=conversation,
sft_format=processor.sft_format,
system_prompt="You are Kiel-3-Omni, a specialized conversational assistant developed by KielTech."
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
# Generate response
outputs = model.generate(
**inputs,
max_new_tokens=128,
do_sample=True,
temperature=0.7,
top_p=0.9
)
response_tokens = outputs[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(response_tokens, skip_special_tokens=True))
Preview Notice
This repository represents an intermediate preview build (v0.2-preview) under active development by KielTech. Performance and capabilities will expand further with the upcoming integration of Phase 3 (Vision Understanding).
Developed by KielTech
