English · 中文
This is a preview checkpoint, not the v0.1 release.
It exists to prove the training → merge → evaluation → publish pipeline on
real hardware and to produce a comparable set of programmatic metrics.
It is not representative of the quality the writer series is aiming for,
and it has a known degeneration problem documented below.
这是 preview 权重，不是 v0.1 正式发布。 它的目的是把训练、合并、
评估、发布这条链路在真实硬件上跑通并给出一份可对照的程序指标，
不代表 writer 系列的目标质量，且存在下文记录的退化问题。
English
VNPen is MewBaka Studio's visual-novel model series. The writer edition is
for script writing, de-AI rewriting, and generating example scenes from a mood
brief. Output format is one script line per line: speaker：text, with
narration written as 旁白：.
How this preview differs from the planned v0.1
|
Planned writer v0.1 |
This preview |
| Base |
Qwen3.5-9B |
Qwen3.5-2B |
| Training data |
full writer mix (314,342 rows/epoch) |
subset, 117,077 rows |
| Epochs |
2 |
1 |
| DPO |
yes |
no |
Text-only: the vision tower is not in these weights
The base Qwen/Qwen3.5-2B is multimodal. Its checkpoint carries 297
model.visual.* tensors (a depth-24 / hidden-1024 / patch-16 ViT) and 15
mtp.* tensors for multi-token prediction.
This project is text-only. In transformers, AutoModelForCausalLM on a
qwen3_5 config builds Qwen3_5ForCausalLM over a Qwen3_5TextConfig — so
the vision weights were never loaded at any point: not for training, not
for merging, not for saving. Verified on the published weights:
config.json: model_type: qwen3_5_text,
architectures: ["Qwen3_5ForCausalLM"], no vision_config- 320 tensors saved, 0 matching
visual / vision / merger
- no
preprocessor_config.json, no video_preprocessor_config.json
This is not the result of stripping a multimodal checkpoint after the fact;
the vision half was simply never read.
Training data subset
Drawn from the full writer epoch-1 mix (314,342 rows) by per-task stratified
sampling, seed 0. rewrite and identity are taken whole; the other three are
sampled to a quota.
| Task / 任务 |
Rows / 行数 |
Tokens |
Share / 占比 |
continuation |
40,000 |
57.5 M |
34.2% |
rewrite |
46,777 |
59.5 M |
40.0% |
scene_brief |
20,000 |
15.2 M |
17.1% |
replay |
10,000 |
1.7 M |
8.5% |
identity |
300 |
0.0 M |
0.3% |
| total / 合计 |
117,077 |
133.9 M |
100% |
Rendered with the project's own training template
(qwen35_train_nothink.jinja): token p50 = 1,174, p95 = 1,889. 12 rows
exceeded seq_len 4096 and were discarded, not truncated.
The visual-novel corpus itself is not redistributed.
Training
|
|
| Method |
LoRA, merged into the base for release |
| Rank / alpha / dropout |
64 / 128 / 0.05 |
| Target modules |
q,k,v,o_proj, gate,up,down_proj, and in_proj_qkv, in_proj_z, out_proj |
| Trainable |
62,521,344 / 1,944,346,432 (3.22%) |
| LR / schedule |
2e-4, cosine, 3% warmup |
| Sequence length |
4096, one sample per sequence (no cross-sample packing) |
| Batch |
micro 4 × grad-accum 8 = 32 |
| Precision |
bf16, FlashAttention-2, gradient checkpointing |
| Loss |
assistant span only |
| Epochs / steps |
1 / 3,650 |
| Hardware |
single RTX 5090 32 GB |
| Wall clock |
7h 11m |
| Throughput |
5168 tok/s (real, padding excluded) |
On the target modules. Qwen3.5 is a hybrid stack: of its 24 layers only
6 are standard attention (q/k/v/o_proj); the other 18 are gated-delta-rule
linear attention whose projections are named linear_attn.in_proj_* /
out_proj. A target list of only q,k,v,o,gate,up,down — the conventional
"all linear layers" list — therefore leaves the token mixing of 18 of 24
layers unadapted. The three linear-attention projections are included here.
in_proj_a and in_proj_b are deliberately excluded: they are [16, 2048]
per-head gates, so rank 64 is capped at rank 16 regardless, and they set the
decay rate of the linear-attention state.
Training loss, one point per 10% of the run:
| % |
step |
loss |
| 0% |
10 |
2.1451 |
| 10% |
360 |
1.9920 |
| 20% |
730 |
1.9068 |
| 30% |
1090 |
1.9399 |
| 40% |
1460 |
1.8036 |
| 50% |
1820 |
1.8710 |
| 60% |
2190 |
1.7867 |
| 70% |
2550 |
1.7063 |
| 80% |
2920 |
1.7405 |
| 90% |
3280 |
1.6784 |
| 100% |
3650 |
1.7489 |
Final 1.7489, minimum 1.5461. Single-point loss swings with batch composition
(a batch of continuation rows averages ~1,400 tokens, a batch of identity
rows ~41), so the per-step figure is noisy by construction.
Programmatic evaluation
131 items from the project's v0.1 eval set, scored by metrics.py.
The model judge was not run — these are the programmatic metrics only,
and they measure format and surface statistics, not writing quality.
Both sides use the same template, the same decoding parameters
(temperature=0.8, top_p=0.95, seed 0, empty thinking block) and the same
stop token. The only difference is the weights.
| Metric / 指标 |
Base Qwen/Qwen3.5-2B |
This preview |
Δ |
| Line-format rate / 行格式正确率 |
91.7% |
100.0% |
+8.27 pp |
| Exact line count + zero stray lines / 行数精确且无杂行 |
1.8% |
5.3% |
+3.51 pp |
| Adjective density / 形容词密度 |
0.0304 |
0.0143 |
-0.02 |
| Mean sentence length / 平均句长 |
12.75 |
10.36 |
-2.39 |
| Sentence-length SD / 句长标准差 |
8.88 |
5.63 |
-3.26 |
| Dialogue lines <= 6 chars / ≤6 字对白占比 |
37.8% |
52.0% |
+14.19 pp |
| Length-distribution JS vs corpus / 句长分布 JS 距离 |
0.4719 |
0.5374 |
+0.07 |
| Narration share / 旁白占比 |
22.0% |
38.3% |
+16.25 pp |
| 8-gram overlap, continuation / 8-gram 重叠（续写） |
0.0124 |
0.0052 |
-0.01 |
| 8-gram overlap, rewrite / 8-gram 重叠（改写） |
0.3818 |
0.1207 |
-0.26 |
| Identity keywords hit / 身份题关键词命中 |
0.0% |
60.0% |
+60.00 pp |
| Proofread exact match / 校对精确匹配 |
0.0% |
0.0% |
0.00 pp |
Reading the numbers:
- Line format is essentially solved (91.7% → 100.0%) and line-count
discipline improved a lot: the median line-count overshoot went from +61
to +13. The strict
format_ok metric stays low because it demands an
exact line count with zero stray lines.
- Less parroting of the input on rewrite (0.38 → 0.12) and lower adjective
density (0.030 → 0.014), which is the direction the de-AI rewrite task asks
for.
- Identity was learned: 0% → 60% of identity items contain both
VNPen
and MewBaka (per keyword: 80% / 60%).
- Two regressions.
≤6-char dialogue share overshoots the corpus baseline
of 29.0% (37.8% → 52.0%), and the sentence-length distribution moved
further from the corpus (JS 0.472 → 0.537). The model over-learned short
lines.
- Proofread is 0% on both sides and that is expected: proofreading belongs
to the
realtime profile's mix, and the writer mix contains none of it.
Neither model emits the required JSON-Lines answer format.
Known limitation: repetition / degeneration
On long generations both this preview and the base model collapse into
repeating a short cycle of lines. Measured over the 57 script-writing items:
|
Base Qwen3.5-2B |
This preview |
| Repeated-line share, median / 重复行占比（中位） |
0.040 |
0.256 |
| Repeated-line share, mean / 重复行占比（均值） |
0.266 |
0.397 |
| Severely degenerate items / 严重退化条目 |
22 / 57 |
20 / 57 |
Hit max_new_tokens without stopping / 未自行停止 |
64 / 131 (48.9%) |
31 / 131 (23.7%) |
The preview has slightly fewer catastrophic cases but a markedly higher
median repeated-line share. Worst observed case: a single line emitted 310
times. The base model does this too (155 times), so a large part of it is
decoding defaults rather than fine-tuning — no repetition penalty was set on
either side, because the evaluation protocol fixed the decoding parameters
in advance and they were reported as specified rather than tuned for a better
number.
If you use this model, set a repetition penalty. Something in the
1.05–1.15 range, and/or no_repeat_ngram_size, removes most of it.
Stop token
config.json inherits eos_token_id: 248044 (<|endoftext|>) from the base,
but the chat template ends an assistant turn with <|im_end|> (248046). Left
alone, generate() waits for a token that a chat prompt never produces and
runs to max_new_tokens every time. generation_config.json in this repo is
therefore set to eos_token_id: [248046, 248044]. If you build a prompt
yourself, pass the stop token explicitly.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
m = "VNPen/vnpen-writer-2b-v0.1-preview"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(m, dtype=torch.bfloat16).cuda()
msgs = [
{"role": "system", "content": "你是 VNPen，由 MewBaka 工作室发布的视觉小说专项模型，擅长视觉小说剧本的写作、改写与示例生成。输出剧本时使用「说话人：文本」的行格式，旁白写作「旁白：」，示例场景一般 30–50 行。使用简体中文。用户提出与写作无关的问题时正常回答。"},
{"role": "user", "content": "写一段黄昏天台、两个人都没把话说开的场景，30 行左右。"},
]
ids = tok.apply_chat_template(msgs, return_tensors="pt",
add_generation_prompt=True).cuda()
out = model.generate(
ids, max_new_tokens=1200, do_sample=True,
temperature=0.8, top_p=0.95,
repetition_penalty=1.1, # see the limitation above
eos_token_id=[248046, 248044],
pad_token_id=tok.pad_token_id,
)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
The chat template is stored in tokenizer_config.json and reproduces the
training-time rendering byte for byte — verified on 50 real training rows,
50/50 identical, generation prompt identical.
Repository contents
| Path |
What |
model.safetensors, config.json |
merged bf16 weights, text-only |
tokenizer.json, tokenizer_config.json |
tokenizer and chat template |
generation_config.json |
with the corrected stop tokens |
lora/ |
the unmerged LoRA adapter |
LICENSE, NOTICE |
Apache-2.0 and attribution |
License
Apache License 2.0, the same as the base model.
Built from Qwen3.5 (Qwen/Qwen3.5-2B), Copyright the Qwen team,
Alibaba Group, licensed under the Apache License, Version 2.0.
Modified by supervised fine-tuning (LoRA, merged) and by keeping
only the text decoder: the vision tower was never loaded.
The upstream Qwen/Qwen3.5-2B repository ships no NOTICE file
(Apache-2.0 §4(d) only requires propagating one when the original has one), so
the NOTICE here is written by this project. Per §4(b): these weights are
modified relative to the base — supervised fine-tuning, and only the text
decoder is kept.
The replay portion of the training data comes from public instruction sets:
shibing624/alpaca-zh (CC BY 4.0), llm-jp/oasst1-21k-ja (Apache-2.0),
yahma/alpaca-cleaned (CC BY 4.0), OpenAssistant/oasst1 (Apache-2.0).
The visual-novel corpus is not redistributed.
Apache-2.0 grants no rights to the Qwen / Alibaba / Tongyi names or logos;
they appear here only as descriptive attribution.
中文
VNPen 是 MewBaka 工作室的视觉小说专项模型系列。writer 版本负责剧本写作、
去 AI 味改写、按氛围生成示例场景。输出格式为一行一句剧本行：
说话人：文本，旁白写作 旁白：。
与计划中的 v0.1 的差别
|
计划中的 writer v0.1 |
本 preview |
| 基座 |
Qwen3.5-9B |
Qwen3.5-2B |
| 训练数据 |
完整 writer 混合（314,342 行/epoch） |
子集 117,077 行 |
| epoch |
2 |
1 |
| DPO |
有 |
无 |
纯文本：视觉编码器不在这份权重里
基座 Qwen/Qwen3.5-2B 是多模态的，checkpoint 里带 297 个 model.visual.*
张量（一个 depth 24 / hidden 1024 / patch 16 的 ViT）和 15 个 mtp.* 张量。
本项目只用文本。transformers 里 AutoModelForCausalLM 作用于 qwen3_5
配置时构建的是 Qwen3_5ForCausalLM，它建立在 Qwen3_5TextConfig 之上 ——
视觉权重在训练、合并、保存的任何一步都没有被加载过。在发布的权重上已核验：
config.json：model_type: qwen3_5_text、
architectures: ["Qwen3_5ForCausalLM"]、不含 vision_config- 保存 320 个张量，匹配
visual / vision / merger 的 0 个
- 不含
preprocessor_config.json 与 video_preprocessor_config.json
这不是"事后从多模态权重里剥离"的结果，而是那一半从来没被读进来过。
训练数据子集
从完整的 writer epoch-1 混合（314,342 行）按任务分层随机抽取，种子 0。
rewrite 与 identity 取全部，其余三项按配额抽样。
| Task / 任务 |
Rows / 行数 |
Tokens |
Share / 占比 |
continuation |
40,000 |
57.5 M |
34.2% |
rewrite |
46,777 |
59.5 M |
40.0% |
scene_brief |
20,000 |
15.2 M |
17.1% |
replay |
10,000 |
1.7 M |
8.5% |
identity |
300 |
0.0 M |
0.3% |
| total / 合计 |
117,077 |
133.9 M |
100% |
用项目自己的训练模板 qwen35_train_nothink.jinja 渲染：token p50 = 1,174、
p95 = 1,889。12 行超过 seq_len 4096，按规则丢弃而非截断。
视觉小说语料本身不随模型发布。
训练
|
|
| 方法 |
LoRA，发布前合并进基座 |
| r / alpha / dropout |
64 / 128 / 0.05 |
| 目标层 |
q,k,v,o_proj、gate,up,down_proj，以及 in_proj_qkv、in_proj_z、out_proj |
| 可训练参数 |
62,521,344 / 1,944,346,432（3.22%） |
| 学习率 / 调度 |
2e-4，cosine，warmup 3% |
| 序列长度 |
4096，一行一序列（不跨样本拼接） |
| batch |
micro 4 × 梯度累积 8 = 32 |
| 精度 |
bf16、FlashAttention-2、gradient checkpointing |
| loss |
只计 assistant 段 |
| epoch / 步数 |
1 / 3,650 |
| 硬件 |
单卡 RTX 5090 32 GB |
| 耗时 |
7h 11m |
| 吞吐 |
5168 tok/s（真实 token，不含填充） |
关于目标层。 Qwen3.5 是混合架构：24 层里只有 6 层是标准注意力
（q/k/v/o_proj），另外 18 层是 gated-delta-rule 线性注意力，投影名为
linear_attn.in_proj_* / out_proj。只写 q,k,v,o,gate,up,down
这份常规的"全线性层"清单，会让 18/24 层的 token 混合完全没有 adapter。
本次补上了那三个线性注意力投影。in_proj_a 与 in_proj_b 刻意排除：
它们是 [16, 2048] 的每头门控，r=64 的有效秩被 16 卡死，
且它们决定线性注意力状态的衰减率。
训练 loss，每 10% 一个点：
| % |
step |
loss |
| 0% |
10 |
2.1451 |
| 10% |
360 |
1.9920 |
| 20% |
730 |
1.9068 |
| 30% |
1090 |
1.9399 |
| 40% |
1460 |
1.8036 |
| 50% |
1820 |
1.8710 |
| 60% |
2190 |
1.7867 |
| 70% |
2550 |
1.7063 |
| 80% |
2920 |
1.7405 |
| 90% |
3280 |
1.6784 |
| 100% |
3650 |
1.7489 |
最终 1.7489，最低 1.5461。单点 loss 随批次组成波动很大
（一批 continuation 平均约 1,400 token，一批 identity 约 41），
所以逐步数值天然有噪声。
程序指标评估
项目 v0.1 评估集的 131 道题，用 metrics.py 打分。没有跑模型 judge ——
以下只是程序指标，衡量的是格式与表层统计，不衡量文笔。
两侧使用同一模板、同一组解码参数（temperature=0.8、top_p=0.95、
种子 0、空思考块）和同一个停止符，唯一差别是权重。
| Metric / 指标 |
Base Qwen/Qwen3.5-2B |
This preview |
Δ |
| Line-format rate / 行格式正确率 |
91.7% |
100.0% |
+8.27 pp |
| Exact line count + zero stray lines / 行数精确且无杂行 |
1.8% |
5.3% |
+3.51 pp |
| Adjective density / 形容词密度 |
0.0304 |
0.0143 |
-0.02 |
| Mean sentence length / 平均句长 |
12.75 |
10.36 |
-2.39 |
| Sentence-length SD / 句长标准差 |
8.88 |
5.63 |
-3.26 |
| Dialogue lines <= 6 chars / ≤6 字对白占比 |
37.8% |
52.0% |
+14.19 pp |
| Length-distribution JS vs corpus / 句长分布 JS 距离 |
0.4719 |
0.5374 |
+0.07 |
| Narration share / 旁白占比 |
22.0% |
38.3% |
+16.25 pp |
| 8-gram overlap, continuation / 8-gram 重叠（续写） |
0.0124 |
0.0052 |
-0.01 |
| 8-gram overlap, rewrite / 8-gram 重叠（改写） |
0.3818 |
0.1207 |
-0.26 |
| Identity keywords hit / 身份题关键词命中 |
0.0% |
60.0% |
+60.00 pp |
| Proofread exact match / 校对精确匹配 |
0.0% |
0.0% |
0.00 pp |
怎么读这些数字：
- 行格式基本解决（91.7% → 100.0%），行数纪律大幅改善：行数超出的中位数
从 +61 降到 +13。严格的
format_ok 仍然很低，是因为它要求行数精确相等
且一个杂行都没有。
- 改写时更少照抄输入（0.38 → 0.12），形容词密度减半（0.030 → 0.014），
正是去 AI 味改写想要的方向。
- 身份学会了：含
VNPen 与 MewBaka 两个词的身份题从 0% 升到 60%
（分词命中 80% / 60%）。
- 两项退步。
≤6 字对白占比 冲过了语料基线 29.0%（37.8% → 52.0%），
句长分布反而离语料更远（JS 0.472 → 0.537）。模型把短句学过头了。
- 校对两侧都是 0%，这是预期内的：校对属于
realtime profile 的混合，
writer 混合里一条都没有，两个模型都不会输出要求的 JSON Lines 格式。
已知缺陷：重复 / 退化
长生成时，本 preview 与底模都会塌缩成几行一组的循环。
在 57 道剧本写作题上测得：
|
Base Qwen3.5-2B |
This preview |
| Repeated-line share, median / 重复行占比（中位） |
0.040 |
0.256 |
| Repeated-line share, mean / 重复行占比（均值） |
0.266 |
0.397 |
| Severely degenerate items / 严重退化条目 |
22 / 57 |
20 / 57 |
Hit max_new_tokens without stopping / 未自行停止 |
64 / 131 (48.9%) |
31 / 131 (23.7%) |
preview 的灾难性个案略少，但中位重复率明显更高。观测到的最坏情况是
同一行出现 310 次。底模同样如此（155 次），说明其中很大一部分来自解码默认值
而非微调 —— 两侧都没有设 repetition penalty，因为评估协议事先固定了
解码参数，这里如实按协议报告，而不是调参调到好看为止。
如果你要用这个模型，请设 repetition penalty。 1.05–1.15 区间，
或配合 no_repeat_ngram_size，可以消除其中大部分。
停止符
config.json 从基座继承了 eos_token_id: 248044（<|endoftext|>），
但聊天模板结束 assistant 轮用的是 <|im_end|>（248046）。不管的话，
generate() 会一直等一个对话格式下永远不会出现的 token，每次都顶到
max_new_tokens。因此本仓库的 generation_config.json 已设为
eos_token_id: [248046, 248044]。若你自己拼 prompt，请显式传停止符。
用法
见上方 English 小节的代码示例（注意其中的 repetition_penalty=1.1）。
chat template 存放在 tokenizer_config.json 中，与训练时的渲染
逐字一致 —— 用 50 条真实训练样本双模板对照，50/50 完全相同，
生成提示词也相同。
仓库内容
| 路径 |
内容 |
model.safetensors、config.json |
合并后的 bf16 权重，纯文本 |
tokenizer.json、tokenizer_config.json |
tokenizer 与 chat template |
generation_config.json |
已修正停止符 |
lora/ |
未合并的 LoRA adapter |
LICENSE、NOTICE |
Apache-2.0 与归属声明 |
许可证
Apache License 2.0，与基座一致。归属声明见上方 English 小节。
上游 Qwen/Qwen3.5-2B 仓库未附带 NOTICE 文件
（Apache-2.0 §4(d) 只要求在原作有 NOTICE 时传递），本仓库的 NOTICE
由本项目撰写。按 §4(b) 声明：本权重相对基座已被修改 ——
监督微调，且只保留文本解码器。
训练数据中的 replay 部分来自公开指令数据集：shibing624/alpaca-zh
（CC BY 4.0）、llm-jp/oasst1-21k-ja（Apache-2.0）、yahma/alpaca-cleaned
（CC BY 4.0）、OpenAssistant/oasst1（Apache-2.0）。视觉小说语料不随模型发布。
Apache-2.0 不授予 Qwen / Alibaba / Tongyi 的名称与标识权利，
此处仅作描述性归属使用。
