0.6B K-5-bounded chat model with general chat, model identity, and format steerability installed by a behavior SFT on the blend base (chatty v2). Part of the LittleLearner scale-up study (pedagogically-controlled knowledge exposure): Qwen3 dense LMs trained on a corpus filtered to U.S. K-5 material (bounded) vs an unfiltered FineWeb-Edu corpus (unbounded), to measure what an interpretable knowledge boundary costs and grants. — Architecture: Qwen3 dense (Qwen3ForCausalLM). — Size: 0.617B params, hidden 1536, 20 layers, 12 query / 6 KV heads, FFN 4096. Context: 4096. — Tokenizer: custom 64k byte-level BPE with per-digit splitting (ChatML special tokens). — Pretraining: 88B tokens on K-5 LittleCurriculum (FineWeb-Edu filtered to U.S. grades K-5). WSD schedule, sharded Muon, MXFP8, Megatron-Core on 8xB200. — SFT: behavior SFT directly on the cooloff-blend base (no intermediate SFT stage): K-5 math CoT (30k) + smoltalk general chat (15k) + K-5 GSM8K + format-control pairs (answer-only, show-steps, length constraints; user-turn and system-turn variants) + LittleLearner identity data. fp32 master parameters, lr 1e-5, 1 epoch. The model chats on casual prompts, states that it is…
Модальности:
Генерация текста
Области применения:
Следование инструкциям Диалог / чат
Задача: Генерация текста
Автор: littlelearner
Теги: qwen3, littlelearner, bounded, instruct, conversational, en, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 116
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
