Boomerang distillation is a phenomenon in LLMs that allows us to distill a teacher model into a student and reincorporate teacher layers to create intermediate-sized models with no additional training. This is the student model distilled from Qwen3-4B-Base from our paper. This model was initialized from Qwen3-4B-Base by copying every other layer and the last 2 layers. It was distilled on 2.1B tokens of The Pile deduplicated with cross entropy, KL, and cosine loss to match the activations of Qwen3-4B-Base. We used the following hyperparameters: — Learning rate: 3e-4 — Learning rate scheduler: cosine — Warmup ratio: 0.01 — Optimizer: AdamW — Adam betas: (0.9, 0.95) — Adam epsilon: 1e-8 — Weight decay: 0.1 — Max. gradient norm: 1.0 — Number of training steps: 500 — Max. sequence length: 2048 — Effective batch size: 2048 — Mixed precision: bf16 — KLDiv weight: 0.1 — Cosine distance weight per layer: 0.1 To interpolate between this model and Qwen3-4B-Base, please use the buildintermediatemodel function from our github repository: Notes: 1. Changing numlayerstopatch changes the size of the intermediate model by patching different numbers of student layers. 2. patchfirstklayers should be…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: Harvard-DCML
Теги: qwen3, conversational, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 407
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
