Abstract. We present a four-stage pipeline for producing small language models (1.7B parameters) that exhibit self-critiquing dual-cognition reasoning from a 30B-parameter teacher. The pipeline chains: (1) proof-weighted knowledge distillation from three teacher variants (Instruct, Thinking, Coder) of Qwen3-30B-A3B, each producing distinct capability profiles in the student; (2) Topological Knowledge Distillation (TKD), which uses the Discrepancy Calculus (DISC) framework to decompose the teacher’s output distribution into smooth, jump, and singular-continuous components via bounded variation theory, allocating training capacity to structural boundaries Trained on H100 at BF16 precision, the resulting models demonstrate dual-cognition reasoning — free derivation followed by adversarial…
Модальности:
Генерация текста
Области применения:
Логика и рассуждение
Задача: Генерация текста
Автор: reaperdoesntknow
Теги: qwen3, sft, trl, dual-mind, reasoning, convergent-intelligence, explore-examine-response, convergentintel
Лайков: 3 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
