Faithful small-scale (~3.2B total / ~1.10B activated per token) replica of the DeepSeek-V4 architecture, sized to be trainable on rented GPUs and to map cleanly onto the full-scale V4-Flash dimensions for weight slicing. This is a randomly-initialized scaffold — generates noise. Its purpose: — reference architecture for ablation / hyperparameter-search experiments — target for weight transfer / slicing from real V4-Pro / V4-Flash mHC (Sinkhorn-Knopp) · CSA + Lightning Indexer · HCA · pure sliding-window · Shared-KV MQA + grouped output projection (per-group woa) · partial RoPE + output -i rotation · attention sink · DeepseekMoE with sqrt(softplus)` routing · hash-routed early layers · clamped SwiGLU · MTP head · YaRN. Every component is bit-equivalent in math to the official inference/model.py + kernel.py:hcsplitsinkhorn (FP4/FP8 quantization and Hadamard rotation are skipped — those are inference optimizations, not architecture).
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: kshitijthakkar
Теги: deepseek_v4, deepseek-v4, mixture-of-experts, moe, mhc, csa, hca, scaffold
Лайков: 3 | Загрузок: 180
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
