A small ~110M parameter language model implementing the DeepSeek-V4 architecture from scratch. This is the pretrained base model — see cmpatino/nanowhale-100m for the SFT/chat version. This model implements key DeepSeek-V4 innovations at a miniature scale: — Multi-head Latent Attention (MLA): Compressed KV cache via latent projections — Mixture of Experts (MoE): Sparse activation — only 2 of 4 experts per token — Hyper-Connections: Multi-copy hidden states with learned Sinkhorn routing replacing residual connections — SwiGLU FFN with configurable limit — Grouped output projection (o_groups) — Dataset: HuggingFaceFW/fineweb-edu (streaming) — Steps: 5,000 — T
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: cmpatino
Теги: deepseek_v4, deepseek, moe, causal-lm, pretrained, conversational, custom_code, en
Лайков: 4 | Загрузок: 97
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
