thepowerfuldeez/imu1_base

This repository contains the IMU-1 Base model, a sample-efficient 430M parameter language model introduced in the paper IMU-1: Sample-Efficient Pre-training of Small Language Models. IMU-1 is trained on 72B tokens and approaches the benchmark performance of models trained on 56× more data. IMU-1 uses a validated recipe combining recent advances: — QK-norm attention with learnable scale — Per-head gating (sigmoid-based) — Value residual learning — LayerNorm scaling (depth-dependent) — GQA (grouped query attention) — SwiGLU activation — RoPE positional encoding — Optimizer: NorMuon with cautious weight decay, muP parametrization — Schedule: Three-stage WSD (Warmup-Stable-Decay) — Post-processing: Checkpoint EMA (β=0.8) — Training Code: sampleefficientgpt — Stage

Модальности:
Генерация текста

Области применения:
Диалог / чат

Задача: Генерация текста
Автор: thepowerfuldeez
Теги: imu_1, language-model, sample-efficient, pretraining, transformer, conversational, custom_code, en
Лайков: 3  |  Загрузок: 61

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.