> ⚠️ Preliminary results — 1 seed, variance not controlled. Every number below is a single training run evaluated once (50 prompts × 200 tokens, fixed seed). Directions are consistent, magnitudes are not validated. Treat as a lab notebook, not a benchmark. Nomade is a 15M-parameter French language model trained from scratch (no 🤗 Transformers, no pretrained weights) on a single GTX 1080 Ti. It is the out-of-domain / robust member of a three-model family: a looped transformer with learned per-token adaptive halting using a percentile threshold — the only variant whose halting is size-invariant and active at inference. — Architecture: decoder-only, LLaMA-style (RoPE, RMSNorm, SwiGLU, QK-Norm, Flash/SDPA attention), looped R=4 (effective depth 32) + per-token adaptive halting: at each iteration the q=30% least-uncertain active tokens freeze (percentile of the live entropy distribution). Zero added parameters. Because q is a proportion, the criterion transfers across model sizes (unlike an absolute threshold). Halting is active during training and at inference. — Size: 15.01M parameters · nembd 256 / nlayer 8 / nhead 4 · context 768. — Language: French, from scratch. — Tokenizer:…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: RDTvlokip
Теги: from-scratch, looped-transformer, adaptive-computation, french, fr
Лайков: 3 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
