Standard trained 10 epochs; Prisma (G²LU + WoRPE) shown at 1 epoch — the point is LR tolerance, not epoch-matched comparison. The regularization stack (mirroring + G²LU + WoRPE) enables training at 100x the standard learning rate without instability. Prisma 357M trained on ~30B tokens (OpenWebText 20% + FineWeb-Edu 10BT continued training), compared against published models at similar scale. Key findings: — Beats GPT-2 medium on 5/8 benchmarks (ARC-C, ARC-E, BoolQ, HellaSwag, PIQA) with 25% less training data. — Beats Baguettotron (200B) on 6/8 benchmarks — including PPL — with 7x less data. — BoolQ 0.620 exceeds all models except LFM2 (10000B) and SmolLM2 (4000B). The anti-memorization properties of G²LU force genuine comprehension instead of statistical shortcuts. — ARC-Easy 0.548 — the largest absolute gain over GPT-2 medium (+11.2pp). FineWeb-Edu knowledge absorbed efficiently through G²LU’s relational features. — Prisma wins on reasoning benchmarks (ARC, HellaSwag, PIQA, BoolQ). Models trained on 20-300x more data win on content prediction (LAMBADA, PPL). The architecture trades raw memorization for data-efficient knowledge application. > Note: usefast=False is required. The…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: y3i12
Теги: prisma, custom_code, en
Лайков: 4 | Загрузок: 18
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
