Headline result: a GPT-2 Medium–shaped checkpoint (~354 M parameters) trained from scratch without softmax, evaluated on a standardized 22-task in-context learning benchmark, with open kernels where sparse inference is bit-identical to dense on this checkpoint. This Hugging Face repo ships weights, config, and substrate constants only. Inference requires the RRT engine on GitHub (RRT-LLM-FOUNDATION, AGPL-3.0). Stock transformers GPT-2 will produce incorrect outputs. Training is complete. No additional checkpoints are planned from this repository. CORE = mean centered accuracy across 22 in-context learning tasks (DCLM protocol, Karpathy nanochat evalbundle). RRT-355M is 0.021 below the GPT-2 medium foil and 0.035 above** the GPT-2 124M floor — a measurable tradeoff, not a capability collapse. Task asymmetry (RRT − GPT-2 medium, centered score): gains on multiple-choice reasoning (arceasy +0.12, agievallsatar +0.09, openbookqa +0.07); largest regressions on continuation tasks (lambadaopenai −0.16, coqa −0.13, squad −0.07). Not evaluated: MMLU, GSM8K, HumanEval, chat/instruction benchmarks, or fine-tuned downstream tasks. Details: eval/evalsummary.json on this repo; full write-up on…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: Tripstoph
Теги: rrt-gpt2, attention, gpt2, sparsity, triton, research, en
Лайков: 3 | Загрузок: 18
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
