Parcae is a novel stable, looped architecture for language models. Unlike traditional fixed-depth architectures that scale by increasing parameter count, looped architectures increase compute (FLOPs) by sending activations through a block of layers in a loop. Parcae addresses training instabilities in prior looped models by recasting looping as a nonlinear time-variant dynamical system and constraining the spectral norm of injection parameters. This checkpoint is the 140M parameter version of Parcae trained on the FineWeb-Edu dataset. You can load the pretrained weights using the following code:
Модальности:
Генерация текста
Задача: Генерация текста
Автор: SandyResearch
Лайков: 3 | Загрузок: 1,327
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
