Rigel is a 2.3B-total, 360M-active-parameter hybrid Mamba-2 + attention mixture-of-experts base model, trained with LM Engine. It used about 117× fewer pretraining FLOPs than Llama-3.2-1B/3B and 53× fewer than Granite-4.2-3B, while landing within a few points of Granite-4.2-3B and Llama-3.2-3B on zero-shot averages and above Llama-3.2-1B. Read the full write-up, including architecture, data curriculum, the multi-chip training infrastructure, and evaluations: Rigel Base: Reaching Llama-3.2 Quality with This repo is the 294,912-token long-context extension, see open-lm-engine/rigel-base-long. This is a base model: it has no chat template and is not instruction-tuned. Rigel’s architecture lives in LM Engine. Installing it and importing lmengine.training registers the architecture with transformers’ Auto` classes, after which the usual Hugging Face API works. Any kernel left out of the list falls back to a pure-PyTorch implementation, so the snippet also runs with an empty list, just slower. To finetune or continue pretraining, point an LM Engine training config at this checkpoint; see the LM Engine README.
Модальности:
Генерация текста
Задача: Генерация текста
Автор: open-lm-engine
Теги: lm-engine, mamba2, moe, hybrid, base-model, en
Лайков: 3 | Загрузок: 340
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
