A ~50M-parameter Mamba (selective state-space) causal language model, pretrained from scratch on English Wikipedia. Mamba replaces the attention mechanism of a Transformer with a selective state-space layer, giving linear-time sequence processing instead of the quadratic cost of self-attention. > This is a base model: pretrained only. It has not been fine-tuned, instruction-tuned, > RLHF’d, or aligned in any way. It is a raw next-token predictor intended for research. — Pretrained only. No fine-tuning, instruction tuning, or alignment. It does not follow instructions and has no safety filtering; it simply continues text. — Small. At ~50M parameters it has limited fluency and reasoning; expect frequent hallucination and repetition. — English only. Trained solely on English Wikipedia; other languages are out of distribution. — Domain-narrow. Only Wikipedia was used as training data. Pretrained on the English subset of Wikipedia: over 3 million articles. Evaluated on a held-out set of 10,000 Wikipedia articles that were not seen during training. The training and evaluation loss curves are shown below.
Модальности:
Генерация текста
Задача: Генерация текста
Автор: Vcecca
Теги: mamba, state-space-model, ssm, causal-lm, pretrained, en, text-generation-inference, endpoints_compatible
Лайков: 4 | Загрузок: 167
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
