LAMBA V1.0 is a ~177M-parameter language model trained from scratch on a Mamba-3 (SISO) + Grouped-Query Attention hybrid architecture, for English (primary) and Turkish (secondary). It is a deliberate, honest first step: small enough to train on free/affordable compute (Google Colab A100), built end-to-end — tokenizer, data, pretraining, instruction tuning, and retrieval-augmented tuning — with no warm-start from any existing model. > TR: LAMBA V1.0, sıfırdan eğitilmiş ~177M parametreli, Mamba-3 + GQA hibrit > mimarili, İngilizce (birincil) ve Türkçe (ikincil) bir dil modelidir. Hiçbir hazır > modelden başlatılmadı; tokenizer’dan eğitime tüm hat baştan kuruldu. > Update (v1.1): now runs on CPU — a pure-PyTorch port (lambacpu.py, no Triton/CUDA) > with a decode cache. Run anywhere with —device cpu`; see How to run. The hybrid puts attention layers in the depth-middle so induction heads can form, with Mamba-3 handling the bulk linear-time sequence mixing. 1. Pretraining — 12.0B tokens (EN 6.6B / TR 2.64B / code 1.56B / math 1.2B), WSD schedule, decontaminated against the eval suite. Final loss ~2.50 (PPL ~12). 2. SFT — bilingual instruction tuning. Pivot-language design: reasoning…
Модальности:
Генерация текста
Области применения:
RAG (поиск + генерация)
Задача: Генерация текста
Автор: kdirgul
Теги: mamba_ssm, mamba, mamba-3, state-space-model, gqa, hybrid, turkish, bilingual
Лайков: 3 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
