XpertGPT (Mixture of Experts with Parallelized Multi-Scale Information Transmission) is a sparse, data-efficient recurrent language model for the BabyLM 2026 challenge (Strict-Small (10M) track, 10M words). It combines sliding window attention global streams with sparse parallel expert blocks using Expert Choice routing. ~48.4M non-embedding parameters / ~52.6M total parameters. Custom code (trustremotecode=True). — Architecture: 6 layers of MoEP-MSIT blocks. Each block combines a lower-dimensional dense global sliding window attention layer (dim = 256) with 4 parallel high-dimensional sparse expert blocks (dim = 384) routed via Expert Choice gating. — Track: BabyLM 2026 Strict-Small (10M) (10M words). — Tokenizer: Custom BPE tokenizer (vocab size: 16384). — Revision / Checkpoint: main Intermediate training checkpoints are provided as git revisions named chckM` for the BabyLM challenge fast-eval. Released under CC BY-NC 4.0 (attribution required, non-commercial only). If you use this model or code, please cite (see CITATION.cff): Provenance and integrity fingerprints are documented in PROVENANCE.md.
Модальности:
Генерация текста
Задача: Генерация текста
Автор: anonym5035
Теги: msit_gptbert, babylm, babylm-2026, mixture-of-experts, msit, xpertgpt, custom_code, en
Лайков: 3 | Загрузок: 24
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
