TR-HASH MoE 200M is a 201.2M-parameter decoder-only base language model with grouped-query attention, a shared SwiGLU path, and deterministic token-ID routed experts. It is trained on a 70B-token unique selection from a larger pretokenized mixture, replayed to 130B total trained-token exposures. This is a base completion model, not an instruction-tuned assistant. It has no chat template or instruction-following guarantee. Each block combines a wide shared SwiGLU path with a small set of experts selected deterministically from the token ID — routing has no learned gate, so there is no contextual router decision at inference time. Trained on a deterministic replay plan over the 200B-token pretokenized mixture (DCLM, FineWeb-Edu, Stack-Edu, FineMath, InfiWebMath, Cosmopedia v2), selecting ~70B unique tokens and replaying to ~130B total trained-token exposures. Training is complete — the run reached its full ~130B planned token exposure. The raw step-by-step training log is included in this repo as traininglog.csv` for anyone who wants the full curve. A validated held-out evaluation has not been run yet; treat this as a released base checkpoint, not a benchmarked one. For…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: AETHORIA-AI
Теги: base-model, pretraining, tr-hash, mixture-of-experts, gqa, custom-code, en
Лайков: 4 | Загрузок: 699
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
