Atom is a 3.4M parameter causal language model developed by Universal Computing Research. It was pretrained from scratch as a compact research model for studying language-model architecture, data curricula, and small-model benchmarking. — Architecture: causal decoder-only language model — Parameters: 3,412,800 — Layers: 7 — Hidden size: 192 — Attention: 3 query heads and 1 key-value head (grouped-query attention) — Head dimension: 64 — Feed-forward size: 480 — Context length: 512 tokens — Positional encoding: rotary position embeddings (RoPE) — RoPE Theta = 5000.0 — Normalization: RMSNorm — Activation: gated SiLU feed-forward network — Vocabulary size: 4,096 tokens — Tokenizer: custom byte-level BPE, exposed Its custom attention implementation combines grouped-query attention with XSE. Atom uses a custom byte-level BPE tokenizer trained specifically for this pretraining corpus. The tokenizer has a vocabulary of 4,096 tokens and includes dedicated padding, beginning-of-sequence, end-of-sequence, unknown, and end-of-text tokens. Atom was trained on a…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: UniversalComputingResearch
Теги: gpt, causal-lm, decoder-only, grouped-query-attention, rope, swiglu, custom-tokenizer, curriculum-learning
Лайков: 4 | Загрузок: 157
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
