This model is a custom-code derivative of AxiomicLabs/GPT-X2-125M, adapted for experimental long-context causal language modeling and architecture research. The repository includes a Hugging Face Transformers-compatible GPT-X2 implementation with optional Symplectic Metric-RoPE Governor support and training utilities built around CIxOpt, a heterogeneous optimizer developed for efficient parameter routing across large projection matrices, sensitive normalization parameters, and optional governor modules. The model is intended as a research checkpoint for compact long-context generation, positional encoding experiments, optimizer testing, and continued fine-tuning. — Base model: AxiomicLabs/GPT-X2-125M — Model family: GPT-X2 — Task: Causal language modeling / text generation — Language: English — Library: Hugging Face Transformers with custom code — License: Apache 2.0, unless otherwise restricted by upstream model or dataset terms This implementation uses a compact decoder-only GPT-X2-style architecture. text modeltype: gptx2 vocabsize: 32768 hiddensize: 576 numhiddenlayers: 30 numattentionheads: 9 numkeyvalueheads: 3 headdim: 64 intermediatesize: 1536 maxpositionembeddings: 32768…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: reaperdoesntknow
Теги: gptx2, causal-lm, custom-code, gpt-x2, long-context, grouped-query-attention, gqa, rope
Лайков: 4 | Загрузок: 3,208
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
