Q-Omni-53M-Experimental A 53M-parameter ternary decoder over one shared text / image / audio vocabulary > [!IMPORTANT] > This is an experimental research release, not a product. It is small, fluent but often factually wrong, and its image/speech capabilities are weak or unusable — see Results below before using it for anything beyond experimentation. > > Learn more: a full write-up of how it works is on the Q-Project blog, and you can run it live in your browser — no server, no Python, no sign-in. Q-Omni-53M-Experimental is a 53.2M-parameter decoder-only model developed by Q-Project. Its weights are ternary ({-1, 0, +1}, quantisation-aware from step one of training), and instead of a trained embedding table it uses a frozen 512-bit fingerprint per token, shared across one 49,280-token vocabulary that spans text, image tokens and audio tokens. A fixed circuit resolves arithmetic, dates and similar exact operations inside the generated token stream. The architecture is our own; it was pretrained from scratch on a single V100 GPU. General small-model ternary-quantisation research (in the spirit of BitNet b1.58) inspired the weight format. 7 layers, hidden 896, 14 query / 2 KV heads…
Модальности:
Генерация текста Мультимодальность
Задача: Генерация текста
Автор: q-project
Теги: qomni, experimental, causal-lm, custom-architecture, ternary-weights, bitnet, frozen-embeddings, multimodal
Лайков: 3 | Загрузок: 181
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.