Model collection | Technical report | Training code (coming soon) | Evaluation code NCP-ArchPreview is a latent-space autoregressive language model developed by The NCP Team at Shanghai AI Lab and LUMIA Lab, Shanghai Jiao Tong University. It learns to predict both the next token and the next concept: a representation spanning a short group of tokens in a learned latent space. Concept predictions guide the token decoder, while generation retains the standard next-token interface. This is the Stage 1 base-model release, following large-scale pretraining on Dolma 3 Mix. The architecture follows the OLMo 3 7B token-level design and adds a Concept Module, a product-quantized concept vocabulary, and hierarchical residual connections, bringing the total parameter count to approximately 8.94B. — Joint token and concept learning. Next Concept Prediction (NCP) supplies explicit supervision over a latent sequence at one quarter of the token sequence length. — Pretraining at scale. The report describes training on 5.73T Dolma-3 tokens. Stage 1 reaches the OLMo-3-7B final training loss using 51.3% of its training tokens, corresponding to 1.95x convergence in token budget. — Stronger Stage 1…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: ArchSpace-Collection
Теги: ncp_olmo3, bfloat16, causal-lm, conceptlm, custom-code, latent-space-model, ncp, olmo3
Лайков: 4 | Загрузок: 593
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
