Nous Consilience 40B is a generative text model, pretrained from scratch in a decentralized fashion over the internet. This model is automatically updated every 500 training steps, with the latest checkpoint uploaded here from the ongoing pretraining dashboard. Model Type: Decoder-only transformer Parameters: 40 billion Architecture: DeepSeek v3 + MLA (Dense version without MoE routers) Pretraining Data: 20T tokens, Merge of FineWeb, FineWeb 2 and The Stack v2 Training Duration: TBD Optimizer: DisTrO, decentralized version For training data, we combined FineWeb (14T), FineWeb-2 with some less common languages removed (4T), and The Stack V2 (~.2T, upsampled to 1T tokens). We chose these datasets over more specialized pre-training datasets that aim to purely increase benchmark performance. Our goal with Consilience is to make a true «base» model — one representative of the entirety of the creative output of humanity, and not merely trying to win the benchmaxxing game. Additionally, we’re training this model continuously without a final data «annealing» step. While annealing helps base models respond more accurately to benchmarks and improves usability, it may potentially constrain…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: PsycheFoundation
Теги: deepseek_v3, conversational, en, zh, ru, de, ja, es
Лайков: 3 | Загрузок: 14
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
