We introduce Trillion-7B-preview, a preview of our latest large language model designed to push the boundaries of multilingual scalability and performance. When comparing performance to training FLOPs for Trillion-7B-preview with competitive models, our model pushes the Pareto frontier, achieving around 66.5% average performance while using significantly fewer compute (~9.3×10²² FLOPs). It outperforms models like Mistral-7B-Instruct-v0.3 and SOLAR-10.7B-Instruct-v1.0 while remaining competitive with models requiring 3-8× more compute such as Qwen2.5-7B-Instruct and EXAONE-3.5-7.8B-Instruct. For full benchmark results, see tables below. — Type: Causal Language Model — Training Stage: Pre-training & Post-training — Architecture: Transformer Decoder with RoPE, SwiGLU, RMSNorm — Number of Parameters: 7.76B — Number of Layers: 32 — Number of Attention Heads: 32 — Context Length: 4,096 — Number of Tokens seen: 2T — Vocab Size: 128,128 Here is a code snippet with applychattemplate that demonstrates how to load the tokenizer and model and generate text. We select a wide variety of benchmarks that evaluate general reasoning, knowledge recall, coding abilities, mathematical reasoning, and…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: trillionlabs
Теги: llama, finetuned, chat, easyquant, gguf, awq, conversational, en
Лайков: 4 | Загрузок: 63
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
