Virtuoso-Lite (10B) is our next-generation, 10-billion-parameter language model based on the Llama-3 architecture. It is distilled from Deepseek-v3 using ~1.1B tokens/logits, allowing it to achieve robust performance at a significantly reduced parameter count compared to larger models. Despite its compact size, Virtuoso-Lite excels in a variety of tasks, demonstrating advanced reasoning, code generation, and mathematical problem-solving capabilities. This is the pre-distilled version of Virtuoso-Lite, a standard SFT+RLHF merge training run used to validate the model before proceeding with the additional training steps outlined below. — Architecture Base: Falcon-10B (based on Llama-3) — Parameter Count: 10B — Tokenizer: — Initially integrated with Deepseek-v3 tokenizer for logit extraction. — Final alignment uses the Llama-3 tokenizer, with specialized “tokenizer surgery” for cross-architecture compatibility. — Distillation Data: — ~1.1B tokens/logits from Deepseek-v3’s training data. — Logit-level distillation using a proprietary “fusion merging” approach for maximum fidelity. — License: falcon-llm-license Deepseek-v3 serves as the teacher model, from which we capture logits…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: arcee-ai
Теги: llama, mergekit, merge, conversational, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 106
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
