Geneva 12B GammaCorpus v2-500k is a fine-tune of Mistral’s Mistral Nemo Instruct 2407 model. Geneva is designed to outperform other models that have a similar size while also showcasing GammaCorpus v2-500k. — Base Model: mistralai/Mistral-Nemo-Instruct-2407 — Parameters: 12B — Layers: 40 — Dim: 5,120 — Head dim: 128 — Hidden dim: 14,336 — Activation Function: SwiGLU — Number of heads: 32 — Number of kv-heads: 8 (GQA) — Vocabulary size: 217 ~= 128k — Rotary embeddings (theta = 1M)** Geneva-12B-GCv2-500k underwent fine-tuning with 1 A100 GPU for ~25 minutes and trained with the Unsloth framework. Geneva-12B-GCv2-500k was trained for 60 Epochs. If you want to use Hugging Face transformers to generate text, you can do something like this: This model, and all Geneva models, are trained with GammaCorpus. GammaCorpus is a dataset on HuggingFace that is filled with structured and filtered multi-turn conversations. GammaCorpus has 4 version with different sizes in each. These are the following versions and sizes: Here is a link to the GCv1 dataset collection: https://huggingface.co/collections/rubenroy/gammacorpus-v1-67935e4e52a04215f15a7a60 Here is a link to the GCv2 dataset collection:…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: rubenroy
Теги: mistral, text-generation-inference, unsloth, trl, gammacorpus, geneva, chat, conversational
Лайков: 4 | Загрузок: 10
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
