Another experiment with the surgical style-tuning, heavily inspired by Gryphe’s methodology (Gemma-4-31B-StyleTune). I stumbled upon the concept of Style-Tuning by chance, and since the initial results were incredibly promising, I wanted to test the boundaries of this method on the Mistral-NeMo-12B architecture using a highly specialized, modern prose dataset. Instead of a traditional full fine-tune that alters the core reasoning layers of the network, this approach surgically calibrates a single component: 1. Total Freeze: All attention mechanisms and MLP layers (Layers 0–39) remain completely frozen. The underlying logic, instruction-following capabilities, and general world knowledge of Mistral-NeMo are preserved. 2. Targeted Vocabulary Shift: Training is strictly isolated to the lmhead`** (output projection). Retraining only the lmhead` doesn’t make the model inherently smarter or dumber—it simply rewires its linguistic preferences. It swaps the generic, predictable «AI-slop» vocabulary for a rich, varied, and sophisticated prose delivery. * A nice style change compared to vanilla. The Opus style is clearly noticeable even with 1 epoch and a 2e-4 learning rate. Temperature:…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: ewald1976
Теги: nemo, mistral, unsloth, style-tune, 12b, creative, conversational, en
Лайков: 3 | Загрузок: 58
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
