A QLoRA fine-tune of an uncensored gemma-4-12B-it (abliteration-derived), distilled from Claude Opus 4.7 chain-of-thought traces. The idea was to see how much of the capability loss caused by abliteration could be recovered by training the model to reason in a more structured, deliberative style, without restoring refusal. Evaluated with lm-evaluation-harness. MMLU and GSM8K use the chat template (multi-turn few-shot), since the bare loglikelihood mode noticeably underrates models with a thinking template on this architecture. By community request the table now includes the abliterated pre-SFT model, so the full base → abliterated → SFT trajectory is visible. Every metric tells the same story: abliteration degrades capability and the CoT fine-tune recovers it. On MMLU the SFT closes ~73% of the gap abliteration opened (0.635 → 0.739); on GSM8K abliteration roughly halves math ability and the fine-tune nearly fully restores it (0.496 → 0.920); on WikiText-2 perplexity the SFT model even edges below the clean base (1.717 vs 1.834 bits/byte). Notes: — Run in bfloat16 (gemma overflows in fp16, producing degenerate output); gemma4unified requires a recent transformers. — GSM8K is…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: Rangle2
Теги: gguf, gemma4_unified, image-text-to-text, gemma, uncensored, abliterated, chain-of-thought, claude-opus
Лайков: 3 | Загрузок: 951
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
