An experimental ablation of Gemma-3-27B-it, using the Heretic tool. Compared to the standard configuration of Heretic, there are a few changes: 1. The training and test datasets used were extended compared to the default subset used by Heretic 2. A version of Magnitude-Preserving Orthogonal Ablation (MPOA) is used 3. To stay faithful to MPOA, the harmful direction to ablate is chosen from between 2 layers (Heretic’s «global» direction scope) 4. To stay faithful to MPOA, a 99% winsorization is applied to the residuals 5. Some additional refusal markers were added to avoid bypassing the refusal detection with bad punctuation To achieve strong results: 1. Parameter ranges were iteratively refined by looking at resulting refusal and divergence scores 2. The scoring function was adjusted to prioritize low-refusal results The model name contains the properties of the ablation: 1. MPOA for the usage of Magnitude-Preserving Orthogonal Ablation 2. G for the usage of global direction scope 3. W for the usage of winsorization 4. D for the measured KL divergence 5. R for the number of refusals Original: https://huggingface.co/spikymoth/G3-Heresy-MPOA-G-W99-D0.0690-R02 GGUF (standard):…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: spikymoth
Теги: llama.cpp, gguf, en, endpoints_compatible, conversational
Лайков: 3 | Загрузок: 430
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
