FP8 quantization of amalia-llm/AMALIA-9B-0626-DPO, the European Portuguese language model, for serving with vLLM. To the best of our knowledge this is the first FP8 quantization of AMALIA. Every other quantization published on the Hub targets llama.cpp (GGUF) or Apple Silicon (MLX). The goal was a version that runs well on server GPUs without measurable quality loss. FP8 was chosen over 4-bit formats for one reason that matters specifically for a language-specialised model: FP8 dynamic quantization requires no calibration data. Weight scales are computed per channel and activation scales at runtime. 4-bit schemes need a calibration pass, and the default calibration datasets are English. Calibrating a European Portuguese model on English text would degrade exactly the property that makes the model worth using. Measured against the BF16 original on an NVIDIA DGX Spark (GB10 Grace-Blackwell, 121 GB unified memory, sm_121a). Perplexity over 401 tokens of European Portuguese across four registers. The colloquial passage deliberately contains comboio, pequeno-almoço, casa de banho and telemóvel, so that any degradation of specifically European vocabulary would show up. Language markers.…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: CYBERS3C
Теги: llama, portuguese, european-portuguese, fp8, vllm, compressed-tensors, quantized, conversational
Лайков: 3 | Загрузок: 16
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
