aiprimebuilds/LFM2.5-24B-A2B-GPTQ - Каталог нейросетей
Генерация текста

aiprimebuilds/LFM2.5-24B-A2B-GPTQ

Добавлено:
aiprimebuilds/LFM2.5-24B-A2B-GPTQ

4-bit GPTQ build of LiquidAI/LFM2.5-24B-A2B, built to serve an office of agents from one RTX 4000 Ada 20 GB. Also tested on a 16 GB RTX 4080 SUPER. Within noise everywhere except GPQA Diamond: 11 fewer correct out of 198. The RTX 4000 runs all 8 users at once, even with 32k prompts. The 4080 SUPER generates faster but fits only about two long prompts at a time; the rest wait. RTX 4000 Ada: 4 users comfortably, 6 with slowdowns. 4080 SUPER: 1–2. The limit is how many conversation histories fit in the KV cache (KV cache tokens ÷ average session size). — Fidelity: wikitext-2 perplexity 15.69 vs 13.81 for BF16. KL divergence 0.365 mean. 78% top-token agreement. — KV cache: every GPTQ number uses an FP8 KV cache. The BF16 reference uses BF16. — Tool calls: 96–98% valid. On vLLM 0.31.0, streaming sometimes cuts off a call’s arguments. — Sampling: the server default is temperature 0.1. For agents, use about 0.4.

Модальности:
Генерация текста

Области применения:
Диалог / чат


Задача: Генерация текста
Автор: aiprimebuilds
Теги: lfm2_moe, gptq, 4-bit, int4, moe, lfm2.5, liquid, vllm
Лайков: 3  |  Загрузок: 240

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.