21.37 tok/s on one RTX 5090 32 GB with 128 GB RAM, using the recipe and patched llama.cpp runtime to stream experts from a Samsung 990 PRO SSD. Ryzen 9950X3D, Linux, 16 GiB GPU expert cache and 64 GiB RAM expert cache. This is a 359.32 GB mixed-precision GGUF, split into ten shards: — IQ2_XXS routed experts; every expert is retained (384 per layer, 40 layers). — Q8 dense and Engram weights, with BF16 embedding/output and existing F32 tensors retained. — Both full Engram tables retained. No abliteration, expert pruning or router bias. — Text-chat template embedded and checked against DeepSeek’s official encoder. This is not a 2-bit quant of every weight. The two Engram shards alone total 208.90 GB. The patched runtime is required for the tested SSD-streaming setup; ordinary GGUF support does not imply support for this architecture or memory layout. Speed: median of three runs per phase, 30–31 input tokens, 128 output tokens, 8,192 configured context, one slot, temperature 0, no KV prefix reuse. Fresh prompts follow eight functional checks; these are not cold-disk runs. All three use the same runtime and cache budgets. These are short-prompt decode measurements, not throughput with…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: acrogenesis
Теги: gguf, llama.cpp, deepseek, mixture-of-experts, quantized, iq2-xxs, endpoints_compatible, imatrix
Лайков: 3 | Загрузок: 2,201
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
