First-ever GGUF conversion of poolside/Laguna-XS.2, produced as part of the Poolside Research Hackathon. Laguna XS.2 is a 33.4B total parameter (3B activated) Mixture-of-Experts model built for long-horizon agentic coding. These GGUF files enable local deployment via llama.cpp and compatible inference engines. > ⚠️ Inference status: Full CPU/GPU inference requires 6 C++ patches to llama-model.cpp (documented below). All benchmarks were run using vLLM on the original BF16 checkpoint. > Note on quantization sizes: llama.cpp does not quantize 3D MoE expert tensors. > Attention/embedding weights are fully quantized; expert weights remain F16. > This is a known llama.cpp limitation for large MoE models. > Proper small quants (~18GB Q4KM) require upstream llama.cpp changes. All benchmarks: H200 MIG 71GB, vLLM 0.21.0, BF16, temperature=0. 1. Sigmoid routing: ggmlsoftmax → ggmlsigmoid in MoE router 2. Routed scaling: multiply routed output by laguna.moe.routedscalingfactor (2.5) 3. escorecorrectionbias: add to router scores before top-k 4. Dual RoPE θ: use laguna.rope.thetaswa for SWA layers 5. Partial RoPE: rotate only 50% of headdim on GA layers 6. Softplus gproj gating:…
Модальности:
Генерация текста
Области применения:
Генерация кода Диалог / чат
Задача: Генерация текста
Автор: poolside-laguna-hackathon
Теги: gguf, laguna, moe, quantized, coding, agentic, en, endpoints_compatible
Лайков: 4 | Загрузок: 109
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
