EXL3 (trellis-coded) weights of Qwen/Qwen3.8-Flash-Next in a GGUF container, for Rulith Inference on AMD Strix Halo (Windows). The quantization is turboderp’s. The EXL3 tensors are taken bit for bit from turboderp/Qwen3.8-Flash-Next-exl3 (ExLlamaV3 1.4.4, codebook mul1, calibration 250 rows x 2048) and only re-packed; all credit for the quantization goes to turboderp and the ExLlamaV3 project. The non-EXL3 tensors (hyper-connection mixers, router, norms, embeddings, the n-gram table codec) follow the layout and types Rulith Inference is tuned for. 4.05 bpw is the recommended one; 3.05 bpw is smaller and generates 7-12% faster, at twice the KL divergence (see the comparison below). In both, the first shard holds only the metadata (11 MB), so a client can read the model’s header before the weights; the third is one tensor, the per-layer token embedding table (39.0 / 32.6 GB), which cannot be split. Also here: mtp-Qwen3.8-Flash-Next-lr512-ownhead.gguf, the multi-token-prediction draft for both folders (the model’s MTP layers plus Rulith’s low-rank draft head with its own output head: an EXL3 target cannot lend its head rows). Image input: mmproj-F16.gguf, the model’s vision…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: Rulith
Теги: gguf, exl3, rulith, endpoints_compatible, conversational
Лайков: 3 | Загрузок: 6,410
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.