GGUF quantizations of Lotus-1, the 35B-A3B roleplay model behind Sutorichat, for KoboldCpp, llama.cpp, LM Studio and text-generation-webui. Only about 3B parameters are active per token, so the model decodes at small-model speed: the quant you pick is about what fits in memory, not about speed. Converted with llama.cpp converthftogguf.py (bf16) and llama-quantize, no importance matrix. The embedded chat template defaults to non-thinking**, so every loader that reads the GGUF template produces the right prompt with no flags. Sizes are before the KV cache; at 32k context budget another 2 to 3 GB. Because experts are offloaded well, partial GPU offload works: a 12 GB card plus 32 GB of RAM runs Q4KM at usable speed with —n-gpu-layers set to what fits. 1. Load the GGUF with a 32k context: — KoboldCpp: koboldcpp —model Lotus-1-Q4KM.gguf —contextsize 32768 —gpulayers 99 — llama.cpp: llama-server -m Lotus-1-Q4KM.gguf -c 32768 -ngl 99 —jinja — LM Studio: load the file, set context to 32768 in the model settings. 2. SillyTavern, API: Text Completion, pick your backend. 3. Advanced Formatting: import sillytavern/Lotus-1.context.json (Context Template) and…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: yashsutorichat
Теги: llama.cpp, gguf, roleplay, character, sillytavern, chat, qwen3.5, moe
Лайков: 3 | Загрузок: 1,163
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.