BananaMind-KV1-8M-2Bit-Experimental is a Hugging Face Transformers causal language model repository for the first Project KV1 checkpoint. This is an 8.13M parameter BananaMind KV1 model trained for text generation with KV-cache-aware 2-bit K/V quantization. The model weights are stored normally in model.safetensors; the experimental part is the KV1 low-bit K/V path used during training and generation, with generation-time cache entries affine-quantized to 2 bits and bit-packed. — Model type: decoder-only causal language model — Architecture: KV1ForCausalLM — Model type ID: bananakv1 — Parameters: 8,130,816 — Weights: FP32 safetensors — Context length: 1024 tokens — Vocabulary size: 8192 — Hidden size: 256 — Layers: 8 — Attention heads: 8 — KV heads: 2 — Intermediate size: 768 — KV cache: KV-cache-aware 2-bit affine K/V path, packed as a 2-bit cache when usecache=True — Training data: HuggingFaceFW/fineweb-edu, sample-10BT subset — config.json maps A — model.safetensors contains the model weights. — tokenizer.json and tokenizerconfig.json contain the tokenizer. — generationconfig.json contains default generation settings. -…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: BananaMind
Теги: banana_kv1, causal-lm, custom-code, trust-remote-code, kv-cache, kv-cache-quantization, kv-cache-aware-training, 2-bit-kv-cache
Лайков: 4 | Загрузок: 22
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
