This is a hybrid Mamba-Transformer model based on the Llama 3.1 architecture, distilled from Llama 3.3 70B into a 8B parameter model using Qwerky’s proprietary distillation method. The model uses MAMBA layers interleaved with attention layers for efficient sequence modeling. The results are a 8B parameter model comparable in quality to Llama’s 3.1 8B but running at speeds as fast or faster than Llama’s 3.2 3B model. CUDA is required to run this model. This model requires a CUDA-compatible GPU and cannot be run on CPU-only systems. Make sure you have: — A CUDA-compatible GPU (NVIDIA GPU with CUDA support) — CUDA toolkit installed — PyTorch with CUDA support — Model Type: QwerkyLlamaMambaHybrid (Hybrid Mamba-Transformer) — Architecture: QwerkyLlamaMambaHybridForCausalLM — Base Model: Llama-3.1-8B — Mamba Type: MAMBA — Vocabulary Size: 128256 — Hidden Size: 4096 — Number of Layers: 32 — Number of Attention Heads: 32 — Intermediate Size: 14336 This model can be loaded using HuggingFace Transformers with Au The model uses custom configuration and modeling files that are automatically loaded via the automap in config.json`. — config.json — Model…
Модальности:
Генерация текста
Области применения:
Диалог / чат Следование инструкциям
Задача: Генерация текста
Автор: QwerkyAI
Теги: qwerky_llama_mamba_hybrid, qwerky, mamba, llama, hybrid, causal-lm, conversational, custom_code
Лайков: 4 | Загрузок: 18
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.