Myosotis-1.5-base is the second flagship release from us, built on the FWKV architecture. Myosotis-1.5 is engineered to never truly forget—using a mathematically clamped exponential decay that guarantees an infinite effective context window while maintaining blazing-fast inference on consumer hardware. Myosotis-1 was trained in two distinct stages, leveraging the Chinchilla scaling law (20× parameter count in tokens) for optimal performance. — Data Mixture (SmolLM-Corpus style): — HuggingFaceTB/smollm-corpus — mlfoundations/dclm-baseline-1.0 — HuggingFaceTB/smollm-corpus — HuggingFaceFW/finepdfs-edu — HuggingFaceTB/smollm-corpus — FlameF0X/arXiv-AI-ML — Optimizer: AdamW (fused) — Precision: BF16 Mixed Precision with Gradient Checkpointing Myosotis-1 is fully compatible with the HuggingFace transformers library. This model is dedicated to the open-source community. Built by Daniel B. (FlameF0X) using vast.ai, PyTorch, and the Hugging Face ecosystem.
Модальности:
Генерация текста
Задача: Генерация текста
Автор: FWKV
Теги: fwkv, feature-extraction, rnn, wkv, efficient-inference, custom_code, en
Лайков: 3 | Загрузок: 281
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
