michel-nano is an ultra-tiny ~6 million parameter base language model trained on 1.14 billion tokens. It was created by merging two intermediate training checkpoints using mergekit to combine their strengths into a single, superior model. This model is a 50/50 SLERP (Spherical Linear Interpolation) merge of two checkpoints from the same training run: Checkpoint A (Step 60,000): Excelled at language modeling and reasoning, achieving the best WikiText perplexity and ARC-Easy scores. Checkpoint B (Step 70,000): Showed stronger grammatical understanding, achieving the best BLiMP score. By merging them, this model inherits the best traits of both—achieving lower WikiText perplexity and higher BLiMP/ARC-E scores than either parent checkpoint individually. Parameters: ~6,000,000 Training Tokens: 1.14 Billion Context Length: 512 tokens Post-training: None (This is a base pre-trained model) The tokenizer is a basic Byte-Pair Encoding (BPE) tokenizer trained from scratch on a subset of 100,000 samples drawn from the same data mixture. It features a compact vocabulary size of 6,000 plus chatml special tokens for future finetuning. As a base model that has not undergone any instruction…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: finnianx
Теги: llama, mergekit, merge, en, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 31
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
