> Cria (noun): a baby llama, alpaca, vicuña, or guanaco. Pronounced ˈkrē-ə. > > ~ Merriam-Webster cRia-LM-75M-Instruct is a 75.7M-parameter instruction-tuned language model built as a Relaxed Recursive Transformer (RRT). It uses a shared 11-layer recurrent block evaluated twice, with pass-specific LoRA parameters on the second traversal. The model starts from cRia-LM-75M, then adds supervised instruction tuning and preference optimization. It ships with a native chat template using and markers. cRia-LM-75M-Instruct keeps the base model’s 13 unique Transformer layers: This gives an effective depth of 24 Transformer layers while storing the main parameters for 13 unique layers. The recurrent block shares its base weights across both passes. Rank-172 LoRA updates provide separate capacity on the second pass. The full architecture and base-model history are documented on the cRia-LM-75M model card. Post-training starts from cRia-LM-75M. Base-model training details are kept on that model card. The first part uses the full training split of HuggingFaceTB/smol-smoltalk. Only assistant turns contribute to the language-model loss. HuggingFaceTB/SmolLM2-360M-Instruct supplies…
Модальности:
Генерация текста
Области применения:
Диалог / чат Следование инструкциям
Задача: Генерация текста
Автор: sz14
Теги: rrt, recursive-transformer, small-language-model, knowledge-distillation, custom_code, instruct-tuned, dpo, conversational
Лайков: 3 | Загрузок: 226
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.