❤️ This repo contains the model LLaMA-8x265M-MoE(970M totally), which activates 2 out of 8 experts (332M parameters). This model is trained from scratch with FP32 precision. We firstly train the model through wikipedia dataset with 1 epoch and then through 10% of C4 dataset (10 data shards among 1024 data shards) with 1 epoch. This is NOT fine-tuned by instruction pairs, so it may not be good enough to act like a chatbot. 📢 This series also includes a dense version (without MoE structure), see 🤗this repo. We use the «Average number of tokens verified» $N$ ( see reference link ） as the metric to evaluate these models. This metric demonstrates that giving the same input to the small speculative model and llama-7b, counting from the first predicted tokens, how many successive tokens in the output sentence of the small speculative model are the same as the output sentence of the llama-7b. Supposed that the small speculative model can have a hit rate $p$ for the next token when giving the same input. Then we have For the MoE model, we only show the accuracy of how this small speculative model approximates the performance of llama-7b. In practice, to achieve physically low latency,…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: JuncaiL
Теги: llama_moe, MoE, custom_code, en
Лайков: 3 | Загрузок: 19
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
