Srijan Srivastava India Srivastavavsrijan321@gmail.com QCoreNest@gmail.com Technical Report (Zenodo)📑 Technical Report (GitHub)📑 Technical Report (Hugging Face)📑 Introduction of the Transformer neural network architecture in the famous Attention Is All You Need paper has created a huge wave of AI development in recent years. The scaled dot-product attention allows for information to be processed with higher efficiency and quality, which the previous RNN-based models lacked. However Transformer-based models comes with their own set of challenges, particularly with parameter efficiency for tiny scale models. At such tiny scale a Transformer model essentially uses more parameter than it really should. This regime is very under-explored and for good reasons however exploring it might allow us to discover interesting insights about the Transformer. So here-in this paper I am introducing Silia, a novel neural network architecture designed for efficient modelling & classification tasks under severe parameter budget. Training on Fineweb-edu dataset shows very competitive performance on HellaSwag, PIQA and LAMBADA benchmarks, validation loss and generation quality to other models with…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: Srijan-Srivastava
Теги: text-generation-inference, en
Лайков: 3 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
