Syed-Hasan-8503/Llama-3-8B-NOLA

Llama-3-8B-NOLA is a fine-tuned variant of meta-llama/Meta-Llama-3-8B on OpenAssistant/oasst1 for 100 steps only. The goal of this experiment was to try out this new technique NOLA and see the the number of trainable parameters. Due to limited compute, the results of this experiment might not be satisfactory. 1x A5000 was used for this experiment. NOLA is a novel approach for fine-tuning large models such as LLMs and Vision Transformers. Similar to LoRA, NOLA uses a low-rank decomposition of weight matrices for the fine-tuning step. However, LoRA face two primary limitations: The parameter count is lower-bounded by the rank one decomposition The extent of reduction is heavily influenced by both the model architecture and the chosen rank. NOLA brings parameter count felexiblity to LoRA. NOLA achieves this by re-parameterizing the low-rank matrices in LoRA using linear combinations of randomly generated matrices (basis) and optimizing the linear coefficients only. This approach allows us to decouple the number of trainable parameters from both the choice of rank and the network architecture. ** train metrics epoch = 0.0907 totalflos = …

Модальности:
Генерация текста

Задача: Генерация текста
Автор: Syed-Hasan-8503
Теги: llama, text-generation-inference, endpoints_compatible
Лайков: 3  |  Загрузок: 34

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.