AIRRC/ELN-Llama-1B-base

This project implements an Evolution Learning Network (ELN) to fine-tune transformer-based models like LLaMA using a combination of Quantized Low-Rank Adaptation (QLoRA) and Genetic Algorithms (GA). The primary objective is to evolve a population of models across multiple generations to optimize for performance (fitness) and specialization, while maintaining diversity. — Efficient model fine-tuning using QLoRA with 4-bit quantization — Evolutionary strategies with tournament selection and blended crossover — Adaptive mutation rates based on generation progress — Comprehensive experiment tracking with WandB — Diversity maintenance through LoRA weight fingerprinting — Name: meta-llama/Llama-3.2-1B — Architecture: Transformer-based causal language model — Type: 4-bit quantization using bitsandbytes — Parameters: — Compute Type: torch.float16 —

Модальности:
Генерация текста

Задача: Генерация текста
Автор: AIRRC
Теги: llama, facebook, meta, llama-3, en, de, fr, it
Лайков: 3  |  Загрузок: 31

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.