Karajan42/open_llama_preview_gpt4 - Каталог нейросетей
Генерация текста

Karajan42/open_llama_preview_gpt4

Добавлено:
Karajan42/open_llama_preview_gpt4

The main goal of this project was to evaluate the impact of a good quality dataset on the finetuning of a partially pretrained model. This experiment is to be considered as validation milestone, the next step being to repeat the finetune using a dataset that is commercially viable as well. Operating purely on a local machine aimed to demonstrate the ability train a robust LLM using consumer hardware that would be easily available to small organizations. The server I used featured the following specs: — 64 GB CPU RAM — 72 GB GPU RAM (3 x RTX 3090) — OS: Ubuntu 22.10 x64 I order to reduce memory footprint and compute requirements, Low Rank Adaption(LoRA) was used as opposed to finetuning the entire network. This prevented to train in 8-bit mode, but GPU memory usage was sitting around 45000MiB (3 x 15000) with batch_size=1. Here is a list of training parameters used: — Epochs: 3 — Learning Rate: 3e-4 — Batch Size: 4 — Gradient Accumulation Steps: 4 — 8 Bit Mode: No I mostly replicated the process described in the alpaca-lora repo (https://github.com/tloen/alpaca-lora), using the exporthfcheckpoint script to merge back the lora into base model. Sure! Here’s a simple recipe for…

Модальности:
Генерация текста


Задача: Генерация текста
Автор: Karajan42
Теги: llama, en, text-generation-inference, endpoints_compatible
Лайков: 2  |  Загрузок: 29

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.