cerc-aai/vicuna-7b-siglip-so400m-finetune-lora - Каталог нейросетей
Генерация текста

cerc-aai/vicuna-7b-siglip-so400m-finetune-lora

Добавлено:

A LLaVA-fork enabling the Mistral-7B & Open-Hermes-2.5 language models to process images This repo and the associated models were created in collaboration between the Robin team at AGI-Collective and Simon Ramstedt, with computing resources from Hessian-AI and OLCF. The Robin Effort: We study different components and methods to merge pretrained vision and language models with the goal to build better visual understanding models. As part of this first milestone we study the pretrained LLMs (Vicuna, Mistral and OpenHermes 2.5) and Vision models (CLIP and SigLIP), further improving capabilities by finetuning the vision encoder. Ideally install into an empty venv (python -m venv venv && source venv/bin/activate) Note that currently Robin-LlaVA inference requires a Nvidia GPU with 24GB+ since CPU inference and quantized inference don’t work reliably yet. Training was done on a machine with 8 x A100 80GB Nvidia GPUs provided by Hessian-AI. We use the following components: — Base LLM: We explore using Vicuna, Mistral and OpenHermes-2.5 — Base Vision Model: We use the SigLIP model since it gives stronger performance on vision benchmarks compared to CLIP — We finetune the Vision Encoder…

Модальности:
Генерация текста


Задача: Генерация текста
Автор: cerc-aai
Теги: llava, endpoints_compatible
Лайков: 2  |  Загрузок: 12

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.