EchoLLaMA is a multimodal AI system that transforms 3D visual data into natural spoken descriptions while enabling interactive dialogue through speech input. This repository contains the implementation of the LLaMA-3.2-1B-Instruct model fine-tuned with Direct Preference Optimization (DPO) for generating rich textual descriptions of 3D scenes. 1. Image Analysis: — DETR (DEtection TRansformer) for object detection — MiDaS for monocular depth estimation — Moondream for holistic image captioning 2. Text Generation: — LLaMA-3.2-1B-Instruct fine-tuned with DPO 3. Speech Synthesis: — Orpheus-3B-0.1-ft TTS model fine-tuned on the Elise English speech dataset 4. Speech Recognition: — SpeechRecognition package for transcribing user speech input — 3D Object Detection Matrix: Constructs a grid-based representation of detected objects with spatial coordinates — Depth-Aware Scene Understanding: Incorporates relative depth values to capture 3D relationships — Natural Language Generation: Produces coherent and contextually rich descriptions — High-Quality Speech Synthesis: Converts textual descriptions into natural-sounding speech — Techniq
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: AquaLabs
Теги: llama, EchoLLaMA, conversational, en, text-generation-inference, endpoints_compatible
Лайков: 4 | Загрузок: 14
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
