This is 8-bit GPTQ version of Meta-Llama-3.1-8B-Instruct. Quantization has been done using AutoGPTQ library. Starting with transformers >= 4.43.0 onward, you can run conversational inference using the Transformers pipeline abstraction or by leveraging the Auto classes with the generate() function. Make sure to update your transformers installation via pip install —upgrade transformers and you have Autogptq, optimum installed. LLaMA-3.1 supports multiple tool use formats. You can see a full guide to prompt formatting here. Tool use is also supported through chat templates in Transformers. Here is a quick example showing a single simple tool: You can then generate text from this input as normal. If the model generates a tool call, you should add it to the chat like so: and then call the tool and append the result, with the tool role, like so: After that, you can generate() again to let the model use the tool result in the chat. Note that this was a very brief introduction to tool calling — for more information, see the LLaMA prompt format docs and the Transformers tool use documentation. To download Original checkpoints, see the example command below leveraging huggingface-cli:…
Модальности:
Генерация текста
Области применения:
Диалог / чат Следование инструкциям
Задача: Генерация текста
Автор: iqbalamo93
Теги: llama, facebook, meta, llama-3, autoquant, gptq, 8 bit, conversational
Лайков: 3 | Загрузок: 162
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
