LMDeploy supports LLM model inference of 4-bit weight, with the minimum requirement for NVIDIA graphics cards being sm80. Before proceeding with the inference, please ensure that lmdeploy(>=v0.0.4) is installed. You can download the pre-quantized 4-bit weight models from LMDeploy’s model zoo and conduct inference using the following command. Alternatively, you can quantize 16-bit weights to 4-bit weights following the «4-bit Weight Quantization» section, and then perform inference as per the below instructions. As demonstrated in the command below, first convert the model’s layout using turbomind.deploy, and then you can interact with the AI assistant in the terminal If you wish to interact with the model via web ui, please initiate the gradio server as indicated below: Subsequently, you can open the website http://{ipaddr}:{port}` in your browser and interact with the model We benchmarked the Llama 2 7B and 13B with 4-bit quantization on NVIDIA GeForce RTX 4090 using profile_generation.py. And we measure the token generation throughput (tokens/s) by setting a single prompt token and generating 512 tokens. All the results are measured for single batch inference. — generate…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: lmdeploy
Теги: internlm, feature-extraction, text-generation-inference, custom_code
Лайков: 3 | Загрузок: 26
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
