Chat & support: my new Discord server Want to contribute? TheBloke’s Patreon page This repo contains an experimantal GPTQ 4bit model for Falcon-40B-Instruct. 4-bit GPTQ model for GPU inference 3-bit GPTQ model for GPU inference * Unquantised bf16 model in pytorch format, for GPU inference and for further conversions Please note this is an experimental GPTQ model. Support for it is currently quite limited. It is also expected to be VERY SLOW. This is currently unavoidable, but is being looked at. This 4bit model requires at least 35GB VRAM to load. It can be used on 40GB or 48GB cards, but not less. Please be aware that you should currently expect around 0.7 tokens/s on 40B Falcon GPTQ. AutoGPTQ provides pre-compiled wheels for Windows and Linux, with CUDA toolkit 11.7 or 11.8. If you are running CUDA toolkit 12.x, you will need to compile your own by following these instructions: These manual steps will require that you have the Nvidia CUDA toolkit installed. There is provisional AutoGPTQ support in text-generation-webui. This requires text-generation-webui as of commit 204731952ae59d79ea3805a425c73dd171d943c3. So please first update text-genration-webui to the latest version.…
Модальности:
Генерация текста
Области применения:
Следование инструкциям
Задача: Генерация текста
Автор: philschmid
Теги: RefinedWeb, custom_code, en, endpoints_compatible
Лайков: 2 | Загрузок: 29
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
