We introduce Bamba-9B, a decoder-only language model based on the Mamba-2 architecture and is designed to handle a wide range of text generation tasks. It is trained from scratch using a two-stage training approach. In the first stage, the model is trained on 2 trillion tokens from the Dolma v1.7 dataset. In the second stage, it undergoes additional training on 200 billion tokens, leveraging a carefully curated blend of high-quality data to further refine its performance and enhance output quality. Besides PyTorch, you would need a few extra dependencies for Mamba models. We found some of these dependencies picky on PyTorch versions when doing pip install, so the best way is to build from source for all Mamba dependencies if you hit dependency issue with your env: For users using our HF versions of the model, you would need to install the latest transformers which includes our newly merged implementation for our Bamba models: You can utilize our newly contributed HF integration to run inference on our Bamba models: We trained our Bamba model w Note that this training effort was started before FSDP2 and also long before we contributed…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: ibm-ai-platform
Теги: bamba, endpoints_compatible
Лайков: 4 | Загрузок: 16
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
