INF-34B has 34 billion parameters with a context window length of 32K, and is trained on about 3.5T well-processed tokens from English and Chinese bilingual corpus. Compared with open source models of the comparable size, INF-34B not only provides competitive performance in the OpenCompass evaluation, but also has impressive potential on both finance and healthcare domains. Besides, the quantized INF-34B runs on graphics cards of 24GB VRAM with negligible accuracy loss, which facilitates commercial applications, especially low-resource scenarios. — Detailed for Training GPT Model: We provide comprehensive details about our model pretraining and alignment, including high-quality data pipeline, instruction data preparation, and quantization results etc. — Superior Performance on Benchmarks: We demonstrate superior performance of the INF-34B models by comparing against two competitors with comparable model size, Qwen1.5-32B and Yi1.5-34B, on the public OpenCompass benchmarks. We release the base and chat models with 34B parameters based on the LLaMA framework, while using LayerNorm with zero-centered gamma instead of RMSNorm for training stability. Please note that you could use our…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: infly
Теги: inflm, conversational, custom_code, 4-bit, awq
Лайков: 3 | Загрузок: 18
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
