This repository contains a series of reference models of varying sizes, released as part of our paper, Densing Law of LLMs. These models were trained to establish a robust scaling law, which serves as a foundational component for calculating the «density» of other Large Language Models (LLMs). The core contribution of our paper is the concept of LLM Density \(rho\), defined as the ratio of a model’s effective parameter size \(hat{N}\) to its actual parameter size \(N\). To accurately determine a model’s effective size, we must first establish a reliable «ruler»—a scaling law that maps training compute to performance on downstream tasks. The models in this repository serve as that «ruler». We trained a series of six models, ranging from 5 million to 800 million parameters, on a consistent dataset. By measuring their loss on various benchmarks, we fitted a precise scaling function. This function allows us to take any other LLM, measure its performance, and infer its effective parameter size by seeing where it lands on our reference scale. These models are released to allow researchers to verify our results, build upon our work, and use this established scale for their own…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: openbmb
Теги: scaling-laws, densing-law, reference-models, en, zh, endpoints_compatible
Лайков: 4 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
