Slovak GPT-J-405M is the second model released in Slovak GPT-J series after its smaller variant Slovak GPT-J-162M. Since then a larger Slovak GPT-J-1.4B was released. Model is based on GPT-J and has over 405M trainable parameters. † ByteLevelBPETokenizer was trained on the same Slovak corpus. Slovak GPT-J models were trained on a privately collected dataset consisting of predominantly Slovak text spanning different categories, e.g. web, news articles or even biblical texts — in total, over 40GB of text data was used to train this model. The dataset was preprocessed and cleaned in a specific way that involves minor but a few caveats, so in order to achieve the expected performance, feel free to refer to [How to use] section. Please, keep in mind that despite the effort to remove inappropriate corpus, the model still might generate sensitive content or leak sensitive information. This model was trained for a bit more than 36.5 billion tokens over 69,001 steps on TPU v3-8 pod. The cross-entropy validation loss at the last step was 2.821. Same as the original GPT-J, Slovak GPT-J learns an inner representation of the language that can be used to extract features useful for…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: Milos
Теги: gptj, Slovak GPT-J, causal-lm, sk, endpoints_compatible
Лайков: 2 | Загрузок: 1,678
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
