Slovak GPT-J-162M is the first model released in Slovak GPT-J series and the very first publicly available transformer trained predominantly on Slovak corpus. Since the initial release two other models were made public, Slovak GPT-J-405M and the largest Slovak GPT-J-1.4B. Model is based on GPT-J and has over 162M trainable parameters. † ByteLevelBPETokenizer was trained on the same Slovak corpus. Slovak GPT-J-162M was trained on a privately collected dataset consisting of predominantly Slovak text spanning different categories, e.g. web, news articles or even biblical texts — in total, over 40GB of text data was used to train this model. The dataset was preprocessed and cleaned in a specific way that involves minor but a few caveats, so in order to achieve the expected performance, feel free to refer to [How to use] section. Please, keep in mind that despite the effort to remove inappropriate parts of the corpus, the model still might generate sensitive content or leak sensitive information. This model was trained for almost 37 billion tokens over 69,001 steps on TPU v3-8 pod. The cross-entropy validation loss at the last step was 3.065. Same as the original GPT-J, Slovak…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: Milos
Теги: gptj, Slovak GPT-J, causal-lm, sk, endpoints_compatible
Лайков: 2 | Загрузок: 160
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
