This is a Japanese character-level GPT-2 Small (90M parameters) language model pre-trained on Japanese Wikipedia, the Japanese portion of CC-100, and the Japanese portion of OSCAR. You can use this model directly with a pipeline for text generation. You can also use this model to get the features of a given text. A character-level vocabulary of size 6K is used. To be precise, rare characters may be split into bytes because byte-level byte-pair encoding (BPE) is used. The BPE tokenizer was trained on a small subset of the training data. Since the data were converted into a one-character-per-line format, merge operations never go beyond character boundaries. Note that the tokenizer maps U+0020 to [UNK] because preprocessing eliminated whitespace characters (U+0020) from training data. Use U+3000 (Ideographic Space) instead. — Japanese Wikipedia (as of 20221020, 3.2GB, 27M sentences, 1.3M documents) — Japanese portion of CC-100 (85GB, 619M sentences, 66M documents) — Japanese portion of OSCAR (54GB, 326M sentences, 25M documents) Note that we filtered out documents annotated with «header», «footer», or «noisy» tags in OSCAR. Also note that Japanese Wikipedia was duplicated 10 times…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: ku-nlp
Теги: gpt2, ja, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 928
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.