The objective of the ‘Telco Transformer’ initiative is to pre-train a language model for the telecom industry to understand complex, contextual relationships in domain specific text data. Business hypothesis hinges on non-standard natural languages, such as components of a telecom system, technical terminology, and rich knowledge from multiple subdomains. This builds a strong case for pretraining a model from scratch. It constitutes a custom tokenizer to capture telco vocabulary, large scale unsupervised pre-training that is paired with supervised fine tuning to perform well on downstream tasks. This model will be able to complete sentences, and answer questions with accuracy that is superior to RAG or a fine tuned model. Model V1 achieves the following results on the evaluation set: — Loss: 3.8171 The following hyperparameters were used during training: — learningrate: 3e-05 — trainbatchsize: 16 — evalbatchsize: 8 — seed: 42 — distributedtype: sagemakerdataparallel — numdevices: 8 — totaltrainbatchsize: 128 — totalevalbatchsize: 64 — optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 — lrschedulertype: cosine — lrschedulerwarmupsteps: 300 — numepochs: 30 -…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: subhashtalluri
Теги: gpt2, generated_from_trainer, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 95
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
