I finetuned PygmalionAI/pygmalion-6b with QLora for 24 hours on 250k samples. Collected from SODA and Teacher GPT dataset. My first attempt on making LLM model as an entry to Chai competition. — Model type: Chatbot — Finetuned from model : PygmalionAI/pygmalion-6b Pygmalion-6b: https://huggingface.co/PygmalionAI/pygmalion-6b For the training data I use 20% of SODA dadtaset mixed with TeacherGPT roleplay dataset. — Training param >batchsize = 128, >microbatchsize = 4, >numepochs = 1, >learningrate = 3e-4, >cutofflen = 512, >valsetsize = 0 — lora hyperparams >lorar = 16, >loraalpha = 16, >loradropout = 0.05, >loratargetmodules = «qproj kproj vproj» — llm hyperparams >bf16 = False, >loadin8bit = False, >groupbylength = False , >resumefromcheckpoint = None Me: Hi Nathan, how are you doing today Nathan: I’m fine… Me: Then tell me about your day. Nathan: — It was good. We had a lot of fun in school and then we went to the park afterwards.
Модальности:
Генерация текста
Задача: Генерация текста
Автор: alkahestry
Теги: gptj, endpoints_compatible
Лайков: 2 | Загрузок: 7
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
