Discord-Micae-Hermes-3-3B is a new finetune on NousResearch/Hermes-3-Llama-3.2-3B. This model serves as a foundation for ongoing exploration into the capabilities of human-adjacent text generation. — Training Schedule: — 17M tokens of 260 thousand single-turn exchanges (STX) – 6 epochs @ 2e-5 — 5.5M tokens of 101 thousand multi-turn chains – 6 epochs @ 2e-5 — Combined dataset – 1 epoch @ 1e-5 — Fine-Tuning Method: LoRA merge (α = 32, r = 8, dropout = 0.1) — LoRA target modules: qproj, vproj — Scheduler: Cosine schedule with 220 warmup steps per phase — Optimizer: torch.optim.AdamW (betas = (0.9, 0.999), amsgrad = False, eps = 1e-8, weight decay = 0.01) — Batching: Effec The model was fine-tuned on the mookiezi/Discord-OpenMicae dataset. — Conversational AI research — Experimentation with dialogue agents trained on Discord data — Chatbots requiring casual, human-like tone — The model inherits potential biases from Discord-style language. — It is not safety-aligned for deployment without moderation. — While it does inhert knowledge from…
Модальности:
Генерация текста
Области применения:
Следование инструкциям Диалог / чат
Задача: Генерация текста
Автор: aaronmoo12
Теги: llama, causal-lm, instruct, chat, fine-tuned, merged-lora, llama-3, hermes
Лайков: 3 | Загрузок: 14
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
