jetbabareal/Sabir-20M

Sabır-20M, sıfırdan Türkçe olarak eğitilmiş, yaklaşık 20 milyon parametreli, kompakt bir dil modelidir. Adını, standart ‘erken durdurma’ kuralları yerine, eğitim verisindeki dilsel kalıpları derinlemesine öğrenene kadar «sabırla» eğitilmesinden alır. Sabır-20M is a compact, 20-million-parameter language model trained from scratch exclusively in Turkish. Its name, «Sabır» (Patience), reflects its training methodology, which involved patiently continuing the training process to allow for deep pattern internalization, rather than adhering to conventional early-stopping rules. — Mimari / Architecture: Custom NanoGPT (Decoder-Only Transformer) — Parametre / Parameters: ~20.44 Million — Veri / Data: ~1.5M Tokens (Turkish Dialogues) — Context Window: 256 Tokens — Tokenizer: Custom SentencePiece (BPE) — Vocab Size: 8,000 1. MODEL MİMARİSİ VE CONFIG (EĞİTİM KODUYLA BİREBİR AYNI) — class ModelConfig: nlayer = 8 nembd = 384 nhead = 6 blocksize = 256 vocab_size = 8000 dropout = 0.1 config = ModelConfig() device = ‘cuda’ if torch.cuda.is_available() else ‘cpu’ class Head(nn.Module): def init(self, headsize): super().init() self.key = nn.Linear(config.nembd, headsize, bias=False) self.query =…

Модальности:
Генерация текста

Задача: Генерация текста
Автор: jetbabareal
Теги: custom-architecture, nanogpt, turkish, from-scratch, tr
Лайков: 3  |  Загрузок: 77

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.