Bhineka-GPT-500M is a bilingual Indonesian-English causal language model built with a Llama-style decoder-only architecture. The final pretraining checkpoint contains 556.3M trainable parameters and was validated on 53.7M held-out tokens across English web, Indonesian web, code, and math domains, reaching an overall validation loss of 2.5355 and perplexity of 12.62. The project is designed as an end-to-end training pipeline, covering dataset download, text cleaning, language filtering, deduplication, tokenizer training, shard building, curriculum sampling, pretraining, supervised fine-tuning, direct preference optimization, and final Hugging Face export. The model is intended for Indonesian and English text generation tasks such as question answering, summarization, rewriting, translation, technical drafting, code assistance, document understanding, and structured Markdown or JSON-style responses. — Model type: decoder-only causal language model — Architecture: Llama-style Transformer — Languages: Indonesian and English — Parameters: 556,269,696 in the validated final pretraining checkpoint — Context length: 2048 tokens — Tokenizer: BPE tokenizer trained for this project -…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: BhinekaIntiLabs
Теги: gguf, llama, causal-lm, indonesian, english, bilingual, llama-style, bhineka
Лайков: 3 | Загрузок: 78
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
