Tucano2-qwen-1.5B-Base is a decoder-only transformer continually pretrained from Qwen3-1.7B-Base. Tucano2 is part of the Polygl0t initiative, which aims to advance language models for low-resource languages. Tucano2-qwen-1.5B-Base shares the same tokenizer as Tucano2-0.6B-Base. Token embedding transplantation via Orthogonal Matching Pursuit was used to adapt Qwen3-1.7B-Base to be more sensitive to the lexical, morphological, and orthographic properties of Portuguese. The model was continually pretrained on approximately 50 billion tokens and achieves state-of-the-art performance across several benchmarks designed to evaluate Portuguese language models. All data, source code, and recipes used to develop the Tucano2 series are open and fully reproducible. — Architecture: a Transformer-based model (qwen3) — Size: 1,510,073,344 parameters — Context length: 4,096 tokens — Dataset(s): — Polygl0t/gigaverbo-v2 — Polygl0t/gigaverbo-v2-synth — Language(s): Portuguese — Batch size: 1,048,576 tokens — Number of steps: 100,000 — GPU: 8 NVIDIA A100-SXM4-80GB — Training time: ~ 215 hours — Emissions: 334 KgCO2 (Germany) — Total energy consumption: 878 kWh This repository has the source code used…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: Polygl0t
Теги: qwen3, text-generation-inference, pt, model-index, co2_eq_emissions, endpoints_compatible
Лайков: 3 | Загрузок: 579
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
