A custom 1.286 billion parameter language model built entirely from scratch — no base models, no fine-tuning, no dependencies on existing LLM frameworks. — Built from scratch — Custom PyTorch implementation. Not a derivative of any existing model. — BPE Tokenizer — Trained a 50,000-token BPE tokenizer on the TinyStories dataset. — Modern Architecture — SwiGLU activations, Rotary Position Embeddings (RoPE), RMSNorm. — Open Source — MIT license. Weights, training code, and inference code are all available. — GGUF Format — Available for use with llama.cpp, Ollama, and other GGUF-compatible runners. The model was trained on the TinyStories dataset — a synthetic dataset of short stories designed for training compact language models. Training was performed on CPU with limited resources (for testing), making this a proof-of-concept for building LLMs from scratch without GPU access.
Модальности:
Генерация текста
Задача: Генерация текста
Автор: AveroLabs
Теги: gguf, sage_1b, transformer, from-scratch, tiny-stories, en, endpoints_compatible
Лайков: 3 | Загрузок: 48
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
