Pythia-410m architecture pretrained with PonderLM-2, the method introduced in PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space (ICML 2026 Spotlight). > TL;DR. Chain-of-Thought scales test-time compute by generating extra > tokens. PonderLM-2 does the same at pretraining time, but in > continuous space: before predicting each next token the model first > emits a few latent thoughts — extra last-hidden-state vectors — and > feeds them back into itself. — Code: LUMIA-Group/PonderLM-2 — Paper: arXiv:2509.23184 — Sibling checkpoint: zeng123/PonderLM-2-Pythia-1.4b Built on top of the Pythia training stack and LLaMA-Factory. The PonderLM baseline implementation is adapted from LUMIA-Group/PonderingLM.
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: zeng123
Теги: gpt_neox, pretraining, causal-lm, gpt-neox, pondering, latent-thoughts, icml-2026, conversational
Лайков: 3 | Загрузок: 125
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
