A natively recursive language model based on Qwen3.5-35B-A3B, trained with Rejection Sampling SFT (RS-SFT) to solve long-context tasks by writing Python code in a persistent REPL. This model is a LoRA fine-tune (rank 32) of Qwen/Qwen3.5-35B-A3B trained on 3,644 mined correct trajectories to generate code that decomposes long-context tasks into manageable sub-problems. Architecture: Mixture-of-Experts, 35B total parameters, 3B active per token Training: RS-SFT (Rejection Sampling SFT) on iteratively mined trajectories Training API: Tinker (remote GPU training) The model operates within an RLM (Recursive Language Model) scaffold: 1. The full input is stored as a context variable in a Python REPL — it never enters the model’s context window 2. The model sees only metadata about the context (length, prefix, available functions) 3. The model writes Python code to chunk, search, and aggregate over the context 4. llmquery(text) recursively invokes the model on substrings 5. FINAL(answer)` terminates with the answer This enables processing of millions of tokens with a 32K context window. Standard GRPO for RLM training creates a zero-sum tradeoff: gains on some benchmarks cause regressions…
Модальности:
Генерация текста
Области применения:
Генерация кода
Задача: Генерация текста
Автор: omar81939
Теги: peft, recursive-language-model, long-context, code-generation, lora, rlm, en
Лайков: 4 | Загрузок: 4
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
