Google proposes wake-sleep cycle for continual-learning language models

Google Research has proposed a Sleep framework that splits a continually learning language model’s operation into active “wake” periods and offline “sleep” periods. The aim is to move information acquired during interaction into longer-lived model parameters while reducing catastrophic forgetting. Even so, this remains a research proposal, not a released production system.

In the wake phase, the model receives new inputs and holds rapidly changing information in high-frequency components that serve as short-term memory. The research paper describes a Continuum Memory System: earlier blocks update more often, while later blocks update less often and are intended to retain more stable, long-term knowledge.

The sleep phase brings little or no external input. Instead, the model updates its parameters. The first stage, memory consolidation, uses “Knowledge Seeding” to distill a pre-update teacher state into a larger-capacity student. It activates a new low-rank expert in a slower memory block, trains that expert through on-policy distillation and reinforcement-learning-based imitation, applies the pending update, and resets temporary experts in the faster block for reuse.

The second stage, called dreaming, generates synthetic practice data. The proposed system randomly activates non-standard mixture-of-experts routes to create candidates, ranks examples with a gradient-based importance measure, and uses the selected examples for supervised and reinforcement-learning refinement. The sleep analogy does not mean the model independently changes its architecture: it follows human-defined training rules and activates dormant, pre-allocated parameter capacity across sleep cycles.

In their own experiments, the authors reported higher scores. On Qwen3-8B, Sleep scored 79.2 on AIME-24, 69.0 on AIME-25 and 46.1 on HMMT-25 using average@16, compared with 76.6, 67.4 and 45.1 for their on-policy self-distillation baseline. In a no-context SQuAD test, their four-level Sleep transformer scored 48.9 in a single-passage setting and 46.2 after continued pretraining on 200 passages, versus 46.7 and 43.2 for SEAL.

The paper also reports that a fine-tuned Hope/Sleep model stayed nearly perfect on BABILong at context lengths up to 10 million tokens. That is narrower than saying a deployed model can natively use a 10-million-token prompt. The authors say all small models in the comparison were fine-tuned under the benchmark protocol and suffered substantial drops without fine-tuning. The BABILong paper confirms that the benchmark provides splits up to 10 million tokens, but it does not independently reproduce the Sleep result.

None of the cited sources independently reproduces the continual-learning, reasoning, knowledge-incorporation or long-context results. The sources also provide no production deployment, release schedule, full public implementation or complete accounting of total capacity, compute and energy as repeated sleep cycles activate additional experts.