The Sequence Radar #559 : Two Remarkable Papers This Week: Self-Improving Agents and the Limits of LLM Memorization
Was this email forwarded to you? Sign up here The Sequence Radar #559 : Two Remarkable Papers This Week: Self-Improving Agents and the Limits of LLM MemorizationAgents that improve themselves and the limits of memorization.Next Week in The Sequence:We dive into safety evals as part of our series about benchmarking. Research cover’s Sakana AI groundbreaking paper about self-evolving models. Our opinion section focuses on the case for spatial intelligence and world models. Engineering will discuss another cool AI framework. You can subscribe to The Sequence below:📝 Editorial: Two Remarkable Papers This Week: Self-Improving Agents and the Limits of LLM MemorizationThis week featured two standout papers that reveal complementary frontiers of AI development: one that pushes the limits of open-ended, self-improving systems, and another that rigorously quantifies how much information large language models can retain. The first, Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents, presents one of the most credible instantiations yet of recursive self-modifying agents. The second, How Much Do Language Models Memorize?, introduces a principled and practically measurable framework for assessing the memorization capacity of modern LLMs. Both contributions illuminate core dynamics of how AI systems evolve, learn, and remember—and together, they paint a vivid picture of our current trajectory in scaling and aligning intelligent systems. The Darwin Gödel Machine (DGM) operationalizes the theoretical idea of self-referential improvement by constructing agents that can rewrite their own code to enhance performance. Built atop frozen foundation models, DGM alternates between self-modification and evaluation, benchmarking candidate agents on real-world coding tasks like SWE-bench and Polyglot. It employs a Darwinian mechanism: each agent is added to an archive, and new agents are generated via mutations of prior ones. Crucially, this enables divergent exploration across the agent design space. The system autonomously discovers new tools, workflows, and strategies, leading to performance improvements from 20% to 50% on SWE-bench. The results suggest that recursive, self-directed optimization is not only feasible but increasingly competitive with manually engineered systems. What distinguishes DGM is its architecture for open-ended discovery. Rather than hill-climbing on a single agent, it maintains a population of diverse, evolving systems—allowing for breakthroughs to emerge from unlikely or initially suboptimal branches. This is a major departure from conventional agent fine-tuning pipelines, which often discard failed explorations. The paper demonstrates that key innovations often trace back to agents that initially underperformed, underscoring the value of preserving and revisiting earlier ideas. With strong safety protocols in place (e.g., sandboxing and human oversight), the DGM framework opens a credible path toward continuously evolving AI systems whose improvements compound autonomously over time. Meanwhile, How Much Do Language Models Memorize? tackles a long-standing and under-specified question at the heart of LLM behavior: what does a model actually retain from its training data? The authors introduce a formal decomposition of memorization into "unintended memorization" (data-specific retention) and "generalization" (distribution-level abstraction). Using a compression-based method inspired by Kolmogorov complexity, they estimate the total number of bits a model can memorize. Their experiments—which span hundreds of transformer models trained on both synthetic and natural datasets—reveal a striking result: GPT-family models retain roughly 3.6 bits per parameter. This figure quantifies model capacity in a practical, interpretable way, and serves as a foundation for analyzing model behavior, privacy risks, and generalization thresholds. Beyond static measurement, the paper derives scaling laws that predict how memorization patterns shift with data and model size. It reveals that models initially memorize data until their capacity saturates, after which generalization begins to dominate—providing a formal underpinning for the widely observed double descent phenomenon. It also shows how membership inference attacks become harder as datasets grow larger relative to model capacity. These results suggest that memorization is a predictable, quantifiable phenomenon, and not merely an emergent artifact. The framework sets the stage for more rigorous evaluation of privacy, reproducibility, and data influence in LLMs. Together, these two papers reveal opposite yet deeply intertwined aspects of AI model development. The Darwin Gödel Machine charts the outer frontier of what self-improving systems might look like when left to explore and evolve. How Much Do Language Models Memorize? brings precision and clarity to a key limitation of such systems: their bounded capacity to retain specific information. One pushes forward the architecture of continual progress; the other grounds that progress in the mathematics of representation. As the field grapples with scale, autonomy, and alignment, both papers offer essential tools for understanding what models can become—and what they can (and cannot) remember along the way. 🔎 AI Research"Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents" – UBC, Sakana AI, Vector Institute "Self-Challenging Language Model Agents" – UC Berkeley & FAIR at Meta "REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards" – OpenThought Lab "SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics" – Hugging Face, Valeo.ai, Sorbonne University "How much do language models memorize?" – FAIR at Meta, Google DeepMind, Cornell, NVIDIA "Protocol Models: Scaling Decentralized Training with Communication-Efficient Model Parallelism" – Pluralis Research 🤖 AI Tech ReleasesMistral CodeMistral released Mistral Code, its new coding assistant. 🛠 AI in ProductionVoice AI at AirbnbAirbnb discusses their use of speech AI capabilities for customer support. 📡AI Radar
You’re on the free list for TheSequence Scope and TheSequence Chat. For the full experience, become a paying subscriber to TheSequence Edge. Trusted by thousands of subscribers from the leading AI labs and universities. |
Similar newsletters
There are other similar shared emails that you might be interested in: