Wednesday / AI 101 series: Reinforcement learning history and what’s happening now in the world of RL

Friday / Interview with Akash Parvatikar, AI Scientist at HistoWiz

SEED-GRPO added semantic entropy, making models more uncertainty-aware.

Curriculum-based GRPO adjusted for difficulty and length, letting weaker models climb steeper learning curves.

GRPO with length penalties or format-aware rewards kept models from gaming the system with bloated outputs.

Flow-GRPO that we previously covered – allowed to apply RL to images.

In the narrow sense, GRPO really was flawed, and many of its fixes look obvious in hindsight.

In the expansive sense, GRPO had become more than an algorithm. It was a family of methods, continuously refined, endlessly iterated, and responsible for most of the reasoning gains seen in the open.

Properly fixed KL-regularized policy gradients.

Token-level RL rather than only chain-of-thought rollouts.

More efficient credit assignment across steps.

Integration of uncertainty signals and world models directly into the reward function.

NVIDIA Pushes Physical AI at CoRL 2025

At CoRL in Seoul, NVIDIA unveiled a full-stack push into robotics: open models, simulation engines, and new hardware.

• Newton Physics Engine – GPU-accelerated, co-developed with DeepMind and Disney, now open source under the Linux Foundation and integrated in Isaac Lab. Already adopted by ETH Zurich, TUM, and Peking University.

• Isaac GR00T N1.6 – a humanoid foundation model enhanced with Cosmos Reason, giving robots humanlike reasoning and the ability to handle locomotion and manipulation together. Partners include LG, Franka, Neura, and Solomon.

• Cosmos WFMs – updated Predict and Transfer models generate long-horizon, multi-view synthetic data at scale, downloaded over 3M times.

• Dexterous Grasping + Arena – new Isaac Lab workflows for robot hands and standardized skill evaluation; Boston Dynamics’ Atlas has already trained with it.

• Jetson Thor – Blackwell-powered on-robot supercomputer, adopted by Figure AI, Unitree, DeepMind, and Meta.

Nearly half of CoRL papers cited NVIDIA tech. With Newton as the body, GR00T as the brain, and Jetson Thor as the deployment engine, NVIDIA is positioning itself as the operating system for physical AI.

OpenAI puts GDP on the scale

OpenAI just unveiled GDPval, a benchmark for evaluating AI on real-world, economically valuable tasks. It spans 44 occupations across the top 9 U.S. GDP sectors and compares AI deliverables to those from professionals with ~14 years of experience. All credits to them – they don’t shy from showing that the rival model beats their own.

The QMA Singularity by Scott Aaronson

‘World Models,’ an Old Idea in AI, Mount a Comeback by Quanta

NVIDIA: OpenAI, Future of Compute, and the American Dream | BG2 w/ Bill Gurley and Brad Gerstner

Code World Model (CWM)

Researchers from Meta FAIR introduced a 32B parameter decoder-only LLM for code generation and reasoning. Trained on 8T pretrain, 5T mid-train, and 172B post-train tokens, CWM uses Python execution traces and 3M agentic ForagerAgent trajectories to model code semantics and planning. It achieves 65.8% pass@1 on SWE-bench Verified (test-time scaled), 68.6% on LiveCodeBench, 96.6% on Math-500, and 76.0% on AIME 2024. Checkpoints are released post mid-training, SFT, and RL under a noncommercial research license →read the paper (pdf)

Deepseek-v3.2-exp

Researchers from DeepSeek-AI introduced a 685B parameter model using DeepSeek Sparse Attention (DSA) to improve long-context training and inference efficiency while maintaining output quality. Benchmarked against DeepSeek-V3.1-Terminus, it achieves similar scores: MMLU-Pro 85.0, GPQA-Diamond 79.9, AIME 2025 89.3, and SWE Verified 67.8. In Codeforces, V3.2-Exp scores 2121 vs. 2046. Sparse attention improves performance in BrowseComp (+1.6%) and Terminal-bench (+1.0%), showcasing gains in agentic and multilingual tasks →DeepSeek on HF

Qwen3-Omni

Researchers from Qwen released a 30B Mixture-of-Experts model achieving state-of-the-art performance across text, image, audio, and video, with no degradation relative to single-modal baselines. It supports 119 written and 19 spoken languages, 10 spoken outputs, and 40-minute audio inputs. Its Thinker-Talker architecture enables 234 ms first-packet latency, real-time streaming, and cross-modal reasoning. It outperforms closed models like Gemini 2.5 Pro and GPT-4o on 32 audio/audio-visual benchmarks and sets overall SOTA on 22. Models are released under Apache 2.0 →technical report

Hunyuan3d-omni and HunyuanImage-3.0

Researchers from Tencent introduced a 3D-native generative framework that supports fine-grained controllable 3D asset generation from text or image with additional inputs like point clouds, voxels, bounding boxes, and skeletons →read the paper They also introduced HunyuanImage-3.0, the largest open-source image generation MoE model with 80B parameters (13B active per token) and 64 experts. Built on a unified autoregressive multimodal framework, it supports advanced reasoning, sparse prompt elaboration, and photorealistic image generation. SSAE evaluation showed strong alignment with 3,500 key points across 12 categories. Human evaluation (1,000 prompts, 100+ raters) confirmed superior aesthetic and semantic performance compared to closed-source models →their GitHub

Manzano

Researchers from Apple introduced a unified multimodal LLM that combines strong image understanding and text-to-image generation using a hybrid tokenizer architecture. A shared vision encoder outputs continuous embeddings for image understanding and discrete tokens for image generation. Manzano employs a unified autoregressive LLM and a diffusion decoder, trained via a three-stage recipe. It achieves state-of-the-art results among unified models, with minimal task conflict and strong scaling: the 30B variant surpasses prior models on text-rich VQA and competitive generation (GenEval: 1.00, WISE: 0.54) →read the paper

🌟🌟 SimpleFold: Folding Proteins is Simpler than You Think (Apple) – fold proteins with a general-purpose transformer trained by flow matching, avoiding domain-specific blocks. ->read the paper

🌟🌟 Video models are zero-shot learners and reasoners (Google DeepMind) – demonstrate emergent zero-shot perception and manipulation abilities in a generalist video model. ->read the paper

🌟MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction (Meta) – append meta tokens and scale vectors at inference to balance accuracy and cost. ->read the paper

🌟 MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources – release long CoT data and select high-variance samples to stabilize RL. ->read the paper

MOSS-ChatV: Reinforcement Learning with Process Reasoning Reward for Video Temporal Reasoning – align reasoning traces to video dynamics with DTW-based process rewards. ->read the paper

🌟 Limits to black-box amplification in QMA by Scott Aaronson and Freek Witteveen – establish oracle-relative limits showing completeness and soundness cannot be amplified beyond doubly-exponential via black-box procedures. ->read the paper

🌟 Behind RoPE: How Does Causal Mask Encode Positional Information? (KAIST, Microsoft) – reveal how causal masking itself induces position-dependent attention and interacts with RoPE to distort relative patterns. ->read the paper

TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them – model judge scoring probabilistically to reduce score-comparison and transitivity failures without extra training. ->read the paper

🌟 What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoT (Meta) – identify failed-step fraction as a better predictor than length or review and validate structure-aware test-time scaling. ->read the paper

Cross-Attention is Half Explanation in Speech-to-Text Models – quantify that cross-attention aligns only partially with saliency, cautioning against over-interpreting attention maps. ->read the paper

🌟 Thinking Augmented Pre-training (Mictosoft) – augment pre-training text with synthetic thinking trajectories to boost data efficiency and downstream reasoning. ->read the paper

🌟 Soft Tokens, Hard Truths (Meta) – train continuous chain-of-thought via RL to increase CoT diversity while deploying with discrete tokens. ->read the paper

SIM-CoT: Supervised Implicit Chain-of-Thought – stabilize implicit CoT by adding step-level supervision with an auxiliary decoder that is dropped at inference. ->read the paper

🌟 Reinforcement Learning on Pre-Training Data (Tencent) – derive rewards directly from next-segment prediction on pre-training corpora to scale RL without human labels. ->read the paper

MAPO: Mixed Advantage Policy Optimization – adapt advantage weighting to trajectory certainty to fix advantage reversion and mirror effects in GRPO-style training. ->read the paper

🌟 VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models (Alibaba) – curriculum-train by selecting samples with informative reward variance to improve math reasoning. ->read the paper

Advancing Speech Understanding in Speech-Aware Language Models with GRPO – optimize speech QA and translation with GRPO and BLEU-based rewards beyond SFT. ->read the paper

🌟 Thinking While Listening: Simple Test Time Scaling For Audio Classification (Stanford) – add lightweight reasoning and sampling at inference to improve audio classification accuracy. ->read the paper

🌟 CAR-Flow: Condition-Aware Reparameterization Aligns Source and Target for Better Flow Matching (Apple) – shorten probability paths with condition-aware shifts to speed training and improve FID. ->read the paper

🌟 ARE: Scaling Up Agent Environments and Evaluations (Meta) – provide a platform and Gaia2 benchmark to stress-test agents in asynchronous, tool-rich, dynamic tasks. ->read the paper

UserRL: Training Interactive User-Centric Agent via Reinforcement Learning – train agents with simulated users and reward designs that improve multi-turn helpfulness. ->read the paper

BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent – mimic human visual attention, cognition, and action with process-and-outcome rewards for GUI automation. ->read the paper

Tree Search for LLM Agent Reinforcement Learning – expand rollouts via tree-structured sampling and compute step-wise advantages from outcome rewards. ->read the paper

RPG: A Repository Planning Graph for Unified and Scalable Codebase Generation – replace free-form plans with graph blueprints to generate full repositories with tests. ->read the paper

LIMI: Less is More for Agency – show that carefully curated, tiny datasets can elicit strong agentic skills more efficiently than large collections. ->read the paper

🌟 Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say (University of Southern California) – route among heterogeneous experts and fuse hidden states in shared layers for single-pass gains. ->read the paper