Journal club week 28, 2026.

Three papers this week, on three unrelated problems. The first is a reinforcement-learning paper that argues the field has been optimizing the wrong objective when it fine-tunes LLMs for reasoning. The second is a real-time interactive video generation system you steer with your voice while it plays. The third is a 4D world model that predicts how a scene moves in 3D and uses that prediction to control a two-armed robot.

They span post-training, generative video, and embodied control, so I'll take them one at a time.

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

Jing Liang et al., arXiv, 2026. arXiv:2606.29526

Reinforcement learning has become the standard way to push reasoning into instruction-tuned models, but the training runs are notoriously touchy — they stall or collapse for reasons that are hard to pin down. One culprit these authors focus on is what they call training-inference mismatch. Modern RL pipelines split the work between two engines: an inference engine like vLLM or SGLang samples the rollouts, while a training engine like FSDP or Megatron computes the gradients. Even when both hold identical weights, differences in numerical precision, decoding, and serving code make them assign slightly different probabilities to the very same token sequence. That gap turns every update into a mild off-policy update, and it never goes away.

The paper's main argument is that everyone has been optimizing the wrong thing. Prior fixes — reweighting by the training-to-sampler ratio (TIS), filtering unstable tokens (MIS), decaying the learning rate, or switching to higher-precision rollouts — all try to stabilize the training policy. But the policy you actually deploy is the inference policy, and improving one does not guarantee the other improves too. They propose optimizing inference-policy improvement directly, a principle they call Monotonic Inference Policy Improvement, and a two-step method, MIPU. Step 1 builds a candidate update anchored on the sampler rather than the old training policy, using a truncated importance-sampling correction so a large pre-existing mismatch doesn't dominate the gradient. Step 2 synchronizes that candidate to the inference engine, runs a small validation rollout, and estimates a post-update inference gap; if that gap is clearly negative, it rolls both engines back to the previous checkpoint rather than accepting a gain that only exists on the training side.

They test under FP8-quantized rollout, which deliberately inflates the mismatch, on Qwen3-4B and Qwen3-1.7B across five math benchmarks. MIPU reaches an average pass@1 of 66.71 on the 4B model (against 64.42 for plain GRPO and 65.66 for learning-rate decay) and 53.97 on the 1.7B model, and unlike the baselines it holds a stable trajectory instead of peaking and collapsing. The ablation is the interesting part: Step 1 alone lifts the 4B average to 65.36 but still drifts, Step 2 alone prevents collapse but has no good candidates to accept (62.81), and only the combination does both. A control that rolls back updates at random — even more often than Step 2 does — still collapses, which points to the acceptance signal, not just the act of rejecting updates, as what stabilizes training.

Relevance

The reframing is the part I'll remember. Once you accept that the sampling engine and the training engine are two different policies, it follows that a training-side improvement is only a proxy for what you actually care about, and under aggressive quantization that proxy can point the wrong way. Treating the deployed inference policy as the real optimization target — and validating an update on that side before committing to it — is a clean idea that composes with the existing mismatch-correction tricks rather than replacing them.

Vidu S1: A Real-Time Interactive Video Generation Model

Jintao Zhang et al., arXiv, 2026. arXiv:2607.03118

Vidu S1 is a system for generating video you can steer while it plays. Instead of writing a prompt and waiting for a clip, you drive a digital character in real time with voice instructions, changing what happens at any moment. The authors report 540p output running as fast as 42 FPS on ordinary consumer GPUs, and playback of unbounded length without the blurring, drift, or distortion that usually creep in as generated video accumulates. It is built on two components the authors call TurboDiffusion and TurboServe, which together handle the generation and serving side of the real-time budget.

You can supply your own image — a real person, an anime character, a pet — and pick a voice, and the character is animated to match. The authors report that Vidu S1 comes out ahead across their evaluation metrics while staying inside the latency budget that real-time interaction demands, and they provide a live demo to try it directly.

Relevance

Most video generation is still a batch affair: submit, wait, review. A model that stays interactive — responding to voice mid-stream and holding quality over arbitrarily long runs on hardware people actually own — moves generated video toward something closer to a controllable avatar than a rendered clip. Whether the quality holds up outside the demo is the open question, but the real-time-on-consumer-GPU target is the part worth watching.

RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation

Haoyu Zhao et al., arXiv, 2026. arXiv:2607.06559

Robots that manipulate objects need to predict not just what a scene looks like next, but how its 3D structure will move when they touch it. World models built on 2D video capture appearance but throw away geometry, so they struggle with 6-DoF pose and depth-aware contact, and they tend to produce artifacts like objects that subtly change scale or morph over time. RynnWorld-4D's premise is that you can recover most of the geometry cheaply by generating three synchronized channels — RGB, depth, and optical flow (they call it RGB-DF) — instead of RGB alone. Depth lifts each pixel into 3D, and depth plus optical flow back-project into a per-point 3D scene flow, so geometry and motion become explicit while the representation stays image-shaped and can still ride on pretrained video-diffusion priors.

The model extends Wan 2.2 (a 30-layer, 5B diffusion transformer) into three branches, one per modality, tied together by Joint Cross-Modal Attention modules inserted every three layers (ten in total) with frame-wise 3D RoPE so the branches stay aligned pixel-for-pixel within each frame. From a single RGB-D image and a language instruction it co-generates the future RGB, depth, and flow in one denoising loop. Training uses Rynn4DDataset 1.0, a curated mix of over 254 million frames from egocentric human video and robot-manipulation datasets, auto-labeled with Qwen3-VL captions, Depth Anything 3 depth, and DPFlow optical flow, and proceeds in three phases: adapt each branch independently, train the joint-attention modules with the backbone frozen, then fine-tune everything.

For control, RynnWorld-4D-Policy treats the frozen world model as a predictive 4D encoder: it reads out internal features in a single forward pass — skipping the multi-step denoising you'd need to actually render the video — compresses them with a Flow Former, and feeds a small flow-matching head that outputs 10-step action chunks. On an RTX 5090 with FP8 the world-model pass is the bottleneck at 990 ms, but action chunking yields an effective 9 Hz closed-loop rate. On generation quality it roughly doubles the depth-accuracy score of prior 4D models (0.610 versus 0.327 for 4DNeX and 0.279 for TesserAct) and is the only one to report explicit flow (0.170 AEPE). On six real bimanual tasks it beats Diffusion Policy and the pi-series foundation policies, most clearly on precision- and coordination-heavy ones: 65.71% on lid placement and bowl stacking, and 28.57% on a hand-over that the foundation policies essentially fail. Ablations show every piece earns its place — dropping the large-scale pretraining alone sends flow error from 0.170 to 0.729.

Relevance

What I like here is the pragmatism of the representation. Full 3D or 4D-Gaussian world models are geometrically honest but expensive and hard to scale; plain video is scalable but geometry-blind. Predicting depth and flow alongside RGB is a middle path that keeps the video-diffusion backbone and its data efficiency while making the 3D motion explicit enough to drive an inverse-dynamics policy. The single-forward-pass feature readout is the practical trick that makes it usable for closed-loop control at all, since running the full denoiser for every action step would be far too slow.

Final notes

If there's a light thread this week, it's that a lot of the progress comes from being precise about what you're actually optimizing or generating — the inference policy rather than the training policy, a 4D scene rather than a flat video — more than from raw scale. The video and robotics papers also share a fixation on real-time budgets on hardware you can buy, which is a good sign for how much of this ends up usable outside a datacenter.

None of it is a finished story: the RL result is still on moderate model sizes, the video work lives behind a demo, and 9 Hz control is fine for tabletop tasks but not for anything fast. Still, all three left me with an idea I hadn't had before, which is about the most I ask of a week's reading.

All posts