Journal club week 43, 2026.

This week covers a new attribution method for multispectral images and two evaluation studies. One examines interactive video generation over extended sessions; the other checks whether payment agents obey policies while completing long sequences of tool calls.

The two evaluation papers ask a similar question: what happens after the first output, when new instructions arrive and earlier choices still constrain the system? Both evaluate behavior across a session, including state that should remain unchanged.

Spectral-Spatial Interpretation

Haotian Ma, Ruqi Yang & Philip Townsend, arXiv, 2026. arXiv:2610.21114

Image saliency maps usually sum over channels, while channel-attribution methods usually sum over space. Both can hide evidence that is local in both dimensions, such as a short-wave infrared band that matters inside a burn scar but not elsewhere. Spectral-Spatial Interpretation (SSI) assigns a score to every grid-cell and channel pair. For each cell, it leaves the rest of the image unchanged, masks subsets of that cell's channels with a baseline, and computes Shapley values for the resulting small game on the target-class logit. RGB inputs with three channels can be enumerated exactly; the 13-band EuroSAT and six-band burn-scar experiments use permutation sampling within each cell.

The construction gives a useful identity: summing SSI over a cell's channels exactly equals the logit change from replacing that whole cell with the baseline. This is a local accounting result, not a guarantee that all cells sum to the total prediction change. It also distinguishes SSI from Owen values, which average over many possible contexts outside the cell. The authors evaluate against Integrated Gradients, DeepSHAP, KernelSHAP, and a rank-1 RISE control on CUB-200, EuroSAT, and HLS Burn Scar. SSI has the best joint deletion and insertion AUCs under both mean and zero masking baselines. On HLS with mean masking, its joint deletion AUC is 0.1545, compared with 0.2160 for Integrated Gradients and 0.2888 for RISE.

On the same dataset, SSI beats a Monte Carlo Owen estimator on the retained metrics while using 4,480 forward passes per image versus 76,800, about 5.8 percent of the evaluation budget. Those counts are analytical and do not make forward passes directly comparable with backward passes from gradient methods. The rank-1 diagnostic is also informative: compressing an attribution matrix into one spatial map times one global channel profile loses more on EuroSAT than on CUB-200, consistent with the greater channel diversity. The main limitations are the usual sensitivity of perturbation explanations to the masking baseline and the cost of computing local games across a fine grid or hundreds of spectral bands.

Relevance

I find the full attribution matrix useful because it makes a testable claim about where a particular band supplies evidence, instead of inferring that from two separately aggregated explanations. For remote sensing, that connects model explanations to physical quantities such as red-edge reflectance or burn severity. The exact cell-level aggregation rule also gives researchers a clean consistency check when comparing attribution methods.

StreamEval: Evaluating Interactive Streaming Video Generation Models

StreamEval Team, arXiv, 2026. arXiv:2610.28402

StreamEval evaluates video systems during scripted interactive sessions rather than from isolated clips. Its 300 cases cover avatar dialogue, prescribed avatar performance, world generation, instruction-based editing, and reference-based editing, with 60 cases per task. Sessions last 15 to 300 seconds and contain 1,347 scheduled inputs in total, including speech, text, controls, and replacement reference images. The scoring checks whether requested changes happen on time, whether earlier changes persist, and whether protected content remains intact. Ten metrics cover execution, consistency, visual quality, identity or reference matching, and audiovisual synchronization. The study evaluates 37 models in 47 task configurations, with three runs per case and equal weight for each duration tier.

The results separate rendering quality from control. In avatar performance, Vidu S2 Avatar scores 86.12 for identity and 97.25 for stability, but only 36.90 for following requested actions, still the best score among seven systems. In text-controlled world generation, Krea Realtime has the highest following score at 25.93. HappyOyster 1.0 Adventure reaches 58.34 for visual quality but scores 5.56 for following and 2.78 for continuity. Reference editing is harder still: Vidu S2 Editing leads three reported systems with an Edit Success score of 15.00. Its example sessions show models retaining the source cape after a new reference requests a different one.

Most content judgments come from multimodal models whose structured observations are converted to scores by fixed rules. The authors calibrate judges with positive and negative probes, and unresolved judgments are reported through their lower scoring endpoint. Two scope details matter when reading the comparisons: the editing tables show only configurations with Overall scores of at least 5, selected after evaluation, and reference editing has just three reported systems. The benchmark therefore gives a broad session protocol, but the model coverage is uneven across tasks.

Relevance

This evaluation can distinguish a model that ignores a new control from one that attempts it but corrupts unrelated content or loses an earlier edit. Those differences are hard to see in clip-level quality scores. I also like the explicit deadlines and restoration checks, since they give streaming memory and control a concrete test protocol that other researchers can repeat.

BizBench: Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business

Enhan Zhao et al., arXiv, 2026. arXiv:2610.20424

This benchmark reconstructs payment workflows in an executable environment with 50 multi-turn tasks, 789 policy and reference documents, and 601 discoverable capabilities behind nine general tools. Tasks combine customer dialogue with shared balances, deadlines, approvals, and commitments across workflows. Documents include superseded rules and apparent conflicts caused by product scope or effective dates, so the agent must decide which instruction applies before acting. Deterministic checks separately score requested outcomes, requirements during execution, and protected objects or side effects. They inspect records and event histories without requiring a match to one reference trajectory. Every task has a replayed passing reference execution, and the evaluators were tested with alternative valid routes and targeted violations.

Claude Opus 5 has the highest strict success rate at 64.0 percent, followed by GPT-6 Sol at 61.3 percent, although those two models have one run per task while most others have three. GPT-6 Sol completes all three attempts on only 40.0 percent of tasks. Qwen3.8-27B attains 42.0 percent goal success but 32.7 percent strict success, showing the gap introduced by process and protection checks. Five DeepSeek-V4.1-Flash runs reach their goals but fail the full criteria; four dip below a required cash reserve before later restoring it. Qwen3.5-9B completes none of its 150 runs and averages 101.6 tool-error responses per run.

Long traces do not reliably indicate progress. DeepSeek-V4.1-Flash averages 191.3 assistant tool calls against 111.8 in the reference executions, with most of the excess spent on database queries and document access. Giving it the applicable documents raises strict success from 43.3 to 50.0 percent, while disclosing all customer-known facts at the start raises it to 70.0 percent. Retrospective caps are also revealing: limiting runs to twice the reference call count reduces strict success to 26.7 percent. These experiments point to customer elicitation and repeated information gathering as substantial sources of difficulty beyond policy retrieval alone.

Relevance

Final-state evaluation can mark an agent successful after an unauthorized transfer or a temporary reserve breach, even though the event history contains the violation. This work shows how to score those histories directly while leaving agents free to choose different valid strategies. I would use the same separation of goal, process, and protection checks in other tool-use domains where an early action cannot be undone by a tidy final state.

Final notes

The SSI result I would most like to see extended is the full-grid evaluation on sensors with many correlated bands. The authors suggest screening or grouping channels, and that step will likely determine whether the method remains affordable outside the current six-band and 13-band experiments.

For the two benchmarks, the detailed protocols are as informative as the headline scores. StreamEval would benefit from human agreement studies for its automated judgments, and the payment study would be stronger with more tasks and repeated runs for every frontier model. Both already provide concrete execution traces that make low scores easier to investigate.

All posts