SAPO: Step-Aligned Policy Optimization for Reasoning-Based Generative Recommendation
Zaiyi Zheng, Guanghui Min, Yaochen Zhu, Liang Wu, Liangjie Hong, Chen Chen, Jundong Li
Abstract
Generative recommendation treats next-item prediction as autoregressive item-identifier generation. Specifically, items are encoded as semantic identifiers (SIDs), which are short coarse-to-fine token sequences whose early tokens capture broad semantics and later tokens refine them. Recent work augments this paradigm with reasoning traces and optimizes them via reinforcement learning with verifiable rewards, typically outcome-reward algorithm with exact-match feedback on the generated SID. However, in large-catalog recommendation, exact-match feedback on the generated SID only reports whether the final item is correct; when a generated SID mismatches, outcome-reward cannot identify which SID-token prediction caused the mismatch and may penalize matched SID-token positions together with the mismatched position. We identify that the natural unit of credit assignment in this setting is a single reasoning step (one thinking block paired with one SID token). We instantiate this idea in SAPO (Step-Aligned Policy Optimization): rather than broadcasting one advantage to the whole response, SAPO computes a separate group-relative advantage for each reasoning step and applies it only to the corresponding thinking block and SID token. Across three real-world recommendation datasets, SAPO stabilizes reinforcement-learning training and consistently improves over existing generative recommendation baselines, with the largest gains where sparse exact-match feedback makes reasoning-step credit assignment important. Our results suggest that reinforcement-learning objectives for structured generation should mirror the decoder's own decomposition of the output.
AI Impact Assessments
(1 models)Scientific Impact Assessment: SAPO — Step-Aligned Policy Optimization for Reasoning-Based Generative Recommendation
1. Core Contribution
SAPO addresses a specific credit assignment problem in reinforcement learning for generative recommendation systems that decode items as hierarchical semantic identifiers (SIDs). The key insight is that when using outcome-level rewards (e.g., exact-match on the full SID tuple), the RL objective cannot distinguish which SID-token position caused a mismatch. A near-miss prediction that matches 2 of 3 SID levels receives the same penalty as a completely wrong prediction. SAPO decomposes the RL credit assignment to the level of individual "reasoning steps" — each pairing a thinking block with its corresponding SID token — and computes separate group-relative advantages per step.
The contribution is cleanly defined: per-step match rewards derived from SID-token correctness, step-level group-relative advantages, and step-normalized token aggregation in the policy gradient surrogate. The method introduces no learned reward model, leveraging only the verifiable structure already present in the SID hierarchy.
2. Methodological Rigor
Strengths in methodology:
Concerns:
3. Potential Impact
Direct impact on generative recommendation: SAPO provides a practical recipe for anyone training reasoning-augmented generative recommenders with hierarchical identifiers. The method is lightweight (Table 7 shows negligible computational overhead), requires no additional learned components, and the code is released.
Broader implications for structured generation RL: The paper's concluding insight — that "reinforcement-learning objectives for structured generation should mirror the decoder's own decomposition of the output" — has implications beyond recommendation. Tool-call schemas, structured answer formats, multi-step code generation, and hierarchical planning all share the property that outputs have a natural decomposition finer than the full sequence but coarser than individual tokens. This principle could influence RL training for code generation (function-by-function credit), multi-hop QA (step-by-step verification), or any domain with verifiable intermediate outputs.
Practical limitations on impact: The method is specifically designed for the SID hierarchy paradigm. Its applicability depends on the adoption of this particular generative recommendation formulation, which, while growing, is not yet dominant in production systems. The Amazon Reviews datasets, while standard, are relatively small by industry standards.
4. Timeliness & Relevance
The paper is well-timed. The convergence of LLM-based reasoning (DeepSeek-R1, Qwen3) with generative recommendation (OneRec, TIGER, SIDReasoner) creates a natural need for RL methods that respect structured output formats. The problem of reward sparsity and credit assignment in GRPO-style training is well-recognized, and SAPO offers a concrete solution in a domain where verifiable intermediate signals are naturally available. The connection to process reward models (Lightman et al., 2023) without requiring learned reward models is particularly appealing.
5. Strengths & Limitations
Key strengths:
Notable weaknesses:
Overall Assessment
SAPO makes a focused, well-motivated contribution at the intersection of structured generation and reinforcement learning for recommendation. The core idea — aligning credit assignment granularity with the hierarchical structure of the output — is sound and potentially generalizable. However, the empirical validation is limited in scale, and the improvements, while consistent, are modest in absolute terms. The paper would benefit from larger-scale experiments and statistical testing. It represents solid incremental work with a potentially influential design principle.
Generated May 19, 2026
Comparison History (21)
Paper 1 introduces a novel methodological advancement (SAPO) addressing a fundamental problem in reinforcement learning—credit assignment—within the rapidly growing field of generative recommendation. Its algorithmic insights into step-aligned optimization for structured reasoning tasks offer broader scientific implications and potential applications across other generative domains. In contrast, Paper 2 provides a valuable but domain-specific benchmark for evaluating LLMs on spreadsheet tasks, which, while practically useful, offers less fundamental algorithmic innovation.
Paper 2 likely has higher impact: it introduces a new, broadly usable multi-page, multilingual benchmark plus a comprehensive evaluation protocol, which can standardize progress across document understanding, OCR, layout analysis, and multimodal LLM pipelines. Benchmarks often drive wide adoption, enable reproducible comparison, and catalyze follow-on work. Paper 1 proposes a meaningful RL credit-assignment refinement for a specific generative recommendation setup, but its scope is narrower (structured SID generation with reasoning traces) and impact depends more on adoption within recommender RLHF-style training. Both are timely; Paper 2’s cross-field applicability is larger.
Paper 2 offers a more specific, technically novel algorithmic contribution (step-level credit assignment for RL in structured generative recommendation) with clear empirical validation on multiple real-world datasets and a broadly reusable insight for RLHF/structured generation beyond recommendation. Its applications map directly to high-impact industry recommender systems, and the methodological framing is rigorous and timely given current interest in verifiable-reward RL for reasoning. Paper 1 is valuable infrastructure, but evidence is limited to internal case studies and impact may depend on adoption rather than a generalizable scientific mechanism.
SAPO introduces a novel and generalizable methodological contribution—step-aligned credit assignment for structured generation in RL—that extends beyond recommendation to any autoregressive structured prediction task. This broader applicability across RL, NLP, and recommender systems, combined with principled theoretical motivation and empirical validation on multiple datasets, gives it higher potential impact. MPDocBench-Parse, while valuable, is primarily a benchmark contribution for a specific subfield (document parsing), which typically has narrower and more incremental impact compared to new optimization methods.
SimGym introduces a novel framework combining VLMs with live browser simulation to replace costly A/B tests in e-commerce—a broadly applicable problem. Its cross-disciplinary impact (HCI, ML, e-commerce) and practical value (reducing weeks-long experiments to under an hour with 77% directional alignment) give it wider real-world applicability. SAPO, while technically sound, addresses a narrower problem (credit assignment in generative recommendation RL), representing an incremental improvement within a specific subfield. SimGym's paradigm of simulated experimentation with grounded VLM agents has broader transformative potential.
Paper 2 addresses a fundamental challenge in reinforcement learning for reasoning-based generation (credit assignment for sparse rewards). By introducing Step-Aligned Policy Optimization (SAPO), it offers a core algorithmic advancement that aligns with the highly impactful trend of reasoning traces (similar to process vs. outcome supervision). While Paper 1 provides a valuable interdisciplinary application of LLMs to survey research, Paper 2's methodological innovation in RL optimization has a higher potential to influence the foundational development of generative AI and recommender systems across multiple domains.
SAPO addresses a concrete, well-defined technical problem (credit assignment in RL for structured generation) with a generalizable solution validated across multiple datasets. Its insight—that RL objectives should mirror the decoder's output decomposition—has broad applicability beyond recommendation to any structured generation task. Paper 1 introduces GEA, a useful validity criterion for LLM-based assessment, but its scope is narrower (educational assessment), the empirical results are preliminary (single study, modest correlations), and the proposed mitigations (better rubrics) are incremental rather than transformative.
Paper 1 addresses a fundamental architectural problem for safe LLM agent deployment — a topic of immense and growing importance as LLM agents are increasingly deployed in real-world settings. Its contribution of a principled three-layer probabilistic framework with compositional safety guarantees has broad applicability across all LLM agent systems, not just one domain. It identifies open research problems that could catalyze an entire research agenda. Paper 2, while methodologically sound, addresses a more narrow problem (credit assignment in generative recommendation), with impact limited primarily to the recommendation systems community.
CAM-Bench likely has higher scientific impact because it provides a broadly reusable community resource (1,000 Lean 4 targets) that enables standardized, mechanically verifiable evaluation of LLM mathematical reasoning in undercovered applied/computational domains. Its dependency-recovery/normalization pipeline and released artifacts can catalyze follow-on work in formalization, benchmarking, curriculum learning, and tool development across ML, PL, and mathematics. SAPO is a solid methodological improvement for RL credit assignment in generative recommendation, with clear practical value, but its impact is narrower to a specific task/setup and may be more incremental.
Paper 1 addresses a fundamental and broadly applicable bottleneck in LLM agent systems—action space representation—which affects the entire growing field of LLM agents. Its framework (LAR) introduces a novel conceptual contribution (latent action reparameterization) that is complementary to existing optimizations and applicable across diverse agent benchmarks. Paper 2 makes a solid but narrower contribution, improving credit assignment for generative recommendation via step-aligned advantages. While technically sound, its impact is confined to the recommendation domain. Paper 1's broader applicability, novelty in reframing agent efficiency as an action representation problem, and relevance to scaling LLM agents give it higher potential impact.
Paper 2 likely has higher scientific impact because it introduces a broad, timely benchmark/harness for omni-modal, closed-loop tool-using agents—a central emerging paradigm with wide applicability across AI research and industry. Its executable tasks, grounded evaluators, and verification loop can become a standard for measuring progress, enabling reproducibility and catalyzing model/tooling advances across many subfields (agentic LMs, multimodal reasoning, evaluation, HCI). Paper 1 is a solid, novel RL credit-assignment improvement but is narrower in scope (generative recommendation with SIDs) and may impact a more specialized community.
Paper 1 introduces a novel, inspectable framework (Belief Engine) for understanding and controlling belief dynamics in multi-agent LLM deliberation—a fundamental challenge as LLM agents are increasingly deployed in high-stakes settings like negotiation and conflict resolution. It offers broader cross-disciplinary impact (AI safety, social simulation, cognitive science, political science) and addresses the critical need for transparency in LLM behavior. Paper 2 makes a solid but narrower technical contribution (step-level credit assignment for generative recommendation), improving an existing paradigm incrementally. Paper 1's configurable infrastructure has wider applicability and timeliness given growing concerns about LLM agent interpretability.
Paper 1 offers a methodological innovation in reinforcement learning for structured generation by improving credit assignment at the reasoning-step level. This insight has broad applicability to other areas of LLM reasoning and RLHF beyond recommendation systems. In contrast, while Paper 2 provides a valuable and rigorous benchmark, its impact is largely confined to the specific application domain of telecommunications.
Paper 2 (SAPO) likely has higher scientific impact due to broader applicability and timeliness: step-wise credit assignment for RL on structured generation directly targets a common limitation of outcome-reward methods and can generalize beyond recommendation to other token/step-decomposed generative tasks. It is evaluated on multiple real-world datasets and addresses practical large-catalog settings, increasing real-world adoption potential. Paper 1 is novel in integrating retrieved domain evidence into degradation-model selection, but its impact is more specialized to prognostics/stochastic degradation modeling and depends on curated evidence banks and domain-specific rules.
Paper 2 addresses a fundamental challenge in reinforcement learning for generative models (credit assignment in reasoning traces). Given the current rapid advancement and massive interest in LLMs and reasoning-based generation, this methodological innovation is highly timely and likely to influence broader structured generation tasks. While Paper 1 presents a valuable clinical application, it relies on standard machine learning techniques, making Paper 2 more methodologically novel with a higher potential for widespread algorithmic impact.
Paper 2 addresses a critical and timely issue of trust and reliability in LLM-inferred user states. Its proposed psychometric validation framework has broad implications across HCI, adaptive systems, and responsible AI, impacting how researchers and practitioners evaluate AI metrics. While Paper 1 offers a strong algorithmic improvement for generative recommendation systems, Paper 2's foundational methodological contribution and focus on AI reliability give it broader cross-disciplinary scientific impact and real-world relevance.
Paper 2 addresses a critical bottleneck in reinforcement learning for reasoning-based generation: step-level credit assignment. Its approach to aligning RL objectives with structured decoding steps has broad implications for advancing LLM reasoning and generative recommendation systems. In contrast, while Paper 1 presents a solid methodological improvement for NLP using hypergraphs, its application to automatic personality prediction is relatively niche and the broader impact across fields is more limited compared to advancements in LLM reasoning.
Paper 1 addresses a critical real-world problem (predicting challenging behaviors in children with profound autism) with a novel application of foundation models for wearable sensor data in naturalistic settings. It bridges ML and special education with direct safety implications. While Paper 2 makes a solid technical contribution to generative recommendation via step-level credit assignment, it represents an incremental improvement within a narrower ML subfield. Paper 1's interdisciplinary nature, humanitarian impact potential, and pioneering real-world deployment context give it broader and more significant scientific impact.
Paper 2 introduces a novel methodological advancement (SAPO) addressing a fundamental credit assignment problem in reinforcement learning for structured generation. While Paper 1 offers valuable empirical insights into LLM contamination and neuro-symbolic methods in the legal domain, Paper 2's approach to step-aligned policy optimization has broader implications for improving reasoning-based generative models across various domains, offering higher potential impact in the rapidly advancing field of RL-driven generative AI.
Paper 2 (SAPO) likely has higher scientific impact due to a more generally applicable methodological contribution: step-level credit assignment for RL fine-tuning in structured generative outputs, applicable beyond recommendation (e.g., tool-use, program synthesis, structured decoding). It addresses a fundamental limitation of outcome-only rewards under sparse exact-match feedback and shows consistent gains across multiple real-world datasets. Paper 1 (STAR) is valuable and practical for AIOps/RCA reliability, but is more domain-specific and system-engineering oriented, potentially narrowing breadth of impact despite strong real-world relevance.