Kurate.org
Rankings
Back to Rankings

Projected Distillation: On Learning to Solve Hard Problems from Close, Correct Targets

Shripad V. Deshmukh, Christian Bock, Kangrui Ruan, Abhijeet Awasthi, Vijay Lingam, Rein Houthooft, Manish Prajapat

Oct 20, 2026arXiv:2610.27467v1
cs.LGcs.AI
Share
Scorecard· 16/16
6.0/10 impact

Solid, well-controlled study of a timely problem with honest reporting, but the method recombines known ideas (SPOT/SODA-style corrections, DPO) and gains over close baselines are modest.

Abstract

On-policy learning is central to post-training: training on a model's own generations limits distribution shift and helps preserve existing capabilities. On hard problems, however, on-policy RL provides no reward-based learning signal when all attempts fail. On-policy distillation provides next-token supervision from a teacher, but on hard problems, if a rollout goes wrong early, subsequent supervision is conditioned on prefixes that already contain the mistake. Correct behavior can instead come from a gold or teacher solution, but such off-policy targets can differ substantially from the model's generations and, as we show, may erode existing capabilities. We therefore treat on-policyness as a continuum and propose projected distillation, which minimally edits the student's failed attempts into correct targets. The method uses contrastive updates to favor corrections over failed attempts and refreshes these corrections as the student evolves, enabling near-on-policy learning even when all student attempts fail. Theoretically, we prove that the student's learning regret grows with the edit size: it scales linearly with the horizon under minimal correction, as in DAgger, and quadratically under full-answer replacement, as in behavior cloning. Empirically, across two students (Qwen3-8B and OLMo-2-7B) and five domains spanning math, code, science, and tool use, projected distillation gives the highest area under the specialization-retention frontier (best on 8 of 9 cells against GRPO and OPD, and 7 of 9 against SPOT and SODA). Our ablations highlight the importance of online corrections, the contrastive loss, and on-policyness: moving targets farther off-policy steadily worsens the specialization-retention trade-off, with retention falling from 0.916 to 0.744 when minimal corrections are replaced by student-agnostic gold solutions.

AI Impact Assessments

(1 model)

Core Contribution

The paper proposes "projected distillation" for post-training small LLMs on hard (pass@k=0) problems. A black-box projector minimally edits the student's own failed rollouts into verified-correct targets. The student is then trained with a diff-localized DPO contrast between the corrected and failed spans, plus self-SFT on its own correct rollouts. Corrections are recomputed every round against the current student.

The framing is that on-policyness is a continuum. Targets from RL, OPD, minimal edits, free rewrites and gold solutions sit at different distances from the student, and that distance governs the specialization–retention trade-off. The paper adds a regret bound in a "patch MDP" in which drift scales with edit size. It scales as O(H) for minimal edits and O(H²) for full rewrites, mirroring DAgger versus behavior cloning.

The ingredients are not new: minimal-edit corrections (SPOT, LeMa), preference learning on corrections (SODA, SuperCorrect), DPO and SFT. The contribution is mainly the careful isolation of three factors: edit size, online refresh, and objective. It also adds the diff-local contrast and a clean evaluation framing (AUPF).

Methodological Rigor

Strengths:

  • Controlled ablations hold correctness fixed and vary one factor at a time. These cover edit size (RQ2), static versus online pools (RQ3), six loss recipes (RQ4), projector refinement (RQ5) and self-projection (RQ6).
  • Baselines include GRPO, OPD, SPOT and SODA, with matched budgets and a shared projector for the black-box comparison.
  • Results use 3 seeds. The paper separates training-seed from evaluation-seed variance and honestly reports losses (MATH, SciKnowEval, OLMo GSM8K).
  • The appendix is unusually transparent about caveats: the single-seed GRPO cell, the omitted OLMo LiveCodeBench cell, and the heavy-tailed Λ.
  • An independent replication on the setup of Karan et al. (Appendix I) is a useful external check, including a compute comparison.

    • Weaknesses:

  • Hard sets are small. GSM8K has only 122 problems and the MATH hard-set transfer experiment uses 127 held-out problems. Several RQs are run only on the GSM8K hard set with Qwen3-8B.
  • Retention is measured in-domain on held-out slices, not as general capability. Appendix D.3 partly addresses this, but with a single checkpoint.
  • The AUPF metric is custom, anchored at the base, and sensitive to trajectory shape. Specialization levels are low in many cells, such as MATH at 0.03.
  • The Table 2 transfer protocol selects the best checkpoint per column on the evaluation set itself, a single seed is used, and the gains are small in absolute terms (0.011→0.020 pass@1).
  • The edit-size ablation confounds edit size with the length of the rewrite and with likelihood shift. Gold is an outlier that does not fit the ordering. The projector is a strong proprietary model (Claude Opus), which complicates a fair comparison with white-box OPD.
  • Theory: the bound relies on a dense per-token reward and an idealized optimal projector. It also relies on a Lipschitz prefix-sensitivity assumption, and it does not cover the deployed contrastive loss. The authors concede all of this in Remarks 2 and 5. It is a bound on horizon dependence for the target, not a guarantee about the algorithm. The "linear versus quadratic" claim is therefore more illustrative than predictive. The edit-size definition in the theory (m) differs from the empirical token diff.

    • Potential Impact

    The practical message is useful: when RL gives no signal, correct the student's own attempts minimally and refresh them as the student changes. It is cheap, needs only text from a black-box teacher, and preserves existing capabilities. The method requires a verifier and single-turn tasks. Industrial relevance (the authors are at AWS) is real for specializing small models on hard internal tasks. The ablation evidence on the importance of refresh and minimality should inform follow-up work in the SPOT/SODA lineage and in self-distillation with privileged information. The retention-versus-specialization framing and the AUPF metric may be reused, though adoption is uncertain.

    Timeliness

    The paper addresses hard-problem learning when RL has zero reward signal, a live bottleneck. It connects to OPD, self-distillation (OPSD), and the question of whether RL expands capability. The topic is hot and crowded, with many concurrent works. The incremental delta over SPOT and SODA is the main risk to long-term impact.

    Strengths and Limitations

    Strengths:

  • Clean controlled study that isolates why the recipe works.
  • Honest reporting of mixed results.
  • Broad coverage: two model families, five domains, six research questions.
  • Independent replication check with compute accounting.
  • A theory that, though idealized, is transparent about its scope.

    • Limitations:

  • Modest absolute specialization gains in several cells.
  • Wins over SPOT/SODA are 7 of 9 and are frequently within a few points.
  • Small hard sets and small models (7–8B) only.
  • Dependence on a very strong proprietary projector. The mid-capacity 32B projector is somewhat worse, and self-projection recovers only about 40% of specialization.
  • Verifiable-reward tasks only.
  • A theory-practice gap.
  • Workshop-level paper status.

    • Overall this is a solid, carefully analyzed incremental contribution. It is likely to be cited within the post-training and on-policy distillation subcommunity, but it is unlikely to reshape the field.

    Rating:6/ 10
    Significance 5.5Rigor 6.5Novelty 5Clarity 7

    Generated Oct 21, 2026

    Comparison History (0)

    No comparisons yet.