Finetuning with Sampling: SFT Learns Better Than You Think

SFT with more on-policy data

Updated with Aayush Karan’s invited talk / Q&A on 10/13

SFT causes catastrophic forgetting and generalization issues due to being off-policy. But RL struggles with capabilities that the base model didn’t have to begin with (e.g. 0/8 rollouts solve it)

Prior work motivation

https://arxiv.org/pdf/2510.14901 in a previous work, he shows that if you sample entire sequences (using the same metropolis hastings technique as this paper will use) to try to sample sequences from a power-law sharpened distribution instead of from the original output distribution, you basically get GRPO-level performance training-free.

This motivates using sampling as an algorithmic primitive

  • This is different from low temperature/greedy sampling because it’s sharpening the distribution on sequences, not on tokens

Theoretical Motivation

Their method’s idea is to try to transform SFT data to be more on-distribution:

Suppose that $x \in \mathcal{C}$ is the set of all rollouts where the answer is correct, and $p(x)$ is the base model’s distribution over rollouts. The closest distribution (in KL) $q$ to $p$ which has support only on $C$ is \(p_{\mathcal{C}}(x) \propto p(x) \cdot \mathbf{1}\{ x \in \mathcal{C} \}\)We’d like to get our SFT data to look like it was drawn from $p_{\mathcal{C}}(x)$.

Of course, you can get this distribution by just sampling from $p$ and rejecting until you get $x \in \mathcal{C}$, but what if it’s very low probability? It’s only making use of the privileged correct answer information as a verifier, when we would like to use it to generate the data. To do this, we can use Metropolis Hastings

For metropolis hastings, you need your target stationary distribution up to normalization constant, and a proposal distribution $\kappa(x \mid x’)$ for the transition probability between states.

  • this is the acceptance probability that you multiply by when moving from $x’$ to $x$
    • recall Metropolis-hastings makes it so that the probability flow from $x \to x’$ is and $x’ \to x$ are both the same, and equal to the min under the proposal distribution
  • you can get rid of the \(\mathbf{1}\{ x \in \mathcal{C} \}\) if you make sure that your transitions are only on $\mathcal{C}$.

Empirics and Results

In practice, the method looks like

  1. Take off a suffix of the current generation, use the base model prompted with the correct answer as the proposer
  • this is an approximation to actual conditioning on $x \in \mathcal{C}$
  1. Accept the new sample if it has higher likelihood under the base model than the current completion
  • This is an approximation to the metropolis-hastings acceptance condition. They do this just for speed reasons: $\kappa(x \mid x’)$ and $\kappa(x’ \mid x)$ require two LLM forward passes in the Metropolis-Hastings, and they’d rather just spend one

For data where the answer is not verifiable, they use LLM judge consistency with the original answer from the offline SFT ground truth.

Compare to on-policy self-distillation: https://siyan-zhao.github.io/blog/2026/opsd/: they also use the model prompted with the correct answer as a teacher model, but instead of changing the training data (which is still generated by the model being trained), they just edit the loss to be a distillation KL loss

Qwen2.5-7B, Olmo-3-7B, Chemistry, MMLU, AMC

  • Seems like it does result in more on-policy SFT data
  • Note that likelihood is just a proxy for on-policyness, the exact measure is the KL distance between the model policies, which they kept track of and made sure was also decreasing as they sampled

  • Less forgetting, higher accuracy.
  • they have Table 1 showing that it beats GRPO too

  • Through this SFT, it genuinely learns new capabilities that it got 0% success rate on before
  • Doesn’t suffer from mode collapse like RL causes

Miscellaneous Q&A answers

Computational inefficiency: it’s epoch-agnostic, so takes about as long as a 32-epoch RL run if you need that many epochs. Not very efficient.

Resampling as weights change: Similar to RL, you probably want to regenerate the data to be on policy again after doing some SFT weight changes. However, if the SFT dataset is small enough, like the ones used here, you don’t need to do this, just for computational efficiency