Scaling RL training in the age of multi-agent systems

This is more of an engineering paper from what I can tell.

Recall that from REINFORCE, the gradient of the loss is like

We can split this loss up into per-token loss terms log pi_theta (y_t | x, y_<t). Hence, in normal RLVR, you’re supposed to have a loss term per token outputted by the agent (i.e. the 1’s here)

But during inference time, it’s possible that your agent will do things (e.g. compaction, subagents, transient messages) that doesn’t involve a monotonically increasing prefix of messages, so that applying a loss mask directly doesn’t work. Their solution to this is episode-level tracking:

“For each task, we keep track of a list of episodes which are append-only sequences of messages; these episodes are first class platform primitives that together can be thought of as a single trace. These episodes preserve the full expressibility of all possible harnesses — harnesses which require forking or modifications of previous messages simply need to start new episodes for each instance of forking or message mutation.”

and wow now your infra is perfectly set up for RL.