Grading agents: rewards from real outcomes

Training an agent with reinforcement learning needs a way to say whether it did well. For short, well-defined tasks, that can be a unit test or an exact answer. For the kind of work agents are now being asked to do, such as resolving a customer issue, shipping a feature or closing out a month-end task, the right answer is rarely that clean.

Company data offers something useful here. It records not only what people did, but what happened afterward. Those recorded outcomes can become grounding for rewards and evaluations that look like real work.

Most grading approaches for open-ended agent tasks fall into a few families. Hand-written rubrics describe what a good answer should contain. Model-based judges compare an output to a reference or score it against criteria. Human raters review outputs directly.

All three are useful, and all three share a weakness: they grade against someone's idea of a good outcome rather than an outcome that actually occurred. A rubric might reward a support reply for being polite and thorough, when what mattered in practice was whether the customer's problem went away. A judge might prefer a well-explained code change that would have broken production.

Real outcomes do not replace these methods. They give them an anchor. If you know what actually happened after a human handled a task, you have a second signal to check your graders against. For background on how tasks become environments, see what an RL environment actually is.

Operational systems are full of outcome fields, because the people using them needed to know whether things worked. A few examples:

• A pull request was merged, then left alone, or reverted a week later
• A ticket was closed as done, closed as won't-fix, or reopened
• Tests passed or failed on the change that addressed a bug
• A support conversation ended with the customer confirming the fix, or escalated
• A deal moved to closed-won, closed-lost, or stalled in a stage for months
• An invoice was paid on time, chased, or disputed
• An incident was resolved, and a follow-up ticket did or did not appear

None of these were created for AI training. They exist because a team tracked its own work, which is part of what makes them credible as signals. They were recorded at the time, by people with a stake in getting them right.

The basic pattern is to replay a real task to the point where a decision was made, let the agent act, and compare what it did to what happened next. If the original engineer's fix was reverted, an agent that proposes the same fix should not get full credit. If the support rep's second reply resolved the issue, an agent that gets there in one reply has done something worth noticing.

In practice, the outcome rarely becomes a reward on its own. It works better as one input among several:

• As a filter, keeping only tasks where the recorded outcome is clear enough to grade
• As a reference, giving a model-based judge the real resolution to compare against
• As a check on rubrics, flagging cases where a rubric rewards behavior that failed in reality
• As a held-out signal, testing whether a grader's scores track real outcomes at all

Long-horizon tasks benefit most. When a task spans days or weeks across several tools, it is very hard to write a rubric that captures success. It is much easier to look at whether the project shipped and stayed shipped. We wrote about the structure of those trajectories in from ticket to pull request.

Outcomes are honest, but they are not always fair. A few cautions are worth building in from the start.

The human path is not the only correct one. An agent may solve a problem differently from the original team and still be right. Grading strictly against what the human did will penalize valid alternatives, so outcome signals should reward results, not imitation.

Outcomes are also noisy. A deal can be lost for reasons that had nothing to do with the email the agent is being graded on. A ticket can be closed because a project was cancelled, not because the work was done. A merged pull request can look successful for a month before the bug it introduced shows up.

Timing matters too. The further an outcome sits from the action, the more other events intervene. Short-range outcomes, such as tests passing or a customer replying, tend to be cleaner than long-range ones like renewal.

None of this argues against using outcomes. It argues for treating them as evidence, combining them with other graders, and checking them against later events where the data allows.

The biggest practical limit on outcome-based grading is visibility. The outcome of a task often sits in a different system from the task itself. The request arrives by email, the work happens in a ticket and a pull request, and the result shows up as a quiet support queue or a renewed contract in the CRM.

That is why connection depth matters so much. When the same project can be traced across several tools, the outcome is visible. Consistent tokenization keeps that trace intact through anonymization: the customer who raised the issue is the same token in the email, the ticket and the CRM, so the outcome can be tied back to the task that produced it.

We source company data with that depth in mind, from companies across sectors, tool stacks and regions you specify. Every dataset is licensed for training and evaluation only. If your team is building graders or environments around real outcomes, send us a sourcing brief.