Evaluations that look like real work

An agent can post a strong benchmark score and still stumble on its first week of real work. The gap is rarely about raw capability. It is about the shape of the tasks: most evaluations are cleaner, shorter and better specified than anything a working team hands over. Closing that gap starts with building evaluations out of real work.

A typical benchmark task arrives with a precise instruction, all the context the model needs, and a single correct answer. That design makes grading easy and results comparable. It also removes most of what makes workplace tasks difficult.

In a company, the request is often one line in a busy channel. The relevant context is spread across a ticket, a two-month-old email thread and a doc nobody has updated. Someone already tried a fix that did not work. The person who knows the answer is on vacation. A good employee handles all of that without being told to. An agent tested only on clean tasks has never been asked to.

The result is a measurement that rewards the wrong thing. Teams end up optimizing for tasks that look nothing like the deployment they are preparing for.

When we look at workplace archives with evaluation in mind, the useful tasks share a set of features. A realistic evaluation task usually has:

• An underspecified request. The ask is phrased the way people actually phrase it, with assumptions left unstated.
• Scattered context. The information needed to succeed sits in more than one tool, and some of it is out of date.
• Prior attempts. Earlier work, including failed work, is visible and has to be taken into account rather than repeated.
• Other people. There are colleagues to consult, approvals to get, and customers waiting on a reply.
• Constraints. Budgets, deadlines, permissions or policies limit what a reasonable answer looks like.
• A known ending. The archive shows what the team actually did and how it turned out, which gives graders something concrete to compare against.

Not every task needs all six. But a suite in which none of them appear is measuring something narrower than the job.

The basic method is to freeze the archive at a moment in time. Pick a point just before a piece of work began: a ticket was filed, a customer emailed, a quarter closed. Give the agent everything a person on that team could have seen at that moment, and nothing after it. Then ask it to do the work.

Because the archive continues past that point, you already know what happened next. The team shipped a particular fix, sent a particular reply, booked a particular adjustment. That outcome is not always the only good answer, but it is a real one, produced under real constraints. It anchors grading in something other than a rubric written after the fact. The same idea underpins training environments, covered in what an RL environment actually is.

This only works if the archive is connected. If chat, tickets, code and email come from different companies or different time periods, the frozen snapshot has holes where the context should be. The agent fails for reasons that have nothing to do with its ability, and the evaluation stops telling you anything useful.

Two properties keep a workplace evaluation trustworthy over time.

The first is privacy of the source. Data that has never been published is far less likely to have leaked into anyone's training set. That matters more each year as public benchmarks get absorbed into training corpora.

The second is anonymization that preserves meaning. Every person, company, address, email, phone number and account is replaced with a consistent token, so PERSON_014 is the same person in Slack, in email and in Jira. Remove those links and an evaluation task built around "follow up with the account owner" becomes impossible to solve. Keep them, and the task stays intact while nobody in it can be identified. Credentials, keys, card numbers and government IDs are removed outright, and sensitive folders such as HR, payroll and medical are excluded by default.

Licensed data comes with terms that fit evaluation work: use for training and evaluation only, and no attempt to market to, profile, contact or re-identify anyone in the data. Many teams also hold a slice of each source back from training entirely so it can serve as a clean test set.

If you are building an evaluation suite from company data, the brief looks a little different from a training brief. A few things to specify:

• Coverage across sectors and team sizes, so the suite does not reflect one company's habits
• Tool stacks that match the environments your agent will be deployed into
• Enough years of history that tasks have real backstory and visible outcomes
• A mix of operating and winding-down companies, if your use case allows both
• Functions beyond engineering, such as finance, support and sales, where agents are increasingly expected to work

For practical advice on structuring those requirements, see how to write a good sourcing brief.

Real work is recorded inside companies, and those companies have to be found and asked. We take a lab's sourcing brief, build a target list from company and technographic data, and our onshore sales reps call the people who can sign. Data is then evaluated, anonymized in-house and delivered on a rolling basis, so an evaluation suite can grow as new sources come in.

If you are building evaluations that need to look like the job, send us a sourcing brief.