Enterprise AI teams and private evaluation sets

Enterprise AI teams keep getting asked a blunt question: will this agent actually work here? Public benchmarks rarely answer it. A private evaluation set, built from realistic operational data and kept away from every vendor being evaluated, can.

Public benchmarks are useful for tracking general capability. They are much less useful for deciding whether to deploy a particular agent into a particular company's workflows.

The first problem is fit. Most benchmarks test isolated tasks with clean inputs. Enterprise work involves half-finished tickets, conflicting documents, colleagues who answer in a thread three days later, and context spread across five tools. An agent that scores well on a benchmark can still stall on that.

The second problem is exposure. Anything public can end up in training data, intentionally or not. Once a benchmark is widely known, a high score says less about how a model handles new work. Why benchmark contamination makes private data valuable covers this in more detail. For a team choosing between vendors, a contaminated benchmark is not a fair comparison.

The obvious alternative is to evaluate on the company's own records. That can work for some purposes, but it has real drawbacks:

• Privacy and access: pulling internal communications into an evaluation pipeline raises questions from legal, HR and security that can take a long time to settle.
• Vendor exposure: if the evaluation runs inside a vendor's environment, the company's data goes with it, and the test set is no longer private.
• A single point of view: one company's history reflects one set of habits, and an agent tuned to those quirks may look better than it is.
• Contamination over time: if the same internal data is later used for fine-tuning or retrieval, it can no longer serve as a clean test.

None of these is fatal on its own. Together, they explain why evaluations built on internal data often stall before they produce a decision.

A different approach is to build the evaluation from operational data licensed from other companies in the same or similar sectors. The data is real: chat, email, docs, tickets, code, CRM, support and finance workflows from companies that did comparable work. It is anonymized, with every person, company and account replaced by a consistent token across every tool, and it is licensed for training and evaluation.

From that source, an evaluation team can build tasks that resemble its own workload. A support-heavy business might use historical support conversations and the tickets they spawned, asking an agent to resolve each case as the original team did. A finance organization might use approval chains and month-end close threads. An engineering group might use tickets that led to merged pull requests. In each case, the historical outcome gives a reference for grading.

Imagine an insurer's operations team comparing three vendors' agents for handling customer cases. Rather than test them on its own correspondence, it builds a set from licensed support, CRM and finance records at several mid-sized service companies, with each task anchored to a case the original team actually closed. Every vendor's agent runs the same held-out tasks, and the team compares how each one gathered context, who it asked, and whether it reached a resolution consistent with the real one.

Because the source companies are not the enterprise itself, most of the internal privacy questions fall away. Because the data was never public, contamination is far less of a concern. And because it can come from several companies, the evaluation reflects a sector rather than one company's habits. Evaluations that look like real work goes further into task design.

A private evaluation set only stays useful while it stays private. That takes a little discipline.

Run evaluations in an environment you control, and give vendors access to the agent's interface, not to the tasks. Keep a held-out portion that nobody outside the evaluation team ever sees, and reserve it for final decisions. Rotate tasks over time so the set does not become familiar, either to vendors or to your own team.

Version the set as well. When an agent's score changes, you want to know whether the agent changed or the test did. And keep the licence terms in view: the data is for training and evaluation only, and nobody may market to, profile, contact or re-identify anyone in it. That constraint suits evaluation work well, since the point is to measure agents, not to learn about the people in the data.

If you are building an evaluation set this way, start from a description of the work you want to test. That becomes a sourcing brief covering:

• Sector, matched to your own business or to the workflows you plan to automate
• Headcount, so the source companies show handoffs similar to yours
• Tool stack, so the evaluation exercises the systems your agents will actually use
• Years of history, enough to include long-horizon work
• Regions, if language or regulatory context matters
• Operating or winding down, depending on the mix you need

From the brief, Martal Group builds a target list from company and technographic data, its reps call the people who can sign, and the data is evaluated, anonymized and delivered on a rolling basis. Seventeen years of B2B outbound and experience prospecting 24 industries help when the brief names an unusual sector or tool combination. For more on structuring the request, see how to write a good sourcing brief for company data.

To start building yours, send us a sourcing brief.