Why benchmark contamination makes private data valuable

A benchmark is only useful if the model being tested has not seen the answers. As training corpora grow to cover more of the public web, that condition gets harder to guarantee for any test that lives in public. Private operational data, which has never been published, offers a way out, but only if labs treat it with the care they wish had been applied to public benchmarks.

Contamination is the overlap between what a model was trained on and what it is tested on. It rarely happens on purpose. A benchmark is published with its questions and answers. Those questions get discussed in forums, quoted in blog posts, copied into tutorials and mirrored across repositories. Eventually some of that text ends up in a web crawl, and the crawl ends up in a training set.

Filtering helps but does not solve it. Exact-match filters catch verbatim copies and miss paraphrases, translations and partial quotes. Coding benchmarks are especially exposed, because solutions to well-known problems are posted widely and differ only slightly from each other.

For a question-answering benchmark, contamination inflates a score. For agent evaluations, the damage can be harder to see. Agent tasks are often built from public sources: issues from open-source repositories, tickets from public trackers, scenarios adapted from published case studies. If the model has seen the original issue and the merged fix, it can appear to reason its way to a solution that it is partly recalling.

The result is a measurement problem that compounds. Teams compare checkpoints using contaminated tests, pick the one that scores higher, and may be selecting for memorization rather than skill. The error surfaces later, when the agent meets work that was never published and performs worse than the numbers suggested.

Operational records from inside a company, such as chat, email, tickets, code, CRM and support history, were never meant to be public. They are not in web crawls, not quoted in forums and not mirrored in open repositories. An evaluation built from them starts clean.

They are also realistic in a way public benchmarks often are not. A task drawn from a company's actual ticket history comes with the context an employee would have had: the chat thread where the problem was first raised, the customer email that escalated it, the pull request that fixed it. The agent has to work across tools, and the historical outcome gives graders a reference for what a good result looked like. We go further into this in evaluations that look like real work.

Anonymization keeps that structure intact. Every person, company, address, email, phone and account becomes a consistent token across every tool, so PERSON_014 in chat is still PERSON_014 in the tracker, and the task remains coherent without exposing anyone. Credentials, API keys, card numbers and government IDs are removed outright.

Private data does not stay uncontaminated by default. It stays clean because of how it is handled after it arrives. A few practices matter.

• Hold out whole companies, not random rows. Records from the same company share people, projects, vocabulary and history. If some of a company's tickets go into training and others into evaluation, the model has effectively seen part of the test. Splitting at the company level avoids that.
• Decide the split before anyone trains. Evaluation sets should be fixed and recorded before training runs begin, so later decisions cannot leak test material back into the training mix.
• Limit access. The fewer pipelines and people that can read the evaluation set, the lower the chance it gets copied somewhere it should not be.
• Refresh over time. Even well-guarded sets age. Adding new companies periodically keeps the evaluation representative and gives a check on whether older sets have drifted into training by accident.
• Track provenance. Knowing where each record came from, and under what licence, makes it possible to audit a split later.

Licence terms support this. The licences Martal Group arranges permit training and evaluation, so a lab can decide at the split which companies serve which purpose. They also prohibit marketing to, profiling, contacting or re-identifying anyone in the data, and they prohibit resale for marketing or people search.

Evaluation sets need variety more than volume. A test drawn from one company measures how well an agent handles that company. A test drawn from many companies, across sectors, sizes and tool stacks, measures something closer to general skill. That makes sourcing a design decision, not just a procurement step.

It also changes what "good data" means. For evaluation, a mid-sized company with chat, tickets, code and email connected over several years can be more useful than a much larger archive from a single tool, because connected records produce tasks with context and known outcomes. Connection depth and years of history matter more than raw size.

Martal Group sources companies through outbound. A lab sends a brief describing the sector, headcount, tool stack, years of history, regions, and whether it wants operating or winding-down companies. We build a target list from company and technographic data, our reps reach the people who can sign, and each dataset is evaluated, anonymized in house and delivered on a rolling basis. Because sellers are recruited to fit the brief rather than drawn from whoever volunteered, the mix can be shaped to what the evaluation needs.

If you are building held-out evaluations and need company histories that have never been on the public web, send us a sourcing brief.