Synthetic data vs. real data for agent training

Synthetic data has become a standard part of training pipelines, and for good reason: it is cheap, controllable and available in any quantity. But agents meant to work inside real companies keep running into the same limit. A generator can only produce the kind of mess it was told to produce.

Synthetic data earns its place. You can generate thousands of variations of a task, control the difficulty, balance categories that are rare in the wild and label everything perfectly, because you wrote the ground truth yourself. For narrow skills, such as filling a form, parsing an invoice format or calling a fixed API, it is often the fastest path.

It also sidesteps sourcing. There is no counterparty to find, no contract to negotiate and no anonymization step. When a team needs a large volume of examples quickly, or needs examples of a scenario that real data rarely contains, generating them is a reasonable choice.

The trouble starts when the target is not a skill but a job. An agent that triages tickets, coordinates a release or helps close the books at month-end has to operate in an environment shaped by many people, many tools and a lot of history. A simulated company tends to be coherent in a way real companies are not.

Generated workplaces are usually too tidy. Tickets have clear titles, the people in a thread agree on what the problem is, and decisions are recorded where you would expect them. Real organizations leave decisions in a meeting that shows up only as "per our call, going with option B", rename projects halfway through and track the same work in two tools because one team never migrated.

There is also a subtler issue. When one model generates both the task and its surrounding context, the context tends to carry that model's assumptions. An agent trained on the output can learn to exploit the generator's habits rather than the underlying work, and evaluations built the same way can reward it for doing so.

Real company data contains properties that are hard to specify in a prompt, because nobody thinks to ask for them:

• Long horizons, where a single project runs for months across hundreds of messages, tickets and commits
• Ambiguity that is resolved slowly, with wrong turns, reversals and partial information
• Many systems referring to the same work, linked by ticket keys, URLs and the same people appearing in each
• Organizational change: new hires, departures, reorganizations and tools adopted or abandoned
• Real outcomes, such as whether a deal closed, a bug reopened or a deadline slipped

That last property matters a great deal for reinforcement learning. A reward is only as good as the signal it is built on, and real outcomes come from the world rather than from whoever wrote the scenario. We describe how this fits into environment design in what an RL environment actually is.

This is not a case for abandoning synthetic data. Strong pipelines tend to use each for what it does best. Real data provides the distribution: the shape of actual work, the tool combinations companies really use and the failure modes that occur in practice. Synthetic methods then extend it by generating variations around real trajectories, filling gaps in rare scenarios or building tightly controlled test cases.

Real data is especially valuable on the evaluation side. A held-out set of anonymized, real workflows checks whether an agent can handle the world, not just the generator's version of it. Because private company data has never been published, it is also much less likely to have leaked into pretraining corpora.

A useful test for any synthetic set is to ask what a real version would contain that this one does not. If the answer includes several of the properties above, the synthetic set is probably teaching a simplified job. The gap tends to widen as tasks get longer and involve more systems, which is exactly where agents are being pushed.

Not every real dataset closes the gap. A single exported chat channel or one code repository on its own has some of the texture but little of the structure. The data that helps most comes from several systems at the same company, covering several years, with the same people and projects traceable across all of them. Connection depth matters more than raw volume, as we explain in six systems that make a dataset valuable.

It also has to be anonymized in a way that keeps that structure. Every person, company, email and account becomes a consistent token across every tool, so PERSON_014 in chat is still PERSON_014 in the tracker and the email thread. Secrets and government IDs are removed outright, and use is limited to training and evaluation under the licence.

The usual objection to real workplace data is availability. It does not appear on the open web, and the companies that hold it rarely think of it as something they could license. Waiting for it to show up is slow, and what does show up skews toward whoever happened to hear about the opportunity.

Our approach is to go and get it. Labs send a sourcing brief describing the sector, headcount, tool stack, years of history, regions and whether they want operating or winding-down companies. Martal Group builds a target list from company and technographic data, and our reps, drawing on 17 years of B2B outbound, call the people who can sign. The data is then evaluated, anonymized and delivered on a rolling basis.

The result is real operational history matched to the gaps in your mix, rather than whatever arrived first. If you want to see how that would work for your team, send us a sourcing brief.