Long-horizon tasks: the hardest thing to find in training data

Most training data captures work in single moments: a question and an answer, a function and its tests, a ticket and its resolution note. The work agents struggle with most does not happen in moments. It unfolds over days or weeks, across people and tools, with plans that change halfway through, and almost none of it is written down anywhere a crawler can reach.

A long-horizon task is one where success depends on many steps taken over an extended period, with early steps shaping what comes later. Length alone is not the point. The difficulty is that the agent has to hold a goal steady while the situation around it changes.

Consider a hypothetical 40-person software company migrating its billing system. The project starts with a planning document and a kickoff thread. Over six weeks it produces dozens of tickets, a few pull requests that are reverted and reworked, a customer email about a double charge, a debate in chat about whether to delay the cutover, a revised plan and finally a release note. No single artifact explains how the migration was done. The sequence does.

Long-horizon tasks share a few traits:

• They span multiple sessions, so context has to be recovered rather than assumed.
• They involve hand-offs between people with different roles and partial information.
• They include dead ends, reversals and decisions made under uncertainty.
• They cross tools: planning in docs, coordination in chat, tracking in tickets, execution in code, communication in email.
• Their outcome is only clear at the end, sometimes weeks after the first step.

Public text tends to capture endpoints. A company publishes the launch announcement, not the six weeks of argument that preceded it. An open-source project shows its commits and perhaps its issue tracker, but the conversations that shaped its priorities often happened in a private chat or on a call.

Even where public records are rich, they are usually one tool deep. A repository's history is valuable, but it does not show the customer complaint that triggered a fix or the internal discussion that decided which fix to ship. Agents trained on single-tool records learn to act within a tool. They do not learn to carry a task across several.

The obvious alternative is to generate long-horizon tasks synthetically. That works for structure: a model can write a plausible project plan and a plausible set of tickets. It struggles with the texture that makes real work hard.

Real projects contain interruptions that have nothing to do with the task, colleagues who misunderstand each other, priorities that shift because of something happening in another team, and information that turns out to be wrong. Synthetic generators tend to produce projects where every message is relevant and every decision is sensible in hindsight. Agents trained on them learn a tidier world than the one they will be deployed into. We compared the two approaches in synthetic data vs. real data for agent training.

The most complete records of long-horizon work sit inside companies, in the systems they used every day. A company that ran on chat, email, a ticket tracker, a code host and a CRM for several years holds many multi-week projects preserved across those tools, each with its beginning, middle and end.

Recovering them depends on three things.

First, connection depth. The same project has to be traceable across tools, through ticket IDs in commit messages, links in chat, references in email subjects and names in documents. Five connected systems can reconstruct a trajectory that five unconnected datasets cannot.

Second, years of history. Long projects need long records. A company with a few months of chat may hold the end of a migration but not its start. Several years of history mean many complete arcs, including ones that ran across quarters.

Third, consistent anonymization. When every person, company, address and account becomes a token, that token has to be the same in every tool and across the whole period. If the engineer who opened the ticket appears as one token in the tracker and another in chat, the thread of the project breaks. We explain why in why the same person must get the same token everywhere.

Once a long-horizon trace is reconstructed, it serves several purposes. It can be used directly as a demonstration of how a team moved a project forward. It can be cut into checkpoints, so an agent is dropped into the middle of a project and asked what should happen next, with the historical record as a reference. It can seed an environment whose initial state is the project as it stood on a given day, as described in what an RL environment actually is.

It can also anchor evaluations. Because the outcome is known, graders can compare an agent's choices against what the team actually did, without requiring an exact match. And because the material comes from private systems, it has never been part of a public web crawl.

Long-horizon material is scarce partly because it has to be sourced deliberately. Labs looking for it should ask for companies with years of history and several connected systems, rather than the largest possible message count. A company that kept chat, tickets and code together for five years is often more useful for this purpose than a much larger one with a single tool.

Martal Group finds those companies through outbound. A lab sends a sourcing brief that specifies sector, headcount, tool stack, years of history, regions, and whether it wants operating or winding-down companies. We build a target list from company and technographic data, our reps call the people who can sign, and datasets are evaluated, anonymized in house and delivered on a rolling basis under a licence that covers training and evaluation only.

If long-horizon trajectories are the gap in your training mix, send us a sourcing brief.