How AI labs evaluate a new data source

When an AI lab considers a new source of company data, it rarely makes the decision in one step. The source goes through a sequence of checks, each one cheaper to fail than the next, and each designed to answer a different question. Knowing that sequence helps a lab move faster and helps a data supplier show up with the right answers at each stage.

The first filter is not about quality. It is about rights. Before anyone opens a sample, a data acquisition or legal team wants to know whether the lab can use the data and defend that use later.

The questions are direct. Which entity licensed the data, and who signed? If the company is closing, what gave the signer authority over company assets? What does the licence allow, and what does it prohibit? What was excluded, and was that agreed before data moved? Where is the raw copy now?

A source that answers clearly moves on. One that cannot tends to stop here, however good the data looks. Our post on why data provenance is becoming a buying requirement covers what a complete record looks like.

Next comes a structural review, usually from a manifest or a small sample. The lab is trying to learn whether the data has the shape its use case needs before it spends time on content.

This is where connection depth shows up. A lab building agents wants to know whether the same piece of work can be followed across tools: a support ticket that links to an engineering issue, the chat thread where it was discussed, the pull request that fixed it and the email that told the customer. Volume alone does not answer that. A smaller dataset with tight links can be worth more than a larger one made of disconnected exports.

Reviewers typically look at a few things at this stage:

• Systems covered, and whether they overlap in time
• Years of history, and whether activity is steady or concentrated in a few months
• Headcount and roles, which determine how many people and perspectives are present
• Cross-references, such as ticket IDs in commit messages or links between docs and tasks
• Gaps, such as a tool that was replaced midway or a team whose channels were excluded

Our post on what makes a dataset agent-ready goes deeper into these structural properties.

Once the shape looks right, the lab examines anonymization closely. This is partly about the lab's own risk and partly about usefulness, because poor anonymization can damage the data as much as it fails to protect people.

Reviewers check whether each person and company has one consistent token across every tool and every year. They search for residual identifiers in the places that leak most often: email signatures, quoted replies, nicknames, file paths, image text and attachments. They look for secrets that should not be there, such as credentials, API keys and card numbers. They check that excluded categories, such as HR, payroll, medical and privileged material, are really absent.

In our process every person, company, address, email, phone and account becomes a consistent token; credentials, API keys, card numbers and government IDs are removed outright; and faces, logos and signatures in images and PDFs are blacked out. Anonymization happens in-house and the raw archive is deleted within 30 days of sign-off, with a deletion certificate. The details are on our trust page.

With rights, structure and privacy settled, the question becomes whether the data improves something the lab cares about. That depends on the use.

For agent training and post-training, teams look for realistic multi-step work with enough variety in sector, size and tool stack that a model does not simply learn one company's habits. For RL environments, they look for enough depth in one workspace to rebuild it convincingly. For evaluations, they look for well-bounded tasks with outcomes that can be checked, and for material that has never been public, so it is unlikely to be in any model's pretraining data. See why benchmark contamination makes private data valuable for more on that last point.

Fit is usually tested with a pilot: a slice of the data is used in a real experiment, and the result decides whether the source scales.

A single good dataset is useful. A source that can keep delivering against a brief is more useful. The final stage looks at supply.

Can the supplier find more companies like this one? How long does each take from first contact to delivery? Can the brief be adjusted as early results come in? Labs that rely only on inbound offers often find that the answer to all three is uncertain.

This is the problem Martal Group is set up to address. A lab sends a sourcing brief covering sector, headcount, tool stack, years of history, regions and operating or winding-down status. We build a target list from company and technographic data, and our onshore reps, drawing on 17 years of B2B outbound across 24 industries, call the people who can sign. Data is then evaluated, anonymized and delivered on a rolling basis, so a lab can run its stage-four checks on early companies while the campaign continues.

Across all five stages, most delays in evaluating a new source come from questions asked late: a rights issue discovered after a pilot, or an anonymization problem found after delivery. Asking the stage-one and stage-three questions early, and expecting a supplier to answer them in writing, saves the most time.

If you are evaluating new sources of company data, send us a sourcing brief and we will show you how our process maps to each of these stages.