Our anonymization pipeline, step by step

Anonymization is often described as a single step, as if raw data goes in one end and safe data comes out the other. In practice it is a sequence, and the order matters. Scope is fixed before anything is read, secrets come out before tokenization, and the seller checks the result before anything is delivered.

This post walks through that sequence from start to finish. It is meant for sellers deciding whether to license their data, and for the AI researchers and data teams who want to know what has been done to the material they receive. A summary of the same standard is on our trust page.

Nothing is touched until scope and exclusions are agreed and annexed to the contract. The seller decides which systems are in, which clients, channels, folders or repositories are out, and whether direct messages are included at all.

Some areas are excluded by default. HR, payroll, medical and legally privileged folders stay out. Direct messages stay out unless the company includes them and has the right to do so. Because these boundaries are set before processing starts, excluded material never enters the pipeline in the first place.

The evaluation is read-only, and the seller's time for it is under an hour. From there, the raw archive is held and processed in-house. No third party touches raw data at any point: not a labelling vendor, not an outside annotation service and not the buyer.

That custody rule is easy to underrate. If raw records pass through other hands for convenience, the protections that follow are weaker no matter how good they are. Keeping the raw archive in one controlled place is also what makes the later deletion meaningful.

Before any names are replaced, material that should not exist in any form is taken out. That includes:

• Passwords, credentials and API keys pasted into chat, tickets, code or documents
• Payment card numbers
• Government identifiers such as passport or national ID numbers
• Faces, logos and signatures in images and PDFs, which are blacked out

Doing this first matters. A credential is not made safe by being tokenized; it needs to be gone. The same is true of a signature on a scanned contract or a face in a team photo. Our posts on secrets in the archive and faces, logos and signatures cover each in more depth.

Next, every person, company, address, email, phone number and account is replaced with a token. The defining rule is consistency. PERSON_014 is the same person in Slack, in email, in Jira and in a pull request review, and stays the same person across every year of history in the archive.

That requires one mapping built across all in-scope systems, not a separate pass per tool. A person might appear as a display name in chat, a full name in an email signature, a username in the code host and an email address in the CRM. All of those need to resolve to the same token, or the connections that make the data valuable fall apart. Our posts on why the same person must get the same token everywhere and consistent pseudonyms across years of history explain why.

The same applies to companies. A customer referred to by its full name, a nickname and its email domain becomes one company token. The structure of the work survives; the identities do not.

For the AI teams receiving the data, this is the step that keeps it useful. A ticket can still be followed to the pull request that closed it and the email that told the customer, because every participant carries the same label throughout. For the seller, it is the step that turns a workspace full of names into a record of how work was done, without saying who did it.

Before delivery, the seller reviews an anonymized sample drawn from their own data. This is the seller's chance to confirm the output meets the agreed standard, and to apply knowledge nobody outside the company has.

The things worth looking for are the ones that are hard to spot from outside: an identifier written in an unusual format, a name embedded in a file name or URL, an excluded client that turns up in a thread about something else, or a nickname that never appeared in any contact record. Imagine an operations lead at a 60-person logistics software company scanning the sample and recognizing an internal shorthand for a long-standing customer. That is exactly what the review is for, and anything raised is fixed before delivery. Our post on reviewing an anonymized sample suggests what to check.

Only after the seller signs off is the anonymized dataset delivered. It goes to the buyer under a licence limited to training and evaluation, which prohibits marketing to, profiling, contacting or re-identifying anyone in the data, and rules out resale for marketing or people search.

The raw archive is then deleted within 30 days of sign-off. That is a contract obligation, not a policy aspiration, and it ends with a deletion certificate the seller keeps. After that, the only copy outside the company is the anonymized one. Our post on what raw data deleted should mean in a contract explains why the wording matters.

Each stage depends on the one before it. Scoping first keeps excluded material out entirely. Removing secrets before tokenizing ensures nothing dangerous survives in disguise. A single cross-tool mapping keeps the data connected. The seller's review catches what only an insider would notice. Deletion closes the loop.

If you would like to see how this pipeline would apply to your company's systems and exclusions, book a 15-minute call and we will walk through it with your scope in mind.