Anonymization 101: tokenization vs. redaction

Every anonymization method has to answer one question: when you remove a name, what do you put in its place? Redaction puts nothing there, just a black bar or a deleted string. Tokenization puts in a consistent stand-in, so the record still makes sense after the person behind it has disappeared.

Both methods have a job to do. The mistake is using one where the other belongs, and for data that will be used to train AI agents, that mistake usually shows up as a dataset that is either unsafe or useless.

Redaction removes information outright. A phone number becomes a blank, a signature on a scanned contract becomes a solid block, and an API key pasted into a chat message is deleted. Nothing about the original survives, and nothing can be inferred from the gap except that something used to be there.

That is exactly what you want for some categories of data. There is no version of a password or a card number that is useful to keep in a training set, and no stand-in that makes the surrounding text more instructive. The safest amount of a secret to leave behind is none of it.

The cost of redaction is that it breaks references. If every name in a Slack thread is replaced with the same black bar, a reader can no longer tell who asked the question, who answered it, and whether the person who closed the ticket three days later was the same one who opened it. The text is safe, but the story is gone.

Tokenization replaces each real value with a label that stays the same everywhere it appears. A named engineer becomes PERSON_014. Her employer's largest customer becomes COMPANY_203. A street address becomes ADDRESS_051, and an account number becomes ACCOUNT_009.

The key property is consistency. PERSON_014 is the same person in the Slack channel, in the email thread with the customer and in the Jira ticket that tracks the fix. If she is mentioned by first name in one message and by full name and job title in another, both mentions resolve to the same token. The real identity is gone, but the relationships survive.

This is harder than it sounds. A person may appear as a display name, a handle, an email address, a signature line and a nickname, and every variant has to map to one token across every tool and every year of history. Doing it within a single document is easy. Doing it across a whole company is the actual work.

An AI agent learning workplace tasks is not learning facts about specific people. It is learning patterns: how a request turns into a ticket, who gets looped in when a deadline slips, how a customer complaint travels from support to engineering and back. Those patterns live in the relationships between messages, not in the names themselves.

Imagine a 40-person software company where a support lead flags a recurring bug, an engineer picks it up, a product manager pushes back on the priority, and the fix ships a week later. With tokenization, a model can follow that sequence end to end, because the same four tokens show up in the support tool, the chat, the tracker and the code review. With blanket redaction, it sees four disconnected fragments with nobody in them.

That is why tokenization is the default for identities in agent training data. It keeps the structure that makes the data worth licensing while removing the part that would let anyone point at a real person. We wrote more about why that structure matters in why agents need workplace data, not more web text.

Tokenization is for things that need to stay referenceable. Some things do not, and for those, removal is the right call. In our pipeline, the following are removed or blacked out rather than tokenized:

• Credentials, passwords, API keys and access tokens, wherever they appear
• Payment card numbers
• Government identifiers such as social security or passport numbers
• Faces, logos and signatures in images and PDFs

There is also a third option that comes before either method: not including the data at all. HR, payroll, medical and legally privileged folders are excluded by default, and direct messages are left out unless the company chooses to include them and has the right to do so. Exclusion is the bluntest tool, and for the most sensitive material it is the correct one.

In practice, a single document often gets both treatments. Picture an email from a vendor's account manager that includes her name, her phone number, a customer's billing address and, in a forwarded attachment, a scanned contract with a signature. The name, phone number, address and companies become consistent tokens. The signature is blacked out. If the thread had included a card number for a deposit, that would be removed entirely.

The result reads like the original email with the identifying details swapped for labels. It is still clearly a renewal negotiation, still traceable to the CRM record and the follow-up meeting, but no longer about anyone a reader could find. That balance is what makes the data both safe to license and useful to the people training on it.

A few other safeguards round this out. No third party touches the raw data, and the company reviews an anonymized sample before anything is delivered, so it can check that both methods were applied the way it expects. The raw archive is then deleted within 30 days of sign-off, with a deletion certificate. The full pipeline is described on our trust page.

Redaction is for things that should never be seen again. Tokenization is for identities that need to disappear while the work around them stays legible. A dataset that only redacts tends to be safe and thin, and a dataset that only tokenizes risks leaving secrets in place. Good anonymization uses each method where it fits.

If you want to see how your own company's data would look after anonymization, book a 15-minute call.