Ingénierie d’infrastructure API et de systèmes back-end

Français

Sécurité

Tokenisation versus encryption: shrinking the blast radius of sensitive data

Most data breaches are not cryptography failures. They happen because sensitive values – national identity numbers, account numbers, health data – are copied into too many systems, logs and exports. Encryption protects data where it is stored; tokenisation removes it from places that never needed it.

Publié le 6 min de lecture

Ce contenu est actuellement disponible uniquement en anglais.

Two different problems

Encryption answers the question “how do we protect this value where it is stored or transmitted?”. Every system that needs the plaintext must hold, or be able to use, the key. Tokenisation answers a different question: “which systems actually need the real value?”. It replaces the value with a token everywhere except in a small, well-protected vault, and only the few services that truly need plaintext may detokenise.

The practical difference is scope. If a reporting database stores encrypted identity numbers and the reporting service holds the key, a compromise of that service exposes the data. If it stores tokens, the same compromise exposes tokens that are useless outside the vault. In short: encryption limits who can read stored data, tokenisation limits where the data is stored at all.

Token design

Tokens come in several forms, and the choice affects what downstream systems can do with them:

  • Random tokens: generated randomly and stored in the vault alongside the encrypted value. They reveal nothing and are the default choice.
  • Deterministic tokens: the same value always produces the same token, which allows joins and deduplication across systems without detokenising. They also allow frequency analysis, so use them only where joins are required and the value space is large.
  • Format-preserving tokens: tokens that look like the original – same length and character set – so that legacy schemas and validations keep working. They can be produced from a vault or by format-preserving encryption.

Tokens may also keep a non-sensitive fragment visible, such as the last four digits of an account number, for customer support. Every visible fragment reduces protection, so decide this explicitly per data class.

Encryption done properly: envelope keys and rotation

The vault itself must encrypt what it stores, and many systems still need field-level encryption for data that cannot be tokenised. The standard design is envelope encryption: each record or batch is encrypted with a data-encryption key (DEK) using an authenticated cipher such as AES-GCM; the DEK is itself encrypted with a key-encryption key (KEK) held in a key-management service or hardware security module. Applications never see the KEK; they ask the KMS to unwrap DEKs and cache them briefly.

Rotation then becomes manageable. Rotating the KEK means re-wrapping DEKs, not re-encrypting all data. Rotating DEKs can happen gradually as records are rewritten. NIST SP 800-57 describes key lifecycles and cryptoperiods; whatever periods you choose, automate them and record every key operation.

Format-preserving encryption

NIST SP 800-38G specifies the FF1 mode for format-preserving encryption: encrypting a string of digits or characters into another string of the same length and alphabet. It lets you protect fields in legacy systems whose schemas cannot change. FPE is deterministic for a given key and tweak, so it shares the frequency-analysis risk of deterministic tokens, and small domains – short numeric fields – need care because the number of possible values may be too small for strong protection. Use a tweak, such as a field identifier, to prevent identical values in different fields from producing identical ciphertexts.

Designing the vault service

A token vault is a small, critical service. Its API typically offers tokenise, detokenise and sometimes search for deterministic lookups. Each operation is authorised per calling workload and per data class: most services may tokenise, very few may detokenise, and detokenisation is logged with the caller, purpose and request ID.

Because many systems depend on it, the vault must be highly available and fast. Run it across zones, keep its database encrypted with envelope keys, cache nothing sensitive outside it, and rate-limit detokenisation so that a compromised caller cannot bulk-export the vault. Backups of the vault are as sensitive as the vault itself and must be protected accordingly.

Regulatory context

The GDPR names pseudonymisation and encryption as examples of appropriate technical measures in Article 32. Tokenised data remains personal data as long as someone can reverse the mapping, but it significantly reduces risk and can limit breach-notification obligations when the vault itself is not affected. Card-payment environments follow PCI DSS, where tokenisation is a common way to remove systems from the cardholder-data scope. In both cases, the legal assessment belongs to your compliance team; the engineering task is to make the data flow match the assessment.

Migrating an existing estate

Few teams can tokenise everything at once. A practical migration starts with discovery: scan databases, logs, message topics and file exports for patterns that match the sensitive fields, and compare the findings with what the architecture documents claim. The gap is usually larger than expected.

Next, stop the spread. Tokenise at the point of entry – the API or form that receives the value – so that new data reaches downstream systems only as tokens, and add log scrubbing so that no new plaintext appears in logs. Only then backfill: replace historical values system by system, starting with the least necessary copies such as analytics extracts and test databases, which often hold production data they should never have received.

During the transition, some consumers will still expect plaintext. Give them a time-limited detokenisation permission with explicit owners and a removal date, rather than leaving a plaintext copy in place “for now”.

Measure progress with simple numbers: how many systems and how many rows still hold plaintext per data class, and how many detokenisation calls happen per day and by whom. Those numbers are also what auditors and incident responders ask for first.

Exemple: Removing identity numbers from analytics

A platform stored personal identity numbers in its transactional database, its analytics warehouse, daily CSV exports and application logs. A data-flow review showed that only the identity-verification and reporting-to-authorities services needed plaintext. The team introduced a vault issuing random tokens, with deterministic tokens for the warehouse so analysts could still count unique customers. Log scrubbing replaced values at the logging library. Detokenisation was granted to two workloads, rate-limited and logged. Eleven systems stopped holding the plaintext; a later compromise of a reporting dashboard exposed only tokens.

Pièges courants

  • Encrypting fields but giving every service the key.
  • Using deterministic tokens everywhere, enabling frequency analysis.
  • Forgetting logs, exports, caches and backups when tokenising.
  • Unlimited, unaudited detokenisation for convenience.
  • Applying format-preserving encryption to very small value domains.

Questions et réponses

Is tokenisation stronger than encryption?

They solve different problems. Tokenisation reduces where sensitive data exists; encryption protects it where it must exist. A good design uses both: tokens outside the vault, envelope encryption inside it.

Does tokenised data fall outside the GDPR?

Generally not while the mapping can be reversed. It is pseudonymised data, which still counts as personal data but carries lower risk.

Should we build or buy a vault?

Many teams use a managed vault or a secrets platform with tokenisation features. Building one is feasible, but the hard parts are availability, key management and auditing, not the token format.

Liste de contrôle

  • Map every system, log and export that holds each sensitive field.
  • Decide per field: tokenise, encrypt, or both.
  • Choose random tokens by default; deterministic only where joins require it.
  • Implement envelope encryption with KEKs in a KMS or HSM.
  • Restrict and log detokenisation per workload and purpose.
  • Scrub sensitive values at the logging layer.
  • Rehearse key rotation and vault recovery.

Sources primaires

  • NIST SP 800-38G – Recommendation for Block Cipher Modes of Operation: Methods for Format-Preserving Encryption
  • NIST SP 800-57 Part 1 Rev. 5 – Recommendation for Key Management
  • Regulation (EU) 2016/679 (GDPR), Articles 4(5) and 32
  • PCI Security Standards Council – PCI DSS v4.0

Tous les articles