Mutual TLS and workload identity for service-to-service traffic
Inside most platforms, services still trust each other because they share a network. That assumption fails the moment one container is compromised. Mutual TLS gives every workload a cryptographic identity and lets every connection prove who is on both ends – provided certificates are issued, rotated and checked with discipline.
Ce contenu est actuellement disponible uniquement en anglais.
Why network location is not identity
Firewall rules and private subnets answer the question “where did this packet come from?”. In a dynamic platform, addresses are reassigned constantly, sidecars and jobs share nodes, and an attacker who compromises one workload inherits its network position. Treating “inside the cluster” as trusted means that a single breach reaches every internal API.
Zero-trust architectures replace location with identity: every request is authenticated and authorised on its own merits, regardless of where it originates. For service-to-service traffic, mutual TLS is the most widely deployed way to do this, because it combines authentication with encryption in a protocol every runtime already supports.
How mutual TLS works
In ordinary TLS, only the server presents a certificate and the client verifies it. In mutual TLS, the server also requests a certificate from the client, and both sides verify the other against a trusted certificate authority (CA). After the handshake, each side knows the other’s identity as expressed in the certificate, and all traffic is encrypted.
TLS 1.3 (RFC 8446) shortens the handshake and removes obsolete cipher suites, so the performance cost of mTLS on modern hardware is small, especially with connection reuse. The real cost is operational: someone must issue certificates to every workload, keep them short-lived and make sure verification is configured correctly everywhere.
Workload identity and SPIFFE
A certificate is only useful if its contents mean something stable. Host names and IP addresses do not identify a service in an orchestrated environment. SPIFFE (Secure Production Identity Framework For Everyone) defines identities as URIs such as spiffe://example.internal/ns/payments/sa/ledger, placed in the certificate’s subject alternative name. The identity names the workload – namespace and service account, for example – independently of where it runs.
An issuing component attests each workload before giving it a certificate: it checks with the orchestrator which pod is asking, which service account it runs as and which image it uses. Only then does it issue a short-lived certificate for the matching identity. SPIRE is the reference implementation; service meshes such as Istio and Linkerd issue SPIFFE-compatible identities to their sidecars or proxies.
Issuance and rotation
Long-lived certificates turn key theft into a long-lived breach and turn renewal into an outage waiting to happen. Workload certificates should live for hours, not years, and be rotated automatically before expiry. That requires:
- An intermediate CA dedicated to workloads, with its root key kept offline or in a hardware security module.
- An agent or sidecar on each node that requests, renews and delivers certificates to workloads without restarts.
- Trust bundles distributed to every verifier, with a plan for rotating the CA itself.
- Monitoring for certificates that are close to expiry and for workloads that fail to renew.
With rotation fully automated, revocation lists become less important: a stolen certificate expires quickly on its own.
From authentication to authorisation
mTLS tells the ledger service that the caller is spiffe://example.internal/ns/payments/sa/checkout. It does not say whether checkout may call POST /entries. That decision belongs in an authorisation policy: an allow-list of which identities may call which routes and methods, enforced in the service, its sidecar or a policy engine such as Open Policy Agent.
Start with deny-by-default between namespaces and explicit allows for known call paths, derived from observed traffic. Combine workload identity with end-user context where needed – for example a signed token that carries the user on whose behalf the call is made – so that a compromised service cannot act for arbitrary users.
Rolling out without breaking traffic
Switching a running platform to mandatory mTLS in one step usually breaks something: a batch job without a sidecar, a legacy client, a health checker. A safer rollout runs in phases. First, issue identities and enable mTLS in permissive mode, where plaintext is still accepted, and measure which connections are already mutual. Next, fix or exempt the remaining clients explicitly. Then enforce strict mode namespace by namespace, starting with the least critical, while watching handshake-failure metrics.
Edge traffic from partners can use the same mechanism. RFC 8705 defines how OAuth 2.0 clients authenticate with certificates and how access tokens can be bound to a client certificate, so that a stolen token is useless without the matching private key.
Operating mTLS day to day
Once mTLS is enforced, certificate problems become availability problems, so they need the same operational care as any other dependency. Three signals cover most incidents. Handshake failures, broken down by source and destination identity, reveal misconfigured clients and expired trust bundles. Certificate time-to-expiry per workload shows renewal failures hours before they cause an outage. Issuer health – request latency and error rate of the issuing service – catches problems that would otherwise surface only when many workloads try to renew at once.
Make identities visible in logs and traces. When every request record contains the caller’s SPIFFE ID, incident responders can see immediately which service did what, and authorisation policies can be tested against real traffic before they are enforced.
Plan for the rare events too. CA rotation should be rehearsed in a staging environment with both old and new roots trusted during an overlap period. Break-glass procedures – for example temporarily switching one namespace back to permissive mode – must be documented, access-controlled and alerted, so that an emergency fix does not silently become the permanent configuration.
Finally, keep the developer experience simple. Local development and tests should run with the same identity model, using a lightweight issuer, so that authorisation policies are exercised long before production.
Exemple: Restricting access to a ledger service
A platform with 40 services shared one flat network. After a dependency vulnerability was found in an internal reporting tool, the team introduced a mesh issuing one-hour SPIFFE certificates. Permissive mode revealed three batch jobs calling the ledger without proxies; they were moved into the mesh. Strict mTLS was then enforced for the ledger namespace, and an authorisation policy allowed only checkout and settlement identities to write and reporting to read. The reporting tool, had it been exploited, could now read ledger data through its own allowed path but could no longer write entries or reach other services.
Pièges courants
- Enabling mTLS but accepting any certificate signed by the CA, without checking identity.
- Using certificates valid for a year and renewing them manually.
- Identifying workloads by IP address or host name inside certificates.
- Treating authentication as authorisation – every authenticated service can call everything.
- Switching to strict mode platform-wide in one change.
Questions et réponses
Do we need a service mesh for mTLS?
No. Libraries and proxies can perform mTLS directly, and SPIRE can issue identities without a mesh. A mesh mainly saves effort by handling issuance, rotation and policy consistently for every service.
What does mTLS cost in latency?
With TLS 1.3 and connection reuse the handshake cost is paid rarely, and symmetric encryption on modern CPUs is cheap. Measure, but the overhead is usually a small fraction of a typical p99 budget.
How does this relate to OAuth tokens?
They complement each other. mTLS authenticates the calling workload; tokens carry user or client authorisation. Certificate-bound tokens (RFC 8705) tie the two together.
Liste de contrôle
- Define a trust domain and an identity naming scheme.
- Set up a dedicated workload CA with protected root keys.
- Automate attestation, issuance and rotation of short-lived certificates.
- Roll out in permissive mode and measure before enforcing.
- Write deny-by-default authorisation policies per namespace.
- Monitor handshake failures and certificate expiry.
- Plan and rehearse CA rotation.
Sources primaires
- RFC 8446 – The Transport Layer Security (TLS) Protocol Version 1.3
- RFC 8705 – OAuth 2.0 Mutual-TLS Client Authentication and Certificate-Bound Access Tokens
- SPIFFE specifications – SPIFFE ID and X.509-SVID
- NIST SP 800-207 – Zero Trust Architecture