Parsing ISO 20022 and other large XML messages safely at scale
XML is still the backbone of many business-to-business integrations, and ISO 20022 has made it more important, not less. Files of hundreds of megabytes, several schema versions in parallel and untrusted senders turn a routine parsing task into a security and reliability problem.
See sisu on praegu saadaval ainult inglise keeles.
Why XML pipelines still matter
Many integrations between companies, banks and public authorities exchange XML files or XML payloads over APIs. ISO 20022 is the most visible example: it defines a large catalogue of business messages – for example pain.001 for credit transfer initiation, pacs.008 for customer credit transfers between institutions and camt.053 for account statements. On the Swift network, the coexistence period between legacy MT messages and ISO 20022 for cross-border payment instructions ended on 22 November 2025, so systems that previously handled fixed-format MT text now receive structured, much richer XML.
For engineering teams this means larger messages, more optional structure and stricter validation – handled by software that often started life as a simple file importer.
Hardening the parser first
Every XML parser that processes data from outside your trust boundary must be configured defensively before any other work starts. The main threats are well documented by OWASP:
- XML External Entities (XXE): a document defines an entity that references a local file or an internal URL; a permissive parser includes the content, leaking files or making server-side requests.
- Entity expansion (“billion laughs”): nested entity definitions expand into gigabytes of text from a few kilobytes of input.
- Oversized input: very deep nesting, huge attribute values or enormous files exhaust memory or CPU.
The fix is consistent: disable DTD processing entirely, disable external entity and external schema resolution, set limits on document size, depth and attribute length, and use secure-processing features where the library offers them. Load schemas only from local, version-controlled files – never from locations referenced inside the incoming document.
Streaming instead of loading
DOM parsers build the full document tree in memory, which typically needs several times the file size. A 500 MB statement file can exhaust a container’s memory limit. Streaming parsers – SAX, StAX, or pull parsers such as XMLReader in PHP and iterparse in Python’s lxml – read the document as a sequence of events and let you process one logical record, such as a single transaction entry, at a time.
The pattern is to stream until the start of a repeating element, build a small subtree for that element, validate and map it, emit the result and discard the subtree. Memory stays flat, throughput is predictable, and a malformed entry can be reported precisely without aborting the whole file when the business rules allow partial acceptance.
Validation in layers
Validation has three layers, each catching different errors:
- Well-formedness: the parser rejects syntactically broken XML.
- Schema validation: the message conforms to the XSD for its declared version – element order, types, cardinality, code lists defined in the schema.
- Business rules: cross-field consistency, control sums against the sum of entries, permitted combinations defined by market practice guidelines, duplicate detection.
ISO 20022 messages carry their version in the XML namespace, for example urn:iso:std:iso:20022:tech:xsd:pain.001.001.09. Detect the namespace before validating and select the matching schema. Keep a registry of accepted versions with explicit start and end dates rather than accepting “whatever validates”. Usage guidelines from communities and market infrastructures restrict the base schemas further; encode those restrictions as additional rules or narrower schemas.
A canonical model between formats
Mapping every input version directly to every output format creates a combinatorial mess. A canonical model – a typed internal representation of the business content – decouples them: each input version maps into the canonical model, and each output maps from it. Adding a new message version then means one new mapping instead of many.
The canonical model should be at least as expressive as the richest input. When data must be mapped to a less expressive legacy format, make truncation and loss explicit: record which fields could not be carried, alert when structured data such as remittance information or structured addresses is shortened, and keep the original message available for audit.
Quarantine, replay and audit
Rejected messages should never simply disappear into a log line. Move them to a quarantine area with a machine-readable report: which layer failed, the XPath of the offending element, the rule identifier and a human-readable explanation. Senders can then correct and resend, and operators can reprocess after a rule fix.
Store every received file immutably with a checksum and receipt timestamp before parsing. Parsing results and emitted events should reference that original, so that any downstream value can be traced back to the exact bytes received. Idempotency matters here too: if the same file arrives twice, the pipeline should recognise it by checksum and message identifiers instead of processing it again.
Performance and capacity
Streaming parsers in compiled languages process tens of megabytes per second per core; schema validation and business rules usually dominate. Profile with realistic files, including the largest and most deeply structured you expect. Parallelise across files, or across independent records within a file when ordering does not matter, and keep the number of concurrent large files bounded so that memory stays predictable.
Schema compilation is expensive. Compile each supported schema once at start-up and reuse it, and cache compiled rule sets in the same way.
Testing with realistic corpora
Parsers and mappings need tests that look like production. Build a corpus of sample messages for every supported version: minimal valid messages, maximal messages with every optional element populated, messages that exercise each business rule, and known-invalid messages with the expected rejection report. Where market practice guidelines publish sample messages, include them.
Property-based or fuzz testing adds value on top: generate variations of valid messages – reordered optional elements, extreme lengths, unusual Unicode characters – and check that the pipeline either processes them correctly or rejects them with a precise reason, never crashing or hanging. Run the hostile-document corpus for XXE and entity expansion in the same suite, so that a library upgrade cannot silently re-enable dangerous features.
Näide: A statement importer that ran out of memory
An importer loaded camt.053 statement files with a DOM parser. After a corporate customer began sending monthly files of 700 MB, the container was killed for exceeding its 2 GB memory limit, and the job retried endlessly. The team replaced it with a streaming parser that extracted one Ntry element at a time, disabled DTDs and external entities, validated against the declared schema version and checked the control sum at the end. Peak memory dropped to about 150 MB regardless of file size, a malformed entry produced a quarantine report instead of a crash loop, and duplicate files were detected by checksum and message identification.
Levinud lõksud
- Default parser settings that resolve DTDs and external entities.
- Loading multi-hundred-megabyte files with a DOM parser.
- Validating only against the newest schema version.
- Silent truncation when mapping to legacy formats.
- Rejected files without a precise, machine-readable reason.
Küsimused ja vastused
Is JSON replacing XML for ISO 20022?
ISO 20022 defines a business model that can be represented in different syntaxes, and JSON representations exist in some API contexts. XML remains the dominant syntax for message exchange, so pipelines need to handle it well.
Should we validate in production or only in testing?
In production, on every message. Validation is the boundary that keeps malformed or malicious content out of downstream systems.
How do we test parser hardening?
Keep a corpus of hostile documents – external entities, entity expansion, deep nesting, oversized values – and run it in CI against every parser configuration.
Kontrollnimekiri
- Disable DTDs, external entities and remote schema loading.
- Set limits on size, depth and attribute length.
- Use a streaming parser for large or untrusted files.
- Select the schema per message from its namespace version.
- Add business-rule validation including control sums.
- Map through a canonical model and record any data loss.
- Quarantine rejects with precise reasons and keep originals immutable.
Esmased allikad
- ISO 20022 – Financial services – Universal financial industry message scheme
- Swift – ISO 20022 for cross-border payments and reporting (CBPR+), end of coexistence on 22 November 2025
- OWASP – XML External Entity Prevention Cheat Sheet
- W3C – XML Schema Definition Language (XSD) 1.1