Clinical Data

Converting HL7 v2 Pipe-Delimited to XML Step-by-Step: Clinical ETL Pipeline Implementation

Converting an HL7 v2 pipe-delimited message into namespaced XML (urn:hl7-org:v2xml) is the single most common pre-FHIR normalization step in a clinical pipeline, and the place where the most data is silently corrupted. The failure is rarely the XML serializer itself — it is naive delimiter handling: a split('|') that ignores escaped field separators, repetition characters folded into a single string, or an MSH-2 encoding field that contains the very delimiters used to parse it. This page is a focused, runnable recipe for a deterministic, PHI-safe pipe-to-XML transform. It sits within the HL7 v2 message structure breakdown, which defines the segment grammar this converter assumes, and it is the stage immediately before the FHIR & HL7 v2 standards architecture maps each segment to a resource.

Delimiter-to-XML Mapping Reference

Every HL7 v2 message self-describes its delimiters in the MSH segment. MSH-1 is the field separator (the character at byte offset 3, always | in practice), and MSH-2 carries the remaining four encoding characters in fixed order. Read them dynamically; never hardcode ^~\&, because some vendor interfaces deviate. The table below is the complete construct-to-element mapping this converter implements.

HL7 construct Source in MSH Default char Role XML serialization
Field separator MSH-1 | Splits a segment into fields One child element per field, tagged SEG.n
Component MSH-2[0] ^ Splits a field into components Nested SEG.n.m elements
Repetition MSH-2[1] ~ Repeats a field Multiple sibling SEG.n elements
Escape MSH-2[2] \ Escapes a delimiter inside data Resolved to literal char, then XML-escaped
Subcomponent MSH-2[3] & Splits a component Nested SEG.n.m.k elements

Two consequences drive the implementation. First, MSH-1 and MSH-2 are self-referential — MSH-2 literally contains ^, ~, and &, so they must be emitted as atomic text and never re-split. Second, repetition is structural, not lexical: a repeated field becomes multiple sibling elements, not one concatenated value, because downstream FHIR mapping iterates over occurrences.

Each HL7 v2 delimiter tier maps to a distinct v2xml element shape Four rows. The field separator pipe makes one child element per field tagged SEG.n. The repetition tilde produces multiple sibling elements that share the same SEG.n tag. The component caret produces nested SEG.n.m elements inside the field element. The subcomponent ampersand produces nested SEG.n.m.k elements one level deeper. Left column shows the ER7 pipe-delimited fragment with its delimiter highlighted; right column shows the resulting urn:hl7-org:v2xml serialization. ER7 fragment (pipe-delimited) urn:hl7-org:v2xml element field separator | · MSH-1 OBX|1|ST|GLU <OBX.1>1</OBX.1> <OBX.2>ST</OBX.2> <OBX.3>GLU</OBX.3> one child element per field — SEG.n repetition ~ · MSH-2[1] OBX-5 = A~B <OBX.5>A</OBX.5> <OBX.5>B</OBX.5> sibling elements, same SEG.n tag component ^ · MSH-2[0] OBX-3 = GLU^Glucose field splits into ordered components <OBX.3> <OBX.3.1>GLU</OBX.3.1> <OBX.3.2>Glucose</OBX.3.2> </OBX.3> nested — SEG.n.m subcomponent & · MSH-2[3] OBX-3.1 = cap&plasma component splits one level deeper <OBX.3.1> <OBX.3.1.1>cap</OBX.3.1.1> <OBX.3.1.2>plasma</OBX.3.1.2> </OBX.3.1> nested deeper — SEG.n.m.k

Implementation Pattern

The converter below is complete and end-to-end: it extracts encoding characters from MSH, normalizes transport line endings to the spec-mandated carriage return, resolves HL7 escape sequences, and serializes to the urn:hl7-org:v2xml namespace. It relies on xml.etree.ElementTree to perform XML escaping on text nodes, so HL7 escapes are resolved first and XML escaping happens last — never the reverse (see the gotchas below).

import xml.etree.ElementTree as ET


def extract_encoding(raw: str) -> dict:
    """Read the message's own delimiters from MSH-1 and MSH-2."""
    if not raw.startswith("MSH"):
        raise ValueError("Message must begin with an MSH segment")
    enc = raw[4:8]  # the four encoding chars immediately after "MSH|"
    if len(enc) < 4:
        raise ValueError(f"Malformed MSH-2: expected 4 chars, got {enc!r}")
    return {
        "field":        "|",
        "component":    enc[0],   # ^
        "repetition":   enc[1],   # ~
        "escape":       enc[2],   # \
        "subcomponent": enc[3],   # &
    }


def normalize_segments(raw: str) -> list[str]:
    """Force CR-only line breaks per the HL7 v2 spec, then split into segments.

    TCP/MLLP and file transports frequently inject \\n or \\r\\n; left
    unnormalized they cause segment misalignment and orphaned XML nodes.
    """
    cleaned = raw.replace("\r\n", "\r").replace("\n", "\r").strip("\r")
    return [seg for seg in cleaned.split("\r") if seg.strip()]


def unescape_hl7(value: str, enc: dict) -> str:
    """Resolve HL7 escape sequences to literal delimiter characters."""
    e = enc["escape"]
    replacements = {
        f"{e}F{e}": enc["field"],         # \F\ -> |
        f"{e}S{e}": enc["component"],     # \S\ -> ^
        f"{e}T{e}": enc["subcomponent"],  # \T\ -> &
        f"{e}R{e}": enc["repetition"],    # \R\ -> ~
        f"{e}E{e}": e,                    # \E\ -> \
    }
    for token, char in replacements.items():
        value = value.replace(token, char)
    return value


def hl7_to_xml(raw: str) -> str:
    enc = extract_encoding(raw)
    root = ET.Element("HL7Message", {"xmlns": "urn:hl7-org:v2xml"})

    for segment in normalize_segments(raw):
        seg_id = segment[:3]
        seg_el = ET.SubElement(root, seg_id)

        # MSH is special: MSH-1 is the field separator itself, so the field
        # list is reconstructed with it prepended.
        if seg_id == "MSH":
            fields = [enc["field"]] + segment[4:].split(enc["field"])
        else:
            fields = segment.split(enc["field"])[1:]

        for f_idx, field in enumerate(fields, start=1):
            tag = f"{seg_id}.{f_idx}"
            # MSH-1 and MSH-2 contain delimiter chars; keep them atomic.
            atomic = seg_id == "MSH" and f_idx <= 2
            repeats = [field] if atomic else field.split(enc["repetition"])

            for repeat in repeats:  # each repetition -> a sibling element
                field_el = ET.SubElement(seg_el, tag)
                comps = [repeat] if atomic else repeat.split(enc["component"])
                if len(comps) == 1:
                    field_el.text = unescape_hl7(repeat, enc)
                else:
                    for c_idx, comp in enumerate(comps, start=1):
                        comp_el = ET.SubElement(field_el, f"{tag}.{c_idx}")
                        comp_el.text = unescape_hl7(comp, enc)

    ET.indent(root)  # pretty-print; drop in latency-sensitive paths
    return ET.tostring(root, encoding="unicode")

Subcomponent splitting (&) is omitted here for readability; extend the inner else branch with one more split(enc["subcomponent"]) level when your target XSD requires SEG.n.m.k nodes. The same delimiter resolution underpins more granular work such as parsing HL7 repeating groups and handling HL7 escape sequences inside ETL scripts.

Validation & Testing

XML that is merely well-formed is not enough for clinical routing — it must also be structurally correct, meaning repetitions are siblings and escaped delimiters survived as data. Assert against a golden message rather than eyeballing output. The check below confirms that a \F\-escaped pipe round-trips to a literal | inside the value and that a repeated OBX-5 produces two sibling elements, not one concatenated node.

NS = {"v2": "urn:hl7-org:v2xml"}

golden = (
    "MSH|^~\\&|LAB|HOSP|EHR|HOSP|20260101120000||ORU^R01|MSG001|P|2.5.1\r"
    "OBX|1|ST|note^free text||A\\F\\B~second rep"
)

import xml.etree.ElementTree as ET
tree = ET.fromstring(hl7_to_xml(golden))

obx5 = tree.findall(".//v2:OBX/v2:OBX.5", NS)
assert len(obx5) == 2, f"expected 2 repetitions, got {len(obx5)}"
assert obx5[0].text == "A|B", f"escape not resolved: {obx5[0].text!r}"
assert obx5[1].text == "second rep"
assert tree.find(".//v2:MSH/v2:MSH.2", NS).text == "^~\\&"  # stayed atomic
print("OK: repetitions, escapes, and MSH-2 atomicity verified")

For schema conformance, validate the serialized output against a version-specific HL7 v2 XSD before any downstream routing. A pre-built XSD can be checked from the command line with xmllint --noout --schema messages_2.5.1.xsd message.xml, which surfaces cardinality violations and missing mandatory segments early — exactly the failures that otherwise corrupt the downstream type coercion stage.

Gotchas & Compliance Constraints

Double-escaping is the most common silent corruption. If you XML-escape (& → &amp;) before handing text to a DOM builder, the builder escapes the ampersand again, producing &amp;amp; in the output. Resolve HL7 escapes to literal characters first, then let one — and only one — layer perform XML escaping. The converter above delegates that entirely to ElementTree.text.

MSH-1/MSH-2 self-reference. The encoding field carries the delimiters used to parse every other field, so any generic field/component split applied to MSH-2 will shred it into spurious MSH.2.1, MSH.2.2 nodes. Treat the first two MSH fields as atomic, as the atomic guard does. Mishandling this is also the root of many version-detection bugs when distinguishing releases per the HL7 v2.5 vs v2.7 differences.

PHI never enters XML staging unprotected (HIPAA minimum necessary). The intermediate XML is still PHI. Before serialization, deterministically hash identifying fields — replace PID-3 (Patient Identifier List) and PID-7 (Date of Birth) with HMAC-SHA256 digests under a pipeline-managed salt to preserve join integrity without exposing raw values, and drop OBX-5 free-text when only coded observations (OBX-3) are required downstream. Log only MSH-10 (Message Control ID), MSH-9 (Message Type), and a payload hash; never persist raw payloads in staging tables, and route any message that fails XSD validation to an access-controlled dead-letter queue with its error offset rather than dropping it silently.