Implementing HL7 ACK Retry with Exponential Backoff

An MLLP sender that fires clinical messages at a receiver and treats every non-AA response identically is a silent data-corruption engine: it either drops messages the receiver was merely too busy to accept, or it hammers a broken payload into an infinite loop. A correct retry policy has to read the acknowledgment code in MSA-1, retry only the failures that are genuinely transient, and resend with a stable message control ID so the receiver can deduplicate. This page is the runnable companion to the HL7 ACK/NACK handling patterns reference: where that page frames the acknowledgment contract, here we build a production MLLP sender that retries on NACK-or-timeout with exponential backoff and jitter, keyed to an idempotent resend.

Acknowledgment codes and their retry policy (quick reference)

The retry decision hinges entirely on the code in MSA-1 of the ACK message the receiver returns. HL7 v2 defines two acknowledgment axes: the original mode codes (AA/AE/AR) and the enhanced mode codes (CA/CE/CR) used when MSH-15/MSH-16 request commit-level acknowledgment. The critical distinction is between transport failures (no ACK arrived, the socket reset, the receiver reported it was busy) and content failures (the message parsed but was semantically rejected). Only the former are safe to retry — retrying a content error just replays the same rejection forever. The table below is the lookup artifact to keep next to your sender.

The single most dangerous cell in that table is AE. An AE means the message reached the application and was rejected on its merits — an unmappable OBX value, a code that fails a required binding, a malformed PID. None of that is fixed by sending the identical bytes again. Retrying AE transiently masks a mapping defect and, if unbounded, turns one bad message into a permanent hot loop against the receiver. Content errors go to a dead-letter queue for reconciliation; transport errors and busy-rejects go through backoff.

Implementation pattern

The state machine below is the contract the sender implements. A message is framed and sent; the sender awaits an ACK. AA/CA completes it. A timeout, connection reset, or busy AR/CR triggers a backoff-and-retry with the same control ID. An AE/CE or a non-transient AR/CR is a terminal reject straight to the dead-letter queue, and exhausting the retry budget dead-letters the message as well.

The complete async sender is below. It frames the message with MLLP block characters, awaits the ACK, parses MSA-1, classifies the outcome, and either completes, backs off and retries with the same MSH-10, or dead-letters. The control ID is generated once — before the retry loop — and reused on every resend so an idempotent receiver can drop duplicates rather than create a second clinical record.

import asyncio
import random
import re
import uuid
from dataclasses import dataclass

# MLLP framing: <VT> ... <FS><CR>
SB, EB, CR = b"\x0b", b"\x1c", b"\x0d"

BASE_DELAY = 0.5      # seconds, the b in base * 2**attempt
DELAY_CAP = 30.0      # seconds, the ceiling on any single backoff
MAX_ATTEMPTS = 6      # total sends before dead-lettering


@dataclass
class AckResult:
    code: str          # MSA-1: AA/AE/AR/CA/CE/CR, or "" on timeout
    transient: bool    # safe to retry?
    text: str          # MSA-3 diagnostic, for the audit trail


def backoff_delay(attempt: int) -> float:
    """Exponential backoff with full jitter, capped.

    delay = random(0, min(cap, base * 2**attempt)). Full jitter spreads
    a thundering herd of senders instead of synchronising their retries.
    """
    ceiling = min(DELAY_CAP, BASE_DELAY * (2 ** attempt))
    return random.uniform(0, ceiling)


def classify_ack(ack_bytes: bytes) -> AckResult:
    """Parse MSA-1 from an ACK and decide whether a retry is warranted."""
    text = ack_bytes.decode("ascii", errors="replace")
    msa = next((s for s in text.split("\r") if s.startswith("MSA")), "")
    fields = msa.split("|")
    code = fields[1].strip() if len(fields) > 1 else ""
    diag = fields[3].strip() if len(fields) > 3 else ""

    if code in ("AA", "CA"):
        return AckResult(code, transient=False, text=diag)       # accepted
    if code in ("AE", "CE"):
        return AckResult(code, transient=False, text=diag)       # content error
    if code in ("AR", "CR"):
        # AR is only transient when the receiver reports it was busy/overloaded.
        busy = bool(re.search(r"busy|overload|throttl|unavailable", diag, re.I))
        return AckResult(code, transient=busy, text=diag)
    return AckResult("", transient=True, text="no MSA / unparseable")

The classify_ack function is where the retry policy lives. AA/CA and AE/CE are both non-transient — accepted and content-error respectively — so neither retries. AR/CR is conditionally transient: only a diagnostic naming a busy or overloaded receiver flips it to retryable, matching the MSA-1 semantics in the HL7 v2 message structure breakdown. A missing or unparseable ACK (the timeout case) is transient because the receiver’s state is unknown and the idempotent control ID makes a safe resend possible.

async def send_once(host: str, port: int, message: bytes,
                    timeout: float = 10.0) -> AckResult:
    """One MLLP send/ack round trip. Raises on transport failure."""
    reader, writer = await asyncio.open_connection(host, port)
    try:
        writer.write(SB + message + EB + CR)
        await writer.drain()
        # readuntil the MLLP end block; a timeout here is a transient failure.
        frame = await asyncio.wait_for(reader.readuntil(EB + CR), timeout)
        return classify_ack(frame.strip(SB + EB + CR))
    finally:
        writer.close()
        await writer.wait_closed()


async def send_with_retry(host: str, port: int, payload: str,
                          dead_letter) -> AckResult:
    """Send an HL7 v2 message, retrying transient failures with backoff.

    MSH-10 (the message control ID) is generated ONCE and reused on every
    resend, so an idempotent receiver deduplicates instead of double-storing.
    """
    control_id = uuid.uuid4().hex[:20]          # stable across all attempts
    message = payload.replace("{MSH10}", control_id).encode("ascii")

    for attempt in range(MAX_ATTEMPTS):
        try:
            result = await send_once(host, port, message)
        except (asyncio.TimeoutError, ConnectionError, asyncio.IncompleteReadError):
            result = AckResult("", transient=True, text="transport failure")

        if result.code in ("AA", "CA"):
            return result                        # accepted, done

        if not result.transient:                 # AE/CE/hard AR -> reject
            await dead_letter(control_id, message, result, reason="rejected")
            return result

        if attempt < MAX_ATTEMPTS - 1:           # transient, and budget remains
            await asyncio.sleep(backoff_delay(attempt))

    # Budget exhausted on a transient failure: give up to the dead-letter.
    await dead_letter(control_id, message, result, reason="max_attempts")
    return result

Two properties make this safe. First, control_id is bound before the loop and reused on every resend, so the receiver sees the same MSH-10 and treats attempt two as a duplicate of attempt one — not a new clinical event. Second, only result.transient outcomes reach the sleep; an AE short-circuits to the dead-letter on the first response. This is the ingestion-side mirror of the idempotency discipline in implementing idempotent clinical data loads, where the same control ID is the natural deduplication key at the storage layer.

Validation and testing

The retry logic is deterministic apart from the jitter, so it is straightforward to assert. Seed the RNG to make backoff_delay reproducible, then prove three invariants: backoff grows and is capped, an AE never retries, and every resend reuses the original MSH-10.

import random


def test_backoff_grows_and_caps():
    random.seed(0)
    # With full jitter, the CEILING grows as 2**attempt then flattens at the cap.
    ceilings = [min(DELAY_CAP, BASE_DELAY * 2 ** n) for n in range(8)]
    assert ceilings == sorted(ceilings)          # monotonically non-decreasing
    assert max(ceilings) == DELAY_CAP            # never exceeds the cap
    assert all(0 <= backoff_delay(n) <= c for n, c in enumerate(ceilings))


def test_ae_is_not_retried():
    ack = b"MSH|^~\\&|R|R|S|S|20260715||ACK|1|P|2.5.1\rMSA|AE|abc|LOINC code unmapped\r"
    result = classify_ack(ack)
    assert result.code == "AE"
    assert result.transient is False             # content error never retries


def test_ar_busy_is_transient():
    busy = classify_ack(b"MSA|AR|abc|Receiver busy, try later\r")
    hard = classify_ack(b"MSA|AR|abc|Unsupported message type\r")
    assert busy.transient is True and hard.transient is False


def test_resend_reuses_control_id():
    sent = []

    async def fake_send(host, port, message, timeout=10.0):
        sent.append(message)                     # capture every attempt
        raise asyncio.TimeoutError

    async def dlq(cid, msg, res, reason):
        pass

    import types
    g = globals()
    g["send_once"] = fake_send                    # patch the transport
    asyncio.run(send_with_retry("h", 1, "MSH|...|{MSH10}|P|2.5.1\r", dlq))
    ids = {m.split(b"|")[-3] if b"|" in m else m for m in sent}
    assert len(sent) == MAX_ATTEMPTS and len(ids) == 1   # same MSH-10 every time

test_backoff_grows_and_caps asserts the ceiling sequence is monotonic and flattens at DELAY_CAP; test_ae_is_not_retried locks in the rule that content errors bypass the retry loop; test_resend_reuses_control_id drives the full sender against a transport that always times out and proves all six attempts carry one control ID. Run these in CI so a refactor that reintroduces per-attempt ID generation — the classic duplicate-record bug — fails the build.

Gotchas & compliance constraints

• A per-attempt MSH-10 creates duplicate clinical records. If the control ID is regenerated inside the retry loop, a receiver that already stored attempt one on a delayed ACK will store attempt two as a distinct observation — a phantom duplicate lab result or medication order. The control ID must be generated once and treated as the idempotency key end to end, exactly as the receiver’s deduplication expects.

• Unbounded retries amplify an outage into an incident. A sender with no MAX_ATTEMPTS and no cap converts a transient receiver blip into a self-inflicted denial of service, and a mis-retried AE becomes an infinite hot loop. Bound the attempt count, cap the delay, use full jitter to desynchronize a fleet of senders, and dead-letter anything that exhausts the budget for human reconciliation.

• The ACK trail is an audit artifact, not just a control signal. HL7 acknowledgments record whether each clinical message was accepted, rejected, or abandoned — provenance a HIPAA Security Rule audit (45 CFR §164.312(b)) can require you to reproduce. Persist the MSH-10, the MSA-1 code, the MSA-3 diagnostic, and the attempt count for every message, and mask any PHI that leaks into MSA-3 diagnostic text before it reaches centralized logging.

• Stable control ID — MSH-10 generated once and reused on every resend

• Bounded retries — finite MAX_ATTEMPTS, capped delay, full jitter

• Correct classification — only timeouts, resets, and busy AR/CR retry; AE/CE dead-letter immediately

• Audit persistence — control ID, MSA-1, MSA-3, and attempt count logged with PHI masked

Related

• HL7 ACK/NACK handling patterns — parent guide: the full acknowledgment contract and dead-letter design
• Implementing idempotent clinical data loads — the receiver side: using MSH-10 as a deduplication key
• HL7 v2 message structure breakdown — where MSA-1 and MSH-10 sit in the segment grammar
• FHIR & HL7 v2 standards architecture — how ingestion acknowledgment fits the wider clinical ETL pipeline