Patient Matching and Master Patient Index (MPI) Strategies for Clinical ETL

The United States has no national patient identifier, so every health system that ingests records from more than one source must decide, on its own, whether two demographic records describe the same human being — and it must decide with enough precision that it never merges two different patients, and enough recall that it never fragments one patient into many charts. That decision is the job of a Master Patient Index (MPI), or an Enterprise MPI (EMPI) when it spans organizations: a service that assigns and defends a stable enterprise identifier across every feed. Within HIPAA De-identification & Patient Matching for Clinical ETL, this page is the engineering reference for the matching engine itself — the deterministic and probabilistic algorithms, the blocking that makes them tractable, the survivorship policy that resolves a merge, and the audit trail that makes every link reversible. It is written to be lifted into a Python service and tested against a labeled record-pair set in isolation.

Two failure modes frame everything below. An over-match (false positive) fuses two patients into one enterprise ID: the second patient inherits the first patient’s allergies, medications, and history — a patient-safety event and, because one patient’s PHI now appears in another’s chart, a HIPAA disclosure. An under-match (false negative) splits one patient across multiple IDs: allergy checks miss, duplicate orders fire, and the longitudinal record is silently incomplete. The entire design is a controlled trade between these two, tuned deliberately rather than accidentally.

Prerequisites & Context

An MPI is only as good as the data feeding it. Matching operates on demographics after parsing and normalization, not on raw wire bytes, so confirm the following are in place before implementing anything here:

• Normalized names — case-folded, accent-stripped, whitespace-collapsed, with nickname resolution (Bob→Robert) and a documented policy for hyphenated and multi-token surnames. Store the normalized form alongside the original; never overwrite the source.
• Normalized dates — every date of birth coerced to a single ISO-8601 YYYY-MM-DD representation with a defined handling for partial dates and the 1900-01-01/0000 placeholder values that registration systems emit. See type coercion for clinical data types for the coercion discipline this depends on.
• Normalized identifiers — SSN, MRN, and member IDs stripped of punctuation, with each identifier tagged by its assigning authority namespace so an Epic MRN is never compared against a Cerner MRN as if they shared a numbering space.
• A survivorship / golden-record policy — a written, versioned rule set that decides, when two records link, which field values survive into the enterprise golden record (most-recent-wins, source-priority, or longest-non-null). Matching without survivorship produces links you cannot render into a usable record.
• A clerical-review UI — a human-in-the-loop queue where trained staff adjudicate borderline pairs. Probabilistic matching requires a review band; a design that auto-decides every pair is either too loose or too strict.
• A labeled evaluation set — a few thousand hand-adjudicated record pairs (true matches and true non-matches) used to estimate m/u probabilities and to measure precision and recall before and after every threshold change.

If any of these is missing, resolve it first. In particular, a matching engine run without a survivorship policy and a review UI will produce enterprise IDs that no downstream system can safely consume.

Concept & Spec Detail: Deterministic vs Probabilistic Linkage

Why there is no shortcut

Because there is no national patient identifier — the funding for one has been prohibited by congressional appropriations riders since 1998 — an MRN is only unique within its assigning authority, and an SSN is optional, frequently absent, often shared within a family by data-entry error, and legally constrained in how it may be used. That leaves demographics: name, date of birth, sex, and address. None of these is individually unique or stable, so matching is fundamentally a statistical inference over a vector of noisy field comparisons.

Deterministic matching

Deterministic matching declares a match when a fixed rule over normalized fields is satisfied exactly. A rule is a boolean conjunction — for example, SSN equal AND date of birth equal, or surname equal AND given-name equal AND DOB equal AND sex equal. Deterministic rules are high precision, low recall: when a strong rule fires you can trust it, but any typo, nickname, transposition, or missing field defeats the exact comparison and the true match is missed. Deterministic passes are cheap, explainable, and ideal as a first stage that clears the unambiguous majority of pairs before the expensive scorer runs. Their weakness is brittleness: real registration data is full of Jon/John, transposed birth-date digits, and blank SSNs.

Probabilistic matching (Fellegi–Sunter)

Probabilistic record linkage, formalized by Fellegi and Sunter in 1969, replaces the boolean rule with a weighted score. For each comparison field it estimates two probabilities:

• The m-probability is P(field agrees | the pair is a true match) — one minus the field’s error/change rate. A reliable field such as date of birth has a high m (agreement is expected when the records truly match); a volatile field such as address has a lower m.
• The u-probability is P(field agrees | the pair is a non-match) — the chance two different people agree on the field by coincidence. It is roughly the probability mass of the most common value: sex has u ≈ 0.5 (two random people share a sex half the time), while a full nine-digit SSN has a u near 1e-7.

Each field then contributes a log-likelihood weight, summed across fields into a total match score (a log-likelihood ratio):

• Agreement weight = log2(m / u) — added when the field agrees. Large and positive when m is high and u is tiny (SSN agreement is powerful evidence).
• Disagreement weight = log2((1 - m) / (1 - u)) — added (it is negative) when the field disagrees. Disagreement on a high-m field like DOB is strong evidence against a match.

The summed score is compared against two thresholds. Above the upper threshold, auto-link. Below the lower threshold, auto-reject as a non-match. Between them lies the clerical-review band: pairs the math cannot resolve confidently, routed to a human. This three-way routing is the defining structural feature that separates a real MPI from a naive scorer.

Blocking: making the comparison tractable

Comparing every incoming record against every existing record is O(n²) — infeasible against a master index of tens of millions. Blocking generates candidate pairs cheaply: records are bucketed by a coarse blocking key, and full field-by-field comparison runs only within a block. A key such as Soundex(surname) + birth_year places Smith/Smyth born in 1984 into the same bucket while excluding the vast majority of the index from comparison. Blocking trades a small recall loss (a true match split across two blocks by an error in both key components is missed) for orders-of-magnitude fewer comparisons. Production systems run multiple blocking passes with different keys (one on name+DOB, one on SSN, one on DOB+sex+ZIP) and union the candidate sets so no single key’s blind spot drops a true pair.

Deterministic vs probabilistic at a glance

Example field weighting

Estimated from a labeled pair set, the weights below show why identifiers dominate and why a single field is never decisive alone. Agreement weight is log2(m/u); disagreement weight is log2((1-m)/(1-u)).

Read the table operationally. SSN agreement contributes +23 bits — enough to auto-link almost by itself — but SSN agrees on true matches only 85% of the time (blanks and family sharing), so it cannot be a required field. Sex agreement is worth under 1 bit (half of everyone agrees by chance) yet sex disagreement costs −5.64 bits, so it acts as a tie-breaker and a veto, not primary evidence. A match on surname + given + DOB but a blank SSN still clears a well-set upper threshold; that is precisely the recall a deterministic SSN rule would have thrown away.

Implementation

The engine decomposes into four ordered stages: normalize fields, derive a blocking key, run a deterministic rule, then score and route the survivors. Each stage carries a validation gate.

Step 1 — Normalize the comparison fields

Every comparison operates on canonical forms so that MARÍA, maria, and Maria compare equal and dates compare on a single representation. Normalize once, store the result, never mutate the source value.

import re
import unicodedata
from datetime import date

_NICKNAMES = {"bob": "robert", "bill": "william", "jim": "james",
              "liz": "elizabeth", "peggy": "margaret", "jon": "john"}


def norm_name(raw: str) -> str:
    """Casefold, strip accents, drop non-letters, resolve common nicknames."""
    if not raw:
        return ""
    decomposed = unicodedata.normalize("NFKD", raw)
    ascii_only = decomposed.encode("ascii", "ignore").decode("ascii")
    cleaned = re.sub(r"[^a-z]", "", ascii_only.lower())
    return _NICKNAMES.get(cleaned, cleaned)


def norm_dob(raw: str) -> str | None:
    """Coerce a date of birth to ISO YYYY-MM-DD; reject known placeholders."""
    if not raw:
        return None
    digits = re.sub(r"[^0-9]", "", raw)
    if len(digits) != 8:
        return None
    iso = f"{digits[0:4]}-{digits[4:6]}-{digits[6:8]}"
    if iso in ("1900-01-01", "0001-01-01"):  # registration placeholders
        return None
    try:
        date.fromisoformat(iso)
    except ValueError:
        return None
    return iso


def norm_ssn(raw: str) -> str | None:
    """Strip punctuation; reject impossible/placeholder SSNs."""
    if not raw:
        return None
    digits = re.sub(r"[^0-9]", "", raw)
    if len(digits) != 9 or digits in ("000000000", "123456789", "999999999"):
        return None
    if digits[0:3] == "000" or digits[3:5] == "00" or digits[5:9] == "0000":
        return None
    return digits

Validation gate: assert that normalization is idempotent — norm_name(norm_name(x)) == norm_name(x) — and that every placeholder DOB and structurally invalid SSN returns None rather than a comparable value. A placeholder that survives normalization becomes a false agreement that inflates every score in its block.

Step 2 — Derive the blocking key

The blocking key must be cheap, stable under the errors it is meant to tolerate, and coarse enough to keep blocks small. Soundex on the surname absorbs spelling variants; birth year absorbs day/month transpositions while still partitioning the index.

def soundex(name: str) -> str:
    """Classic Soundex: first letter + 3 encoded consonant digits."""
    if not name:
        return "0000"
    name = name.upper()
    codes = {"B": "1", "F": "1", "P": "1", "V": "1",
             "C": "2", "G": "2", "J": "2", "K": "2", "Q": "2",
             "S": "2", "X": "2", "Z": "2", "D": "3", "T": "3",
             "L": "4", "M": "5", "N": "5", "R": "6"}
    first, encoded, prev = name[0], "", codes.get(name[0], "")
    for ch in name[1:]:
        digit = codes.get(ch, "")
        if digit and digit != prev:
            encoded += digit
        if ch not in "HW":
            prev = digit
    return (first + encoded + "000")[:4]


def blocking_keys(record: dict) -> list[str]:
    """Multiple blocking passes; union of candidate sets covers each key's blind spot."""
    keys = []
    surname, dob, ssn = record["surname"], record["dob"], record.get("ssn")
    if surname and dob:
        keys.append(f"NAME:{soundex(surname)}|{dob[:4]}")          # Soundex + birth year
    if ssn:
        keys.append(f"SSN:{ssn}")                                   # exact SSN pass
    if dob and record.get("zip"):
        keys.append(f"GEO:{dob}|{record['sex']}|{record['zip']}")   # DOB + sex + ZIP pass
    return keys

Validation gate: measure blocking recall against the labeled set — the fraction of true-match pairs that share at least one blocking key. A key so tight that it drops true pairs before scoring caps the engine’s achievable recall; if pair-completeness falls below your target, add a pass rather than loosening the scorer.

Step 3 — Run the deterministic pass

Before spending a score, clear the unambiguous pairs with an exact rule. A high-precision conjunction auto-links or auto-rejects without a scorer call and gives the review UI an explainable reason.

def deterministic_match(a: dict, b: dict) -> str | None:
    """Return 'match' on a strong exact rule, else None (defer to scoring).

    Rule A: SSN present, equal, AND DOB equal -> match.
    Rule B: surname, given, DOB, sex all present and equal -> match.
    """
    if a.get("ssn") and a["ssn"] == b.get("ssn") and a["dob"] and a["dob"] == b.get("dob"):
        return "match"
    strong = ("surname", "given", "dob", "sex")
    if all(a.get(f) and a.get(f) == b.get(f) for f in strong):
        return "match"
    return None

Validation gate: on the labeled set the deterministic pass must return zero false positives — every rule it fires must be a true match. If a rule mislabels even one pair, tighten the conjunction (add a field) rather than accepting the error; the deterministic tier’s entire value is that its output needs no review.

Step 4 — Score with Fellegi–Sunter and route by threshold

Pairs the deterministic pass leaves undecided are scored field by field. Each field contributes its agreement weight when it agrees and its (negative) disagreement weight when it disagrees; an absent field on either side contributes zero. The summed score routes the pair.

import math
from dataclasses import dataclass


@dataclass(frozen=True)
class FieldWeight:
    m: float  # P(agree | true match)
    u: float  # P(agree | non-match by chance)

    @property
    def agree(self) -> float:
        return math.log2(self.m / self.u)

    @property
    def disagree(self) -> float:
        return math.log2((1 - self.m) / (1 - self.u))


FIELD_WEIGHTS = {
    "ssn":     FieldWeight(m=0.85, u=0.0000001),
    "dob":     FieldWeight(m=0.95, u=0.0004),
    "surname": FieldWeight(m=0.90, u=0.005),
    "given":   FieldWeight(m=0.88, u=0.02),
    "sex":     FieldWeight(m=0.98, u=0.5),
    "zip":     FieldWeight(m=0.72, u=0.008),
}

UPPER_THRESHOLD = 14.0   # bits: auto-link at or above
LOWER_THRESHOLD = -3.0   # bits: auto-reject at or below


def score_pair(a: dict, b: dict) -> float:
    """Sum log-likelihood weights across comparable fields (total match score)."""
    score = 0.0
    for field, w in FIELD_WEIGHTS.items():
        va, vb = a.get(field), b.get(field)
        if not va or not vb:
            continue          # field absent on a side: no evidence either way
        score += w.agree if va == vb else w.disagree
    return score


def route(score: float) -> str:
    """Three-way threshold routing into the matching decision."""
    if score >= UPPER_THRESHOLD:
        return "auto_match"
    if score <= LOWER_THRESHOLD:
        return "non_match"
    return "clerical_review"


def match_decision(a: dict, b: dict) -> dict:
    """Full pipeline for one candidate pair: deterministic first, then scored."""
    if deterministic_match(a, b) == "match":
        return {"decision": "auto_match", "method": "deterministic", "score": None}
    s = score_pair(a, b)
    return {"decision": route(s), "method": "probabilistic", "score": round(s, 2)}

Validation gate: replay the labeled set through match_decision and measure precision at the upper threshold and recall across auto-match plus confirmed review. Tune UPPER_THRESHOLD, LOWER_THRESHOLD, and the per-field m/u — never in production, always against the labeled set — until precision meets the safety bar (over-matches are the costly error) while the review band stays small enough for staff to clear. Record the threshold values and the evaluation metrics as a versioned artifact tied to each release.

Edge Cases & Vendor Deviations

The algorithm is clean; production feeds are adversarial. The cases below are the ones a labeled set rarely covers and that cause the most damage in the field.

Two cross-cutting notes. First, the deterministic tier deserves its own careful construction — the exact key that clears the unambiguous majority is engineered in building a deterministic patient match key in Python, including how to compose and test the conjunctions that must never yield a false positive. Second, the probabilistic tier’s m/u estimation, threshold calibration, and expectation-maximization training over unlabeled pairs are covered in depth in probabilistic patient matching with Fellegi–Sunter.

Compliance Note: A Wrong Merge Is a Reportable Event

Patient matching is not a best-effort convenience — under HIPAA it is a regulated operation because a bad decision moves PHI. A false merge places one patient’s diagnoses, medications, and history into another patient’s chart: that is an impermissible disclosure of PHI under 45 CFR §164.502 and a patient-safety event, because the clinician now treats against a contaminated record. A false split hides an allergy or an active problem behind a duplicate chart. Both must be engineered against with the same rigor as de-identification, governed by the same PHI audit logging and access controls that protect the rest of the warehouse.

Three constraints apply specifically to the MPI. First, every matching decision must be auditable: the append-only log records the pair, the method (deterministic rule vs score), the score and the thresholds in force, the field-level agreement vector, and — for review-band pairs — the reviewer identity and timestamp. An auditor must be able to reconstruct why two records were linked. Second, every merge must be reversible: store links as an association between source records and an enterprise ID, never by destructively overwriting one record’s identifiers into another. An unmerge must restore both records to their pre-link state, which is only possible if the survivorship policy recorded which value came from which source. Third, apply minimum necessary in the clerical-review UI: reviewers adjudicating identity need name, DOB, sex, and partial identifiers to decide a match — they do not need the clinical record, so the review screen must not expose diagnoses, notes, or medications. Over-exposing the chart in a review queue turns a matching tool into an unnecessary PHI access surface and a finding waiting to happen.

Troubleshooting

Related

• Building a deterministic patient match key in Python — the high-precision exact-rule tier that clears unambiguous pairs before scoring.
• Probabilistic patient matching with Fellegi–Sunter — m/u estimation, EM training, and threshold calibration for the scoring tier.
• PHI audit logging and access controls — the audit trail and minimum-necessary controls every merge and review must inherit.
• Safe Harbor vs Expert Determination — how the same demographic fields become identifiers to strip once matching is done.
• HIPAA De-identification & Patient Matching for Clinical ETL — the parent architecture overview for both identity operations.