Clinical Data

Probabilistic Patient Matching with the Fellegi–Sunter Model

Deterministic match keys break the moment a name is misspelled, a date of birth is transposed, or an SSN fragment is missing — yet those are exactly the records a master patient index must still link. The Fellegi–Sunter model solves this by scoring each candidate pair on the weight of agreement and disagreement across fields, so partial evidence accumulates into a defensible link-or-not decision instead of an all-or-nothing key comparison. This page is the runnable companion to the broader patient matching and MPI strategies guide: where that page frames when to reach for probabilistic linkage, here we implement the m/u weighting math, the two-threshold routing band, and the tests that keep a false merge from ever reaching production.

The model rests on two conditional probabilities per field. The m-probability is P(field agrees | the two records are a true match) — high for stable identifiers, lower for fields that drift or get mistyped. The u-probability is P(field agrees | the records are a non-match, i.e. they agree only by chance) — driven by how common the value is in the population. From these, every field contributes a base-2 log-likelihood weight: an agreement weight of log2(m / u) when the field agrees, and a disagreement weight of log2((1 - m) / (1 - u)) when it does not. The match score is the sum of per-field weights across the record. Two thresholds turn that score into a decision: at or above the upper threshold the pair auto-matches; below the lower threshold it is a non-match; the span between them is a clerical-review band where a human adjudicates.

Weight quick reference

The table below is the lookup artifact to keep next to your linkage config. Each row is a field with a plausible m and u for a US patient population, and the resulting weights. Note the asymmetry: date of birth contributes a large positive weight on agreement because chance agreement is rare (u tiny), while sex contributes almost nothing on agreement (half the population matches by chance) but a large penalty on disagreement.

Field Example m Example u Agreement weight log2(m/u) Disagreement weight log2((1-m)/(1-u))
Last name 0.90 0.010 +6.49 −3.31
First name 0.90 0.040 +4.49 −3.26
Date of birth 0.95 0.0002 +12.21 −4.32
SSN last 4 0.70 0.0001 +12.77 −1.74
ZIP code 0.85 0.020 +5.41 −2.71
Sex 0.98 0.500 +0.97 −4.64

A weight of +6.49 means an agreement on last name makes a true match 2^6.49 ≈ 90 times more likely than chance; the fields simply add up. A candidate pair that agrees on last name, first name, and date of birth but disagrees on sex scores 6.49 + 4.49 + 12.21 − 4.64 = 18.55 — comfortably a match. A pair agreeing only on sex and ZIP scores 0.97 + 5.41 = 6.38 minus whatever the name/DOB disagreements subtract, which usually lands it below the lower threshold.

Implementation pattern

The implementation has three parts: per-field agreement functions (exact for structured identifiers, fuzzy for names), a scorer that sums the log2 weights, and a router that maps the total to match / review / non-match. The diagram shows how a single scalar score is partitioned by the two thresholds.

Fellegi–Sunter match score axis with lower and upper thresholds A horizontal match-score axis running from negative to positive summed log2 weights. Two vertical thresholds divide it into three bands: scores below the lower threshold route to non-match, scores between the lower and upper thresholds route to the clerical-review band for human adjudication, and scores at or above the upper threshold route to auto-match. Example paired records are plotted along the axis to show which band each falls into. Match score = Σ per-field log2 weights → two thresholds, three routes Non-match auto-reject Clerical review human adjudicates Auto-match link, provisional MPI id score lower threshold e.g. +6.0 upper threshold e.g. +14.0 −2.1 +9.4 +18.6
The summed log-likelihood score is partitioned by two thresholds into non-match, clerical-review, and auto-match routes.

Start with the field comparators. Structured identifiers (date of birth, SSN fragment, sex) use exact agreement after normalization; names use Jaro–Winkler string similarity so Cathy/Kathy still counts as agreement above a cutoff.

"""Fellegi-Sunter probabilistic patient matching."""
from dataclasses import dataclass
from math import log2

# jellyfish provides a battle-tested Jaro-Winkler implementation.
from jellyfish import jaro_winkler_similarity


@dataclass(frozen=True)
class Field:
    """A linkage field: its m/u probabilities and how to compare it."""
    name: str
    m: float                 # P(agree | true match)
    u: float                 # P(agree | non-match, by chance)
    fuzzy: bool = False      # names use similarity, identifiers use exact
    threshold: float = 0.92  # Jaro-Winkler cutoff counted as agreement

    def agreement_weight(self) -> float:
        return log2(self.m / self.u)

    def disagreement_weight(self) -> float:
        return log2((1 - self.m) / (1 - self.u))

    def agrees(self, a: str, b: str) -> bool:
        a, b = (a or "").strip().lower(), (b or "").strip().lower()
        if not a or not b:
            return False  # missing values never count as agreement
        if self.fuzzy:
            return jaro_winkler_similarity(a, b) >= self.threshold
        return a == b


FIELDS = [
    Field("last_name",  m=0.90, u=0.010, fuzzy=True),
    Field("first_name", m=0.90, u=0.040, fuzzy=True),
    Field("dob",        m=0.95, u=0.0002),
    Field("ssn_last4",  m=0.70, u=0.0001),
    Field("zip",        m=0.85, u=0.020),
    Field("sex",        m=0.98, u=0.500),
]

The scorer walks each field, adds the agreement weight when it agrees and the disagreement weight when it does not, and returns the total. A missing value on either side is treated as a disagreement here — some MPIs prefer a neutral zero weight for missing fields, which is a deliberate tuning choice, not a default.

def match_score(rec_a: dict, rec_b: dict, fields=FIELDS) -> float:
    """Sum per-field log2 weights for a candidate record pair."""
    total = 0.0
    for f in fields:
        if f.agrees(rec_a.get(f.name, ""), rec_b.get(f.name, "")):
            total += f.agreement_weight()
        else:
            total += f.disagreement_weight()
    return total


def route(score: float, lower: float = 6.0, upper: float = 14.0) -> str:
    """Map a score to a decision using the two Fellegi-Sunter thresholds."""
    if score >= upper:
        return "match"
    if score < lower:
        return "non_match"
    return "clerical_review"

Thresholds are not magic constants: the upper value is chosen so the estimated false-match rate below it is acceptable, and lower so the estimated false-non-match rate above it is acceptable. The gap between them is the review workload you are willing to staff. In practice you plot the score distribution of labeled pairs and read the thresholds off the tails.

The m and u probabilities above are illustrative. In production you estimate them, either from a labeled sample of known match/non-match pairs (count how often each field agrees within each class) or, when labels are scarce, with the Expectation–Maximization (EM) algorithm, which fits m and u to the observed agreement patterns by treating true-match status as a latent variable. Libraries such as Splink implement the EM route directly.

def estimate_mu_from_labels(pairs, field_name):
    """Direct m/u estimate from labeled pairs: (record_a, record_b, is_match)."""
    m_agree = m_total = u_agree = u_total = 0
    field = next(f for f in FIELDS if f.name == field_name)
    for a, b, is_match in pairs:
        agrees = field.agrees(a.get(field_name, ""), b.get(field_name, ""))
        if is_match:
            m_total += 1
            m_agree += int(agrees)
        else:
            u_total += 1
            u_agree += int(agrees)
    # Laplace smoothing avoids a zero u that would blow up log2(m/u).
    m = (m_agree + 0.5) / (m_total + 1)
    u = (u_agree + 0.5) / (u_total + 1)
    return m, u

Validation and testing

A patient-matching scorer is a safety component: a single false merge fuses two people’s records. Test it against labeled pairs and report precision and recall at your chosen thresholds, and assert that a hand-verified true pair always clears the upper threshold.

def test_known_true_pair_auto_matches():
    a = {"last_name": "Nguyen", "first_name": "Cathy", "dob": "1984-03-11",
         "ssn_last4": "4821", "zip": "02139", "sex": "F"}
    b = {"last_name": "Nguyen", "first_name": "Kathy", "dob": "1984-03-11",
         "ssn_last4": "4821", "zip": "02139", "sex": "F"}   # typo'd first name
    score = match_score(a, b)
    assert score >= 14.0, f"true pair scored {score:.2f}, below upper threshold"
    assert route(score) == "match"


def test_precision_recall_on_labeled_set(labeled_pairs):
    tp = fp = fn = 0
    for a, b, is_match in labeled_pairs:
        predicted = route(match_score(a, b)) == "match"
        tp += int(predicted and is_match)
        fp += int(predicted and not is_match)
        fn += int(not predicted and is_match)
    precision = tp / (tp + fp) if (tp + fp) else 0.0
    recall = tp / (tp + fn) if (tp + fn) else 0.0
    # Merges are irreversible in practice: hold precision high.
    assert precision >= 0.99, f"precision {precision:.3f} too low"
    assert recall >= 0.90, f"recall {recall:.3f} too low"

Report precision and recall separately for the auto-match band and for auto-match-plus-review, so you can see how much recall the clerical band is buying you. Freeze the labeled set as a regression fixture and re-run it whenever weights, thresholds, or the fuzzy cutoff change — a threshold nudge that quietly drops precision below 0.99 should fail CI, not surface as a merged-chart incident.

Gotchas and compliance constraints

  • Correlated fields double-count evidence. Fellegi–Sunter assumes fields are conditionally independent given match status. City, ZIP, and state are not independent — agreement on ZIP nearly guarantees agreement on city — so summing all three inflates the score and manufactures false confidence. Collapse correlated fields into one composite comparator, or use a model (Splink supports term-frequency and correlation adjustments) that accounts for the dependency.
  • Common surnames need frequency-based u. A fixed u for last name treats agreement on Smith as equally rare as agreement on Zebrowski, which is wrong: Smith agrees by chance far more often. Use value-specific u-probabilities derived from population frequencies so common surnames earn a smaller agreement weight and do not push unrelated people over the upper threshold.
  • Threshold tuning trades false merges against false splits — and they are not equal. Lowering the upper threshold catches more true matches (fewer false splits) but risks fusing two distinct patients (a false merge), which can attach one person’s diagnoses, allergies, or medications to another and cause direct clinical harm. Bias thresholds toward avoiding false merges and route the ambiguous middle to human review rather than auto-linking it.

The clerical-review band is where PHI meets HIPAA’s minimum-necessary standard. Reviewers adjudicating a pair need only the fields required to decide identity — name, DOB, partial identifiers, address — not the full clinical record of either patient.

Probabilistic linkage is the layer that catches the records deterministic keys drop; treated as a tested, threshold-governed, audited component, it raises match rates without trading away the safety of the master patient index.