Removing the 18 HIPAA Identifiers in Python

Safe Harbor de-identification succeeds or fails on completeness: strip 17 of the 18 identifier categories perfectly and the record is still Protected Health Information, because the one you missed re-identifies the patient. This page is the runnable companion to Safe Harbor vs. Expert Determination — where that guide weighs the two §164.514 methods against each other, here we build a concrete, testable Python scrubber that removes all 18 categories from a clinical record and then proves nothing leaked before the record is allowed to leave the boundary. The design principle throughout is deny-by-default: a field is emitted only if a policy explicitly permits it, and the whole record is rejected if any unrecognized field appears.

The 18 categories in §164.514(b)(2) are not all handled the same way. Some are dropped outright, some are coarsened so a residual clinical signal survives, and a few are replaced with a keyed pseudonym so downstream joins still work without carrying the original value. The quick-reference table below is the lookup artifact to keep next to your policy definition — it maps each category to its removal action and the structured field it usually arrives in.

Categories 8 and 9 are pseudonymized rather than dropped because a record-level analytics warehouse still needs a stable, non-reversible key to link a patient’s observations over time; an HMAC token gives you that linkage without retaining the original medical record number. Everything else that uniquely names a person is dropped. Category 18 is the dangerous one — free-text notes can contain any of the other 17 as unstructured strings, so it gets its own regex scrubbing stage.

Implementation pattern

Start with a declarative policy. Modelling the removal action as an enum and mapping every known field to one keeps the rules auditable — a compliance reviewer reads the policy table, not the control flow. Crucially, any field not present in the policy is a hard error, so a new column added to an upstream extract cannot silently pass through un-scrubbed.

import hashlib
import hmac
import re
from datetime import date
from enum import Enum


class Action(str, Enum):
    DROP = "drop"                  # remove the field entirely
    GENERALIZE = "generalize"     # coarsen (ZIP -> 3 digits, date -> year)
    PSEUDONYMIZE = "pseudonymize"  # replace with a keyed, non-reversible token
    SCRUB_TEXT = "scrub_text"     # category 18: free text, regex-masked
    KEEP = "keep"                  # clinical payload, retained verbatim


# The full field-level policy. Comments reference the Safe Harbor category.
# A field missing from this map is rejected before anything is emitted.
POLICY = {
    "patient_name":  Action.DROP,          # 1  names
    "street_address": Action.DROP,         # 2  geo < state
    "city":          Action.DROP,          # 2
    "zip":           Action.GENERALIZE,    # 2  keep 3 digits unless low-population
    "birth_date":    Action.GENERALIZE,    # 3  year only, aggregate ages > 89
    "admit_ts":      Action.GENERALIZE,    # 3  year only (day-level shift is separate)
    "phone":         Action.DROP,          # 4/5 telephone & fax
    "email":         Action.DROP,          # 6
    "ssn":           Action.DROP,          # 7
    "mrn":           Action.PSEUDONYMIZE,  # 8  medical record number
    "member_id":     Action.PSEUDONYMIZE,  # 9  health-plan beneficiary
    "account_no":    Action.DROP,          # 10
    "license_no":    Action.DROP,          # 11
    "vin":           Action.DROP,          # 12
    "device_serial": Action.DROP,          # 13
    "url":           Action.DROP,          # 14
    "ip":            Action.DROP,          # 15
    "fingerprint":   Action.DROP,          # 16 biometric
    "photo_uri":     Action.DROP,          # 17 full-face image
    "note":          Action.SCRUB_TEXT,    # 18 free text -> scrub_free_text()
    "loinc_code":    Action.KEEP,          # clinical payload, deliberately retained
    "value":         Action.KEEP,
    "unit":          Action.KEEP,
}

Category 2 (geography) and category 3 (dates) need generalizers rather than deletion, because a de-identified dataset is worthless if you strip every temporal and spatial signal. For ZIP codes, Safe Harbor lets you keep the first three digits — unless those three digits cover a population of 20,000 or fewer, in which case HHS requires them zeroed. That restricted set is fixed and small enough to hard-code.

# Three-digit ZIP prefixes whose combined population is <= 20,000; HHS
# requires these to be replaced with "000" under Safe Harbor 2(b).
RESTRICTED_ZIP3 = {
    "036", "059", "063", "102", "203", "556", "692", "790",
    "821", "823", "830", "831", "878", "879", "884", "890", "893",
}


def generalize_zip(zip_code: str) -> str:
    z3 = zip_code[:3]
    return "000" if z3 in RESTRICTED_ZIP3 else z3


def generalize_birth_year(iso_date: str, today: date | None = None) -> str:
    """Year only; anyone over 89 is collapsed into a single 90+ bucket."""
    today = today or date.today()
    year = int(iso_date[:4])
    if today.year - year > 89:      # age exceeds 89 -> no discriminating year
        return "1935-or-earlier"
    return str(year)


def pseudonymize(value: str, *, key: bytes) -> str:
    """Keyed, non-reversible token so joins survive without the raw ID."""
    return hmac.new(key, value.encode(), hashlib.sha256).hexdigest()[:16]

The pseudonym is a keyed HMAC-SHA256 digest, not a bare hash: a bare sha256("MRN-88213") is trivially reversed by a dictionary attack against a small identifier space, whereas an HMAC with a secret key held on the compliance side is not. The same key must be used across runs so a patient’s records join consistently.

Category 18 is where most Safe Harbor implementations leak. A free-text clinical note can contain a phone number, an SSN, a calendar date, a URL, or an IP address as plain text that no structured-field policy will ever see. The scrubber runs an ordered set of regexes over the note and masks each match with a category tag.

# Ordered so broad patterns run before narrow overlaps can misfire.
PATTERNS = {
    "PHONE": re.compile(r"\b(?:\+?1[-.\s]?)?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}\b"),
    "SSN":   re.compile(r"\b\d{3}-\d{2}-\d{4}\b"),
    "EMAIL": re.compile(r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b"),
    "URL":   re.compile(r"\bhttps?://\S+"),
    "DATE":  re.compile(r"\b\d{1,2}/\d{1,2}/\d{2,4}\b|\b\d{4}-\d{2}-\d{2}\b"),
    "IP":    re.compile(r"\b(?:\d{1,3}\.){3}\d{1,3}\b"),
}


def scrub_free_text(text: str) -> str:
    for tag, pattern in PATTERNS.items():
        text = pattern.sub(f"[{tag}]", text)
    return text

Now the orchestrator ties the policy, the generalizers, and the scrubber together, and — this is the load-bearing part — asserts the output against a strict allowlist before returning. The allowlist is the operational form of the §164.514(b)(2) “no actual knowledge” standard: you are declaring, field by field, exactly what may exit the boundary, and anything else aborts the release.

GENERALIZERS = {
    "zip": generalize_zip,
    "birth_date": generalize_birth_year,
    "admit_ts": lambda ts: ts[:4],   # year only; day-level shift is a separate step
}

# The ONLY fields permitted in a released record. Deny-by-default.
ALLOWED_OUTPUT_FIELDS = {
    "zip", "birth_date", "admit_ts",   # generalized
    "mrn", "member_id",                # pseudonymized tokens
    "note",                            # scrubbed free text
    "loinc_code", "value", "unit",     # clinical payload
}


def deidentify(record: dict, *, key: bytes) -> dict:
    out: dict = {}
    for field, value in record.items():
        action = POLICY.get(field)
        if action is None:
            raise ValueError(f"unmapped field {field!r} -- refusing to emit")
        if action is Action.DROP:
            continue
        if action is Action.KEEP:
            out[field] = value
        elif action is Action.GENERALIZE:
            out[field] = GENERALIZERS[field](value)
        elif action is Action.PSEUDONYMIZE:
            out[field] = pseudonymize(str(value), key=key)
        elif action is Action.SCRUB_TEXT:
            out[field] = scrub_free_text(value)

    leaked = set(out) - ALLOWED_OUTPUT_FIELDS
    assert not leaked, f"identifier leak: {sorted(leaked)}"
    return out

The assert is deliberately the last line. Even if a future edit adds a policy rule that keeps a field it should have dropped, the allowlist catches it and the record never ships. Two independent controls — the policy and the allowlist — must both agree before any field is released.

Validation and testing

Prove the scrubber with a golden record: one input that carries every identifier class, and a test that asserts none of the original values survive in the output. This is the fixture that fails CI the moment someone weakens a regex or loosens the policy.

import pytest

GOLDEN_INPUT = {
    "patient_name": "Jane Q. Doe",
    "street_address": "742 Evergreen Terrace",
    "city": "Springfield",
    "zip": "03601",                      # 036 is a restricted low-pop prefix
    "birth_date": "1948-03-11",
    "admit_ts": "2023-04-12T09:30:00",
    "phone": "603-555-0142",
    "email": "jane.doe@example.com",
    "ssn": "078-05-1120",
    "mrn": "MRN-88213",
    "member_id": "HP-4491203",
    "url": "https://portal.example.org/jdoe",
    "ip": "10.14.22.9",
    "note": ("Call patient at 603-555-0142 or jane.doe@example.com "
             "re: 04/12/2023 visit; SSN on file 078-05-1120."),
    "loinc_code": "4548-4",
    "value": "6.8",
    "unit": "%",
}

FORBIDDEN = [
    "Doe", "Evergreen", "Springfield", "603-555-0142", "jane.doe@example.com",
    "078-05-1120", "MRN-88213", "HP-4491203", "portal.example.org",
    "10.14.22.9", "04/12/2023",
]


def test_no_identifier_survives():
    out = deidentify(GOLDEN_INPUT, key=b"rotate-me-in-kms")
    blob = repr(out)
    for token in FORBIDDEN:
        assert token not in blob, f"leaked {token!r}"
    # 036 is low-population -> zeroed; not just truncated to three digits.
    assert out["zip"] == "000"
    # Age under 90 -> year retained; note fully masked.
    assert out["birth_date"] == "1948"
    assert "[PHONE]" in out["note"] and "[SSN]" in out["note"]


def test_unmapped_field_is_rejected():
    with pytest.raises(ValueError):
        deidentify({"genome_hash": "abc"}, key=b"k")

The first test is a completeness proof, not a spot check: it scans the entire serialized output for every raw value, so a scrubber that drops the structured phone but leaves the same number embedded in note still fails. The second test locks in the deny-by-default contract — a novel identifier field the policy has never seen cannot slip through. Run both against a corpus of real (internally held) notes, because free-text formats vary far more than structured fields, and every new format is a new regex gap.

Gotchas & compliance constraints

• Category 18 free text is the silent leak. Structured-field scrubbing is easy to get right and easy to over-trust. The identifiers that actually escape live in narrative notes, Observation.note, and comment fields — a phone number typed into a nursing note is still PHI. Treat every free-text field as SCRUB_TEXT, keep the regex corpus under test, and remember that regex masking is best-effort: a misspelled or obfuscated identifier can slip past, which is exactly the residual-risk gap that pushes higher-stakes releases toward Expert Determination.
• Dates need generalization, not deletion — but coarsening to the year destroys clinical sequence. If two events in the same year must keep their relative ordering (a lab draw before an admission), truncating both to 2023 collapses the interval you needed. Use consistent per-patient date-shifting instead, as covered in date-shifting clinical timestamps for de-identification; the admit_ts generalizer here is the minimal fallback, not the preferred path when temporal analysis matters.
• Low-population ZIPs and rare quasi-identifier combinations re-identify. Keeping ZIP3 plus year of birth plus a rare diagnosis can single out one person even when every category is technically “removed.” The RESTRICTED_ZIP3 set handles the population floor Safe Harbor names explicitly, but the deeper protection is testing the released set for small equivalence classes — see implementing k-anonymity for clinical datasets.
• The §164.514(b)(2) “no actual knowledge” standard is a human obligation, not just code. Safe Harbor is only satisfied if the covered entity has no actual knowledge that the residual information could re-identify an individual. If an analyst knows a pseudonymized cohort is a single rare-disease clinic, running the scrubber does not make the release compliant. The allowlist assert enforces the mechanical part; a documented risk review must cover the knowledge part.

Related

• Date-shifting clinical timestamps for de-identification — how to keep temporal sequence while satisfying category 3
• Implementing k-anonymity for clinical datasets — catching re-identification from rare quasi-identifier combinations the 18 categories miss
• Safe Harbor vs. Expert Determination — parent guide weighing the two §164.514 de-identification methods
• HIPAA de-identification & patient matching — the surrounding compliance and identity-resolution topic area