Membership inference

Membership inference is the canonical privacy attack in machine learning, and it applies directly to synthetic data. It asks a specific, uncomfortable question: given a real record, can an attacker tell whether that record was in the data used to train the generator?

If the answer is yes, the synthetic data leaks. Not the values — the presence. Knowing that a particular person's record was in a hospital's training set, or a lender's, is itself sensitive information, and it is exactly the kind of leak that statistical-similarity metrics never notice.

Membership inference belongs to the privacy pillar of synthetic data validation. It is the metric that most reliably catches a memorising generator, and it is also the metric most often skipped — for a structural reason this page explains.

Definition

An attacker has the synthetic dataset and a candidate real record. They must decide: was this record in the generator's training set (a member), or not (a non-member)?

The attack exploits a simple tendency. Generators fit their training data more closely than the population at large, so synthetic rows tend to sit nearer to training records than to comparable records the generator never saw. An attacker who can measure "how close is this candidate to the synthetic data" therefore has a signal about membership.

The metric is the attacker's success rate above chance. Perfect privacy: the attacker does no better than a coin flip (AUC 0.5). Serious leakage: the attacker distinguishes members from non-members with high accuracy. In between, the AUC — or an equivalent like advantage over chance — quantifies how much presence information the synthetic data gives away.

How it's computed

The standard black-box procedure — the attacker sees only the synthetic data, not the generator — runs like this:

1. Assemble candidates — a set of real records known to have been in training (members) and a set known not to have been (non-members, the holdout).
2. Score each candidate by its relationship to the synthetic data. The simplest scorer is nearest-neighbour distance: how far is this candidate from the closest synthetic row? Stronger scorers train a classifier on the synthetic data and use its confidence, or estimate density.
3. Threshold and evaluate. Sweep the threshold on the score, predicting "member" below it, and compute how well the predictions match the truth. Report AUC (or precision at a chosen false-positive rate).

Step 1 is the crux, and it is why membership inference is different from every other privacy metric on this site.

Why it needs a holdout — and cannot be faked without one

The attack is a contrast between members and non-members. Without genuine non-members — real records the generator never saw — there is nothing to contrast against, and the attack cannot run.

The tempting shortcut is to split whatever real sample you have and call one half "non-members." It does not work. If the generator was trained on the full real dataset and the sample you uploaded is drawn from that same dataset, both halves are members. The attack finds no difference between them, reports chance-level AUC, and the dataset appears perfectly private — no matter how badly it leaks. A validation tool that does this silently will pass a memorising generator every time.

The only honest options are:

• Run it with a real holdout. The user supplies real records that were withheld from generation. Now members and non-members exist, and the attack has power.
• Report it as not tested. If no holdout is available, say so. Not "passed". Not a green tick. A metric that could not run and a metric that ran and passed are different facts, and a report that conflates them is misleading about the one thing privacy readers most need to know.

Practically, teams that generate synthetic data should get into the habit of holding out a slice of real data before training the generator, precisely so this attack — and the related residual-risk versions of singling-out, linkability and inference — can be run. It costs a few percent of training data and buys the only privacy evidence that actually tests memorisation directly. See why privacy attacks on synthetic data need a holdout.

What a bad score usually means

Memorisation. The dominant cause. The generator overfit, synthetic rows cluster tightly around training records, and distance-based scoring separates members from non-members cleanly. This will usually show as a suspicious lower tail in distance to closest record as well; membership inference confirms it and quantifies the exposure.

Small or highly distinctive training data. Rare records — outliers, unusual combinations of attributes — are the easiest to infer membership for, because the generator has little choice but to reproduce something recognisably like them. High AUC concentrated on outlier candidates is a specific finding: the tails leaked even if the bulk didn't.

High-cardinality or quasi-identifier columns. Near-unique combinations (postcode × birth date × occupation) act as fingerprints; a generator that preserves them faithfully makes membership trivially inferable. The remedy is generalisation of those columns before generation, not a different generator.

Insufficient regularisation or differential privacy budget. For generators that support it, membership inference is the practical test of whether the privacy budget was actually adequate — a formal guarantee that still yields high attack AUC means the parameters were too loose for this data.

Remedies are generator-agnostic: regularise or stop earlier, generalise quasi-identifiers, and, if the generator supports a privacy budget, tighten it and re-test. Membership inference is the metric to re-run after each change, because it is the one that measures the thing you changed.

Reading membership inference alongside the other pillars

Membership inference is the counterweight to utility. A memorising generator has near-perfect TSTR — it is training your model on real rows in disguise — and fails membership inference badly. Read together, the two tell you whether high utility was earned by learning the population or borrowed by copying individuals. This is why a critical privacy failure should cap the overall grade regardless of how good utility looks.

It is also the metric whose absence should worry you most. If a validation report shows a privacy score with no membership inference result and no statement of why, ask what was used as the control group. If a generator's built-in privacy report shows membership inference passing, ask the same question — the answer determines whether the pass means anything.

Worked example

UCI Adult census, 4,200 rows × 15 columns, SDV Gaussian Copula, metrics engine 0.6.0, with a 1,800-row holdout supplied — without it this section would not exist.

	Attack AUC	Attacker advantage	Metric score
SDV Gaussian Copula	0.4906	0.00	100.0
Memorising control	1.0000	1.00	0.0

The SDV run sits just below a coin flip. The attacker's distance-based rule separated the 4,200 training records from the 1,800 held-out ones no better than guessing, which is what no membership leakage looks like.

The control is the number worth staring at. AUC 1.0 — every training record correctly identified, every held-out record correctly rejected, no errors. And on the same run utility scored 69.3, with ROC-AUC of 0.98 and 0.99 on its two main targets, beating the honest generator's 0.74 and 0.69. That is the whole argument for this metric in one pair of numbers: the control looks like the better generator on every quality measure and is a republication of the source. Membership inference is what separates them, and since engine 0.6.0 a critically high result is itself a critical finding that caps the grade.

Running the same Adult data without a holdout gives the honest degradation. Membership inference reports not tested — no holdout provided, never a pass. The three attack-based metrics stay available but switch regime, labelled naive risk — upper bound in the report's own unit column: singling out 0.139 (95% CI 0.109–0.169), linkability 0.016 (0.006–0.026), attribute inference 0.237. Compare those with the residual figures measured against the holdout — singling out 0.080, linkability 0.006, inference 0.009 — and the gap between an upper bound and a measurement is visible directly.

How SynthProof reports it

SynthProof runs membership inference only when a holdout is supplied, and reports it as attacker advantage above chance with the scoring method stated. When no holdout is available, the report shows the metric as not tested — no holdout provided, never as passed, and the attack-based Anonymeter metrics are labelled naive risk (upper bound) rather than residual risk. Distance-based checks — exact-copy detection, DCR, NNDR — always run regardless. A critically high membership inference result is a critical finding that caps the overall grade, so no fidelity or utility score can carry a dataset that memorised its training records. When the attack succeeds mainly on outliers or on specific quasi-identifier combinations, the findings say so, with a general remedy.

Related metrics

Distance to closest record is the always-available early warning; membership inference is the confirmation. Singling-out, linkability and inference are the sibling attacks that also gain their full meaning from a holdout. TSTR is the utility metric this one keeps honest.

Back to the guide: Synthetic data validation: the complete guide.