Why privacy attacks on synthetic data need a holdout

There is a quiet flaw in a great deal of synthetic data privacy reporting, and it comes down to a single question that most reports don't answer: what was the control group?

Attack-based privacy metrics — membership inference, and the residual versions of singling-out, linkability and inference risk — work by contrast. They compare how the synthetic data behaves toward real records the generator was trained on against how it behaves toward real records the generator never saw. That second group is the holdout, and without it the attacks have nothing to compare against. They don't fail loudly. They return reassuring numbers that mean nothing.

This page explains the mechanism, the shortcut that doesn't work, what an honest report does instead, and what teams generating synthetic data should change in their process. It is written to be linked to when someone asks "why is there a third file?"

The mechanism

A generator that has learned a population well produces synthetic rows that resemble typical members of that population. A generator that has memorised its training set produces synthetic rows that resemble specific training records. From the outside, both look plausible. The way to tell them apart is to ask whether the synthetic data is closer to the individuals it was trained on than to comparable individuals it wasn't.

That is a comparison between two groups of real records:

• Members — records that were in the generator's training data.
• Non-members — records from the same population that were not in the training data. The holdout.

If synthetic rows sit noticeably closer to members than to non-members, the generator leaked something about the specific individuals it saw. If they sit equally close to both, whatever similarity exists is the population's own structure — the generator learned, rather than copied.

Every attack-based privacy metric is a version of this comparison. Membership inference is the comparison stated directly: can an attacker tell members from non-members? Singling-out, linkability and inference use the comparison to separate residual risk — the extra advantage an attacker gets on training individuals — from naive risk, which includes the population's ordinary predictability. In every case the holdout is what supplies the contrast.

The shortcut that doesn't work

The tempting fix, when only one real sample is available, is to split it in half and call one half the control group. Half "members," half "non-members." Run the attacks.

The problem is that the split happens after generation. If the generator was trained on the full real dataset — or on a sample of it that overlaps with what was uploaded — then both halves were seen. Both are members. The attack compares members to other members, finds no difference, and reports chance-level success. The dataset appears private.

It appears private regardless of how badly it leaks. A generator that memorised every training record will produce synthetic rows that sit equally close to both halves — because both halves are training records — and the split-sample attack will pass it. The number produced is not a low-privacy-risk estimate. It is a measurement of nothing, formatted to look like a pass.

This is worth stating plainly because the failure is silent. There is no error. The metric runs, returns a value in the expected range, and the report shows a green result. A team reading it has no way to know the control was empty unless the report says how the control was constructed — and most don't.

What an honest report does instead

There are two, and only two, honest outcomes when a holdout is unavailable.

Report the attack-based metrics as naive risk, labelled as an upper bound. Naive singling-out, linkability and inference risk can be computed with the synthetic data and a real sample alone — the attack calibrated against a no-data baseline, but not against a control group. The number is real; it just conflates "learned the population" with "copied the individuals" and therefore overstates the risk for a good generator. Reported honestly, as an upper bound, it is still useful: a low naive risk is genuinely reassuring, and a high one says "get a holdout and find out."

Report membership inference as not tested. Membership inference has no naive form. It is the member/non-member contrast; without non-members it cannot run. The honest report says "not tested — no holdout provided." Not "passed." Not a green tick with a footnote. A metric that could not run and a metric that ran and passed are different facts, and privacy is the pillar where conflating them does the most damage.

And when a holdout is provided, the same report shows the attack-based metrics as residual risk and runs membership inference properly. Which regime the numbers were computed under should be stated on the report itself, in the same place as the numbers, so a reader — a DPO, an auditor, a customer — cannot mistake one for the other.

Meanwhile, the distance-based checks — exact-copy detection, distance to closest record, nearest-neighbour distance ratio — need no control group and always run. They catch copying and near-copying reliably. What they don't catch is subtler leakage, and that is exactly the gap the attack-based metrics exist to fill.

What this means for teams generating synthetic data

The practical consequence is a small change to process with a large payoff: hold out a slice of real data before training the generator, and keep it.

How much. Enough for the attacks to have statistical power — a few thousand rows is usually plenty; a few hundred is thin. A few percent of a large dataset, more of a small one. It costs the generator little; the population is still well represented in what remains.

How to hold it out. Randomly, before generation, from the same population as the training data. Not a different time period, not a different segment — the point is that non-members are comparable to members in every way except having been seen. A holdout drawn from a different distribution introduces a contrast that isn't about memorisation and biases the attacks.

What to keep. The holdout itself, with the same schema as the training data, stored with the same care as the training data — it is real data. And a note of the split, so it can be reproduced. When validating, you upload three files: the synthetic data, a sample of the training data, and the holdout.

When you can't. Sometimes the generator was trained by someone else, or long ago, and no holdout exists. Then you are in the naive-risk regime, and the useful move is to read the naive numbers as the upper bound they are, get the distance-based checks right, and — for anything that has to go in front of a regulator — say honestly that membership inference was not tested and why. Or hold out from the next generation run.

The question to ask

If you are evaluating a synthetic data validation tool, or reading a generator's built-in privacy report, or receiving a report from a vendor, one question separates measurement from decoration:

What was the control group, and if there wasn't one, does the report say so?

Good answers: "a real holdout the generator never saw," or "none, so attack risks are shown as an upper bound and membership inference as not tested." A bad answer is any privacy score with membership inference passing and no mention of where the non-members came from. If the answer is "we split your sample," the privacy numbers on that report should be disregarded.

How SynthProof handles it

SynthProof treats the holdout as an optional third upload, and the report states the regime it ran in next to the privacy numbers. With a holdout: attack-based metrics as residual risk, membership inference run and reported. Without: attack-based metrics as naive risk labelled upper bound, membership inference shown as not tested — no holdout provided, never as passed. Distance-based checks run in every mode with a real sample. A synthetic-only run issues no privacy score at all. The engine will not construct a control group from the uploaded sample, because doing so would produce a number that looks like a pass and isn't.

Related

Membership inference · Singling-out, linkability and inference · Distance to closest record

Back to the guide: Synthetic data validation: the complete guide.