Singling-out, linkability and inference risk

These three risks are where synthetic data validation meets data-protection law. European guidance on anonymisation — the Article 29 Working Party's opinion on anonymisation techniques, still the reference point for regulators — defines effective anonymisation by three tests: can an individual still be singled out, can records still be linked, and can information about an individual still be inferred? Data that passes all three is anonymous; data that fails any of them may still be personal data.

For synthetic data those questions can be operationalised as attacks and measured, most commonly with the open-source Anonymeter framework. This page explains what each risk means, how it is measured, what "naive" versus "residual" risk is, and why the distinction matters more than the number.

The three risks

Singling-out. Can an attacker, using the synthetic data, isolate a real individual? Concretely: find a combination of attribute values that identifies exactly one record in the real data. The attack tries to construct such predicates from the synthetic data — "the only person over 60 in postcode X with occupation Y" — and checks how often they single out a real person. Success means the synthetic data preserved fingerprints of individuals.

Linkability. Can an attacker use the synthetic data to link two records that belong to the same real person? Given two datasets holding different attributes about the same population, the attack uses the synthetic data as a bridge to reconnect them. Success means the synthetic data preserved joint structure precisely enough to re-associate split records.

Inference. Can an attacker use the synthetic data to learn a sensitive attribute of a real individual from their other attributes? Given a real person's known attributes (quasi-identifiers), the attack uses the synthetic data to predict the secret — a diagnosis, an income band — and checks accuracy on real records. Success means the synthetic data leaks attribute values, not just presence.

The three are complementary. Singling-out is about identifiability, linkability about re-association, inference about disclosure. A dataset can pass one and fail another, and a DPIA will typically want all three addressed.

How they're measured

Anonymeter's approach — and the pattern any careful implementation follows — is to compare the attack's success on the synthetic data against a baseline attack that doesn't use the synthetic data at all, and, where possible, against a control group of real records the generator never saw.

For each risk:

1. Run the attack using the synthetic data against a set of real target records. Record the success rate.
2. Run a baseline attack that guesses without the synthetic data (random predicates, random links, marginal-frequency guesses). This calibrates how much of the success is just the population's structure.
3. Where a holdout exists, run the attack against control records — real records that were not in the generator's training set. The gap between success on training records and success on control records is the residual risk: the leakage attributable to the generator having seen those specific individuals.
4. Report the risk with a confidence interval. Attack success rates are estimates; the interval width matters, especially on small datasets.

Risk is expressed on a 0–1 scale, where 0 means the synthetic data gives the attacker no advantage over the baseline and values approaching 1 mean it hands the attacker the answer.

Naive risk versus residual risk

This is the distinction that determines whether a number means anything.

Naive risk is the attack's success using the synthetic data against training records, calibrated only against the no-data baseline. It answers: how much does the synthetic data help an attacker, compared to guessing? It can be computed from a synthetic dataset and a real sample alone.

Residual risk additionally subtracts what the attack achieves against control records — real people the generator never saw. It answers: how much of that help came from the generator memorising these individuals, as opposed to learning the population well? It requires a holdout.

Why the difference matters: a good generator that learned the population faithfully will show non-trivial naive inference risk simply because the population is predictable — knowing someone's occupation and postcode genuinely helps predict their income, in real data and in faithful synthetic data alike. That is not a privacy failure; it is the data being useful. Only the residual risk isolates the part that is a failure: the extra advantage the attacker gets on the specific individuals who were in training.

So without a holdout, naive risk is an upper bound. It may overstate the problem for a good generator and it cannot distinguish "learned the population" from "copied the individuals." A report that shows naive risk should label it as such. A report that shows naive risk and calls it the privacy risk is overstating what it measured. And a report that computes a "control" by splitting the real sample in half has produced a number with no meaning at all — both halves were in training, the contrast is empty, and every dataset looks safe. See why privacy attacks on synthetic data need a holdout.

What a bad score usually means

High singling-out risk almost always traces to a few columns with high cardinality or rare combinations — postcode, exact dates, unusual categorical values — preserved too faithfully. The finding is column-level: which combinations do the successful predicates use? The remedy is to generalise those columns (postcode district instead of full postcode, year instead of date) before generation, and it usually costs little utility.

High linkability risk points to joint structure that is too exact. It is more common with generators that preserve multivariate detail well — a virtue for utility, a risk here — and on datasets with several columns that together approach uniqueness. Same remedy family: coarsen the columns that make records nearly unique.

High inference risk, naive only is often not a problem — see above. Check the residual value with a holdout before acting.

High inference risk, residual means the generator memorised individuals' secrets. This is a real leak. It is typically accompanied by a bad membership inference result and a suspicious lower tail in distance to closest record. Remedy: regularise or stop earlier, tighten the privacy budget if the generator has one, and re-test.

Reading these alongside the other pillars

These three metrics are the ones a compliance reader will look for by name, because they map onto the criteria a DPIA has to address. That makes two things especially important. First, the report must say which regime the numbers were computed under — naive or residual — because a DPO reading a naive number as a residual one will draw the wrong conclusion in the direction of false confidence. Second, they are evidence for an anonymisation assessment, not the assessment itself. Whether a given residual risk is acceptable depends on context, on the other safeguards in place, and on legal judgement that no metric provides. A validation report should say so plainly.

Worked example

UCI Adult census, 4,200 rows × 15 columns, SDV Gaussian Copula, metrics engine 0.6.0, run twice — once without a holdout and once with an 1,800-row one. Privacy pillar 98.5 in the holdout run.

Risk	Naive (no holdout)	95% CI	Residual (with holdout)	95% CI
Singling out	0.139	0.109 – 0.169	0.080	0.041 – 0.118
Linkability	0.016	0.006 – 0.026	0.006	0.000 – 0.015
Attribute inference	0.237	0.435 – 0.522 *	0.009	0.000 – 0.159 *

* Inference is reported as the mean across columns, and its interval belongs to the worst column — workclass in the naive run, relationship in the residual one — so the interval deliberately does not bracket the mean. The worst column is the number a reader should act on.

Every residual figure is below its naive counterpart, which is what the regimes mean: naive risk counts attacks that would have succeeded against any real record, so it over-reports on purpose. Read the naive column as a ceiling.

The same two runs against the memorising control:

Risk	Naive	Residual	95% CI (residual)
Singling out	0.996	0.996	0.992 – 1.000
Linkability	0.820	0.814	0.779 – 0.848
Attribute inference	0.992	0.984	0.991 – 1.000 *

Here residual risk does not drop, and that is the entire point of the distinction. When a leak is real, controlling for what an attacker could have guessed anyway removes almost nothing — singling out stays at 0.996 either way. A high naive risk that collapses under a holdout was an artefact of the data; one that survives is exposure attributable to the synthetic data itself.

The intervals matter as much as the values. These attacks sample, so a bare point estimate implies a precision the method does not have — the singling-out figure for this dataset moves by a few thousandths between runs on identical input. An interval of 0.041–0.118 says what the measurement actually supports.

On drivers, the findings name the columns the successful singling-out queries actually used: native-country, hours-per-week, fnlwgt, relationship for the SDV run, and fnlwgt, capital-gain, hours-per-week, capital-loss for the control. fnlwgt and hours-per-week appear in both — high-cardinality numeric columns are what make a census record unique, and coarsening them is the generic remedy. Linkability names the column split it found most linkable rather than the balanced one it starts from.

How SynthProof reports it

SynthProof runs singling-out, linkability and inference attacks with confidence intervals and always states the regime on the report. With a holdout, the three are reported as residual risk. Without one, they are reported as naive risk — upper bound, in those words, and membership inference is shown as not tested. Findings name the column combinations that drove successful singling-out and linkability attacks, with the general remedy of generalising them before regeneration. In the Compliance tier the same results are additionally presented in the vocabulary of the Article 29 criteria for use as evidence in a DPIA — as evidence for the assessment, never as a legal determination that the data is anonymous or "GDPR compliant."

Related metrics

Membership inference is the sibling attack that also needs a holdout and typically fails alongside high residual inference risk. Distance to closest record is the always-available early warning. For the compliance framing: Is synthetic data personal data under GDPR?.

Back to the guide: Synthetic data validation: the complete guide.