Synthetic data validation: the complete guide

How to validate synthetic data across fidelity, privacy and utility — what each metric measures, why one can't compensate for another, and how to read a report.

Updated 16 August 2026 · Figures from metrics engine 0.4.0

Synthetic data is only useful if you can show two things about it: that it behaves like the real data it stands in for, and that it doesn't quietly carry the real data along with it. Validation is how you show that. This guide explains what synthetic data validation actually measures, why the three pillars — fidelity, privacy and utility — have to be assessed together, and how to read a validation report without being misled by a single good number.

It is written for the people who have to sign off on synthetic data in practice: data engineers evaluating a generator, ML teams deciding whether to train on synthetic records, and privacy or compliance staff who need evidence rather than assurances. It is vendor-neutral. The methods here apply to any generator's output.

What "validation" means here

A quick disambiguation, because the phrase is overloaded. Some sources use "synthetic data validation" to mean checking that synthetic survey responses or simulated audiences behave plausibly. That is a different problem. This guide is about tabular synthetic data — rows and columns generated to substitute for a sensitive or scarce real dataset — and the question of whether a given synthetic dataset is fit to be used in place of the real one.

Validation is also not the same as generation quality metrics that a generator prints about its own output. Those can be useful, but they are produced by the same system that produced the data. Independent validation means measuring the output on its own terms, with the same tests applied to every generator, so results are comparable and not self-graded.

The three pillars

Every serious framework — academic and industrial — converges on the same three questions. They are usually called fidelity, privacy and utility, and the essential point about them is that they pull against each other. A dataset can score well on any two while failing the third, and the failure is exactly the one you were most worried about.

Fidelity: does it look like the real data?

Fidelity asks whether the synthetic dataset has the same statistical shape as the real one. It is measured at three levels of increasing subtlety.

At the level of single columns, you compare distributions: for a numeric column, a Kolmogorov–Smirnov statistic between the real and synthetic distributions; for a categorical column, the total variation distance between category frequencies. A generator that produces the right marginals gets full marks here, and most modern generators do.

At the level of pairs of columns, you compare relationships: is the correlation between age and income in the synthetic data close to the real one? Are the conditional frequencies of product_category given region preserved? This is where weaker generators start to fail — they get every column right in isolation and lose the structure between them.

At the level of the whole table, you can use joint-distribution measures or a discriminator test: train a classifier to tell real rows from synthetic ones and see how well it does. Near-chance performance means the tables are statistically indistinguishable.

Fidelity failures have characteristic causes: collapsed correlations (dependencies the generator didn't learn), invented correlations (structure that isn't there in the real data — a finding, not noise), and mode dropping (rare categories or tails simply missing). A good report doesn't just give you a fidelity score; it tells you which dependencies collapsed.

Privacy: does it leak the real data?

Privacy asks whether the synthetic dataset reveals anything about specific real individuals. This is the pillar most often misunderstood, because it is tempting to reason "the data is synthetic, so it must be private." It isn't. A generator that memorises its training data produces synthetic rows that are near-copies of real ones, with perfect fidelity and no privacy at all.

Privacy is measured through attacks — attempts to extract real information from the synthetic data — and the score reflects how badly the attacks fail. The standard battery:

Exact-copy and near-copy detection. How many synthetic rows are identical to, or within a tiny distance of, a real row? This catches memorisation directly. It needs no special setup and should always run.

Distance to closest record (DCR) and nearest-neighbour distance ratio (NNDR). For each synthetic row, how far is the nearest real row — and how does that compare to how far real rows are from each other? Synthetic rows that sit much closer to real ones than real ones sit to their own neighbours are suspicious.

Singling-out, linkability and inference risk. These are the three criteria that European data-protection guidance uses to define anonymisation, and tools like Anonymeter operationalise them as attacks: can an attacker isolate an individual using the synthetic data? Link records across datasets? Infer a sensitive attribute of a real person? Each yields a risk estimate with a confidence interval.

Membership inference. Given a real record, can an attacker tell whether it was in the generator's training set? This is the canonical privacy attack in machine learning and it applies directly to synthetic data.

There is a subtlety here that most guides skip and that changes how you should read any privacy score. See the holdout section below.

Utility: is it useful for the job?

Utility asks the practical question: if I use the synthetic data instead of the real data, do I get the same answers? The standard method is train on synthetic, test on real (TSTR): train a model on the synthetic dataset, evaluate it on held-out real data, and compare against a model trained on real data (TRTR). If a model learned from synthetic rows performs almost as well on real rows as one learned from real rows, the synthetic data captured what mattered for that task.

Utility is task-dependent, which is both its strength and its limitation. A dataset can have excellent utility for predicting default and poor utility for predicting churn, because the generator preserved the structure that drives one and not the other. Good validation runs TSTR across several plausible target columns rather than one, and reports per-target results, so a bad score can be traced to the specific class or relationship that was lost. Under-represented minority classes are the classic culprit: the synthetic data has 0.4% fraud cases where the real data has 2%, and every fraud model trained on it is blind.

For a full treatment see Train on synthetic, test on real (TSTR).

Why one pillar can't compensate for another

This is the single most important thing to understand about reading a validation report. The three scores are not interchangeable and must not be averaged into a single number without safeguards.

Consider the extremes. A generator that outputs the real data with a few rows shuffled has perfect fidelity and perfect utility — and zero privacy. A generator that outputs uniform noise has decent privacy and no fidelity or utility. A generator that reproduces every marginal distribution exactly but independently has good column-level fidelity, terrible pairwise fidelity, and utility that collapses for any task depending on relationships between columns.

A well-designed report therefore does two things. It shows all three pillars separately, so you can see the shape of the trade-off the generator made. And it refuses to let one pillar rescue another: a critically low score on any pillar caps the overall grade, regardless of how good the others are. If a report gives a leaky dataset an "A" because its statistics are beautiful, the report is wrong.

The three modes of validation

What you can measure depends on what you can upload. There are three practical modes, and an honest report says which one it ran in.

Synthetic only. You have the synthetic dataset and nothing else. You can check internal consistency, plausibility of distributions, duplicates and outliers, and column-level statistics. You cannot measure fidelity (compared to what?), you cannot run privacy attacks (against which real records?), and you cannot run TSTR. This mode produces a partial report and should not issue a headline score; a partial report that reads like a full one is misleading.

Synthetic plus a real sample. You upload the synthetic dataset and a representative sample of the real data with the same schema. All three pillars can now run: fidelity is comparative, privacy attacks test leakage against the real sample, and TSTR trains on synthetic and tests on real. This is the standard mode. Note that a sample is enough — the metrics are statistical and stabilise well below full-dataset size — and that a serious validation service deletes the real sample the moment the job completes.

Synthetic plus a real sample plus a holdout. The strongest mode, and the one that makes privacy scores fully meaningful, for the reason explained next.

The holdout problem

Attack-based privacy metrics — singling-out, linkability, inference, membership inference — all work by comparing how the synthetic data behaves toward records the generator saw versus records it didn't. If synthetic rows sit suspiciously close to training records but not to unseen ones, the generator leaked. If they sit equally close to both, whatever similarity exists is just the population's structure, not memorisation.

That comparison requires a control group: real records that were never in the generator's training set. And here is the trap. You cannot manufacture a control group by splitting the real sample you have in half. Both halves stand in the same relation to the generator — either both were in training or both were not — so the attacks have no contrast to work with, every risk estimate drifts toward zero, and every dataset appears safe. A validation tool that silently does this will certify leaky data as private.

The honest approach is to treat the holdout as an optional extra upload and to be explicit about the consequences:

With a holdout, singling-out, linkability and inference are reported as residual risk — the risk beyond what the population's own structure explains — and membership inference can run at all.
Without one, those three are reported as naive risk, labelled as an upper bound, and membership inference is reported as not tested. Not "passed". Not tested.

Exact-copy detection and nearest-neighbour distance need no control group and always run. If you are evaluating a validation tool or a generator's built-in privacy report, this is the question to ask: what did you use as the control group, and if there wasn't one, is that stated on the report?

How to read a validation report

Whatever tool produced it, a trustworthy report has some recognisable properties. Use this as a checklist.

Which mode did it run in? If it doesn't say, assume the weakest. A headline score on a synthetic-only run is a red flag.

Are the three pillars shown separately, and does a critical failure cap the grade? If you see one aggregate number and nothing else, you can't tell what the generator traded away.

Is "not tested" distinguished from "passed"? A metric that couldn't run should say so, with the reason. Silence is not a pass.

Is the report bound to the data? A cryptographic fingerprint (SHA-256) of the exact files measured, printed on the report, means the scores can't be waved at a different dataset later.

Is it versioned? Metrics evolve. A report should state the engine version (and, if it benchmarks against other datasets, the benchmark snapshot version) so a score from six months ago can be interpreted correctly.

Does it tell you what to fix? Scores alone are a dead end. Good reports include findings: which correlations collapsed, which column combinations drive singling-out risk, which minority class was under-represented — with a general remedy, not a tuning recipe for one particular generator.

What happened to your real data? If you uploaded a real sample, the report should state when it was deleted. If the tool retains it, you should know that before you upload.

Where benchmarking fits

A score of 78 on fidelity is hard to interpret in isolation. Is that good for a financial transactions dataset? Typical? Poor? Percentile benchmarking — how this dataset scores relative to others of the same category that have been validated with the same engine — turns an absolute number into a relative one. It is most credible when the benchmark pool is versioned and frozen (so two reports on the same day are comparable), the categories are coarse and honest, and the pool holds only scores and dataset shape metadata, never the data itself. Treat percentile claims sceptically if a tool can't tell you how the pool was built.

A worked example

Take a public dataset — the UCI Adult census data — and a synthetic version produced by a common open-source generator. Run the three pillars.

Fidelity scores 84.8/100: the marginals hold up, but the weakest pairwise relationship — education against education-num — retains only 17.9 of 100 on contingency similarity, and nine other categorical pairs sit below the reporting threshold. Privacy scores 98.4/100, and because the run supplied an 1,800-row holdout alongside the real sample, those are residual-risk figures rather than upper bounds: membership inference actually ran and found no signal, scoring 100/100. Utility scores 35.1/100, the weakest pillar by far, driven by native-country, where a model trained on the synthetic data lands below chance. The overall grade is B at 75.3/100 — no critical floor applied, because 35.1 sits above the 10-point floor for utility, and the weighted average is left to express a weak-but-not-disqualifying result. This is the shape of a competent generator: statistically close, privacy-safe on this evidence, and genuinely lossy for some downstream tasks.

You can inspect the full report — every metric, every finding, hash-bound and versioned — at /sample. It is a real run, published unedited, including the pillar the generator did poorly on.