Independent synthetic data validation
Synthetic Data Validation Report
Full assessment — synthetic data against a real sample (Mode 1)
- Job
- sample-adult-census
- Category
- public-sector
- Issued
- 2026-08-15 20:00 UTC
- Synthetic dataset
- 4,200 rows × 15 columns
- Real sample
- 4,200 rows × 15 columns
- Holdout sample
- 1,800 rows × 15 columns
Weighted across privacy (40%), fidelity (30%) and utility (30%). Structural integrity is reported separately as a precondition check.
Fidelity
Statistical similarity to the source data
Overall distributional similarity scores 85/100. Weakest match: capital-gain at 9/100.
| Measurement | Value | Score | What it means |
|---|---|---|---|
| Column shapes | 0.887 score | 89 | How closely each column's marginal distribution matches the source (KS complement for numeric, total-variation complement for categorical). |
| Column pair trends | 0.809 score | 81 | Whether relationships between pairs of columns survived generation — correlations for numeric pairs, contingency for categorical pairs. |
Privacy
Resistance to re-identification attacks
Privacy scores 98/100 (full attack suite). Highest residual risk: singling out at 92/100.
| Measurement | Value | Score | What it means |
|---|---|---|---|
| Exact record copies | 0.00% % of synthetic rows | 100 | Share of synthetic rows that are byte-for-byte copies of a real record. Any non-zero value means real records were reproduced verbatim. |
| Nearest-neighbour distance | 6.911 ratio vs real holdout | 100 | Distance from synthetic records to their closest real record, divided by the same distance for a real holdout. 1.0 or above means synthetic records are no closer to the source than unrelated real records are. |
| Membership inference | 0 attacker advantage | 100 | How much better than guessing an attacker does at deciding whether a given record was in the training data. 0 is a coin flip; 1 is perfect identification. |
| Singling out | 0.08 residual risk | 92 | Whether the synthetic data lets an attacker build a query that isolates exactly one real individual (Article 29 WP criterion 1). Measured against the holdout you supplied, so this is residual risk — the exposure attributable to the synthetic data itself. |
| Linkability | 0.004 residual risk | 100 | Whether two separate datasets about the same people can be joined through the synthetic data (Article 29 WP criterion 2). Measured against the holdout you supplied, so this is residual risk — the exposure attributable to the synthetic data itself. |
| Attribute inference | 0.017 residual risk | 98 | Whether an attacker who knows some attributes of a real person can infer an unknown one from the synthetic data (Article 29 WP criterion 3). Averaged across columns. Measured against the holdout you supplied, so this is residual risk — the exposure attributable to the synthetic data itself. |
Utility
Train on synthetic, test on real
Models trained on the synthetic data retain 35% of the skill of models trained on real data, averaged over 3 target column(s).
| Measurement | Value | Score | What it means |
|---|---|---|---|
| TSTR — class | 0.737 roc_auc | 60 | Model trained on synthetic data scores 0.737 (roc_auc) on held-out real data, against 0.895 for the same model trained on real data (chance: 0.500). |
| TSTR — sex | 0.689 roc_auc | 45 | Model trained on synthetic data scores 0.689 (roc_auc) on held-out real data, against 0.916 for the same model trained on real data (chance: 0.500). |
| TSTR — native-country | 0.04 f1_macro | 0 | Model trained on synthetic data scores 0.040 (f1_macro) on held-out real data, against 0.081 for the same model trained on real data (chance: 0.040). |
Structural integrity
Internal consistency of the synthetic data
Structural integrity scores 99/100.
| Measurement | Value | Score | What it means |
|---|---|---|---|
| Duplicate rows | 0.00% % of rows | 100 | Repeated identical rows inside the synthetic data. A high rate means the generator collapsed onto a few records. |
| Missing values | 0.88% % of cells | 98 | Share of empty cells across the synthetic data. |
| Constant columns | 0.00% % of columns | 100 | Columns where every row holds the same value. These carry no information and usually indicate a generation failure. |
| Extreme values | 0.27% % of numeric cells | 95 | Values more than 4 standard deviations from their column mean — often impossible values invented by the generator. |
| Category collapse | 0.00% % of categorical columns | 100 | Categorical columns where a single value covers more than 95% of rows. Legitimate for genuinely skewed fields, a red flag when it appears across many columns. |
Findings
What the scores above mean in practice: where each weakness sits and the general direction of a fix. Findings are derived from the measurements and shape statistics in this report — not from the data, which is why they survive the deletion recorded in the footer.
Material
1Weaknesses worth acting on before this dataset is relied upon.
1 column distribution diverges from the source
capital-gain
Weakest is capital-gain at 9/100 distribution match; 1 column scores below 70.
What to look at: Per-column drift usually means the generator under-fitted these marginals — common with heavy tails, rare categories, and columns whose type was inferred wrongly upstream. Confirm each column is being treated as the type it actually is, and check whether rare categories survive generation at all.
Worth knowing
2Observations that may or may not be a problem in your context.
10 categorical column pairs do not co-occur as they do in the source
education x education-num, education x capital-gain, relationship x capital-gain, marital-status x capital-gain, capital-gain x native-country, occupation x capital-gain (+4 more)
10 pairs score below 55/100 on contingency similarity.
What to look at: Category combinations that are common in the source are rare here, or vice versa. Check whether rare joint categories are being dropped, which also tends to remove exactly the minority groups a fairness review will ask about.
Downstream performance drops materially for 1 target
native-country
native-country: 0.04 vs 0.08 on real training data (f1_macro, above chance)
What to look at: The data is usable but lossy for these targets. Whether that is acceptable depends on the downstream decision — fine for a development fixture, not for benchmarking a production model.
Method and assumptions
- Fidelity uses Kolmogorov–Smirnov and total-variation complements per column, plus correlation and contingency similarity for every column pair.
- Privacy combines exact-copy detection, nearest-neighbour distance against a real holdout baseline, membership inference, and the singling-out, linkability and attribute-inference attacks.
- Utility trains gradient-boosted models on the synthetic data and evaluates them on held-out real data, measured against the same model trained on real data and against chance.
- Attack metrics used the 1,800-row holdout you supplied as the control group, on the stated basis that those records were excluded from your generator's training data. The reported risks are therefore residual risk: exposure attributable to the synthetic data rather than to what any attacker could guess.
Verification
Scores are bound to the exact files measured by their SHA-256 fingerprints. Hash your copy of a file to confirm it is the one this report describes — the fingerprints outlive the source data, which is deleted.
- Synthetic dataset SHA-256
- a29d58a45c555dde1151f7c8d4fa7d4c72ac086f49a52f0ea0fcfd76f1ea2504
- Real sample SHA-256
- 5e860e6b9dee5e06aa512030c3624534859c091f6b43386b13d877601ffbced2
- Holdout sample SHA-256
- 6780882f9b4d2d60c0e67101a0d0eb95186d112f435185684404d25cc290c820
- Metrics engine
- v0.4.0
- Benchmark snapshot
- not applicable — benchmarking not enabled
- Source data
- deletion pending
This report is independent third-party evidence intended to support your Data Protection Impact Assessment or anonymisation assessment. It is not a legal determination, a certification, or a statement that any dataset is compliant with the GDPR or any other regulation. Scores are reproducible from the metrics engine version stamped above.