Independent synthetic data validation
Synthetic Data Validation Report
Full assessment — synthetic data against a real sample (Mode 1)
- Job
- e2e-california_housing-synthetic-holdout
- Category
- other
- Issued
- 2026-08-16 09:09 UTC
- Synthetic dataset
- 4,200 rows × 9 columns
- Real sample
- 4,200 rows × 9 columns
- Holdout sample
- 1,800 rows × 9 columns
Weighted across privacy (40%), fidelity (30%) and utility (30%). Structural integrity is reported separately as a precondition check.
Fidelity
Statistical similarity to the source data
Overall distributional similarity scores 86/100. Weakest match: Longitude at 18/100.
| Measurement | Value | Score | What it means |
|---|---|---|---|
| Column shapes | 0.769 score | 77 | How closely each column's marginal distribution matches the source (KS complement for numeric, total-variation complement for categorical). |
| Column pair trends | 0.951 score | 95 | Whether relationships between pairs of columns survived generation — correlations for numeric pairs, contingency for categorical pairs. |
Privacy
Resistance to re-identification attacks
Privacy scores 99/100 (full attack suite). Highest residual risk: singling out at 95/100.
| Measurement | Value | Score | What it means |
|---|---|---|---|
| Exact record copies | 0.00% % of synthetic rows | 100 | Share of synthetic rows that are byte-for-byte copies of a real record. Any non-zero value means real records were reproduced verbatim. Reported next to the real sample's own duplicate rate, because repeated rows arise by coincidence in low-cardinality data. Near copies — within 0.05 embedded distance of a real record — are reported alongside, against the same measurement between two halves of the real sample. |
| Nearest-neighbour distance | 2.239 ratio vs real-to-real baseline | 100 | Distance from synthetic records to their closest real record, divided by the same distance measured between two independent halves of the real sample. 1.0 or above means synthetic records are no closer to the source than unrelated real records are. |
| Nearest-neighbour distance ratio | -1.62 percentage points above the real-to-real baseline | — | A row is 'anchored' when its closest real record is less than 0.5x the distance of its second closest — it sits on one individual rather than between neighbours. Reported as the excess over how often real records anchor to each other, because that rate is far from zero on ordinary data. Not scored: it flags individual rows that a percentile averages away. |
| Membership inference | 0.022 attacker advantage | 98 | How much better than guessing an attacker does at deciding whether a given record was in the training data. 0 is a coin flip; 1 is perfect identification. |
| Singling out | 0.051 residual risk | 95 | Whether the synthetic data lets an attacker build a query that isolates exactly one real individual (Article 29 WP criterion 1). Measured against the holdout you supplied, so this is residual risk — the exposure attributable to the synthetic data itself. Measured risk falls between 0.017 and 0.084 at 95% confidence; the attack samples, so this interval is the measurement. |
| Linkability | 0.002 residual risk | 100 | Whether two separate datasets about the same people can be joined through the synthetic data (Article 29 WP criterion 2). Reported for the worst of 3 column splits. Measured against the holdout you supplied, so this is residual risk — the exposure attributable to the synthetic data itself. Measured risk falls between 0.000 and 0.015 at 95% confidence; the attack samples, so this interval is the measurement. |
| Attribute inference | 0.007 residual risk | 99 | Whether an attacker who knows some attributes of a real person can infer an unknown one from the synthetic data (Article 29 WP criterion 3). Averaged across columns. Measured against the holdout you supplied, so this is residual risk — the exposure attributable to the synthetic data itself. The worst column (AveRooms) falls between 0.000 and 0.074 at 95% confidence; the attack samples, so this interval is the measurement. |
Utility
Train on synthetic, test on real
Models trained on the synthetic data retain 24% of the skill of models trained on real data, averaged over 3 target column(s).
| Measurement | Value | Score | What it means |
|---|---|---|---|
| TSTR — MedHouseVal | 0.581 r2 | 73 | Model trained on synthetic data scores 0.581 (r2) on held-out real data, against 0.797 for the same model trained on real data (chance: 0.000). |
| TSTR — Longitude | -3498.943 r2 | 0 | Model trained on synthetic data scores -3498.943 (r2) on held-out real data, against 0.957 for the same model trained on real data (chance: 0.000). |
| TSTR — Latitude | -242.394 r2 | 0 | Model trained on synthetic data scores -242.394 (r2) on held-out real data, against 0.957 for the same model trained on real data (chance: 0.000). |
Structural integrity
Internal consistency of the synthetic data
Structural integrity scores 99/100.
| Measurement | Value | Score | What it means |
|---|---|---|---|
| Duplicate rows | 0.00% % of rows | 100 | Repeated identical rows inside the synthetic data. A high rate means the generator collapsed onto a few records. |
| Missing values | 0.00% % of cells | 100 | Share of empty cells across the synthetic data. |
| Constant columns | 0.00% % of columns | 100 | Columns where every row holds the same value. These carry no information and usually indicate a generation failure. |
| Extreme values | 0.14% % of numeric cells | 97 | Values more than 4 standard deviations from their column mean — often impossible values invented by the generator. |
| Category collapse | — | — | Not scored: no categorical columns. |
Findings
What the scores above mean in practice: where each weakness sits and the general direction of a fix. Findings are derived from the measurements and shape statistics in this report — not from the data, which is why they survive the deletion recorded in the footer.
Material
3Weaknesses worth acting on before this dataset is relied upon.
2 column distributions diverge from the source
Longitude, Latitude
Weakest is Longitude at 18/100 distribution match; 2 columns score below 70.
What to look at: Per-column drift usually means the generator under-fitted these marginals — common with heavy tails, rare categories, and columns whose type was inferred wrongly upstream. Confirm each column is being treated as the type it actually is, and check whether rare categories survive generation at all.
2 column relationships present in the source were lost in generation
Latitude x Longitude, AveRooms x AveBedrms
Latitude x Longitude collapsed -0.93 → -0.02; AveRooms x AveBedrms collapsed +0.90 → +0.43
What to look at: Lost joint structure is the signature of a generator modelling columns independently, or of correlated columns being generated in separate passes. Anything trained on this data will under-estimate these relationships. Prefer a method that models the joint distribution over these columns together.
Models trained on the synthetic data do not transfer for 2 targets
Longitude, Latitude
Longitude: -3498.94 vs 0.96 on real training data (r2, above chance); Latitude: -242.39 vs 0.96 on real training data (r2, above chance)
What to look at: For these targets the synthetic data carries almost none of the predictive signal the real data has — a model trained on it is close to guessing. Check whether the relationships connecting the predictors to these targets survived generation; the fidelity findings above usually name them.
Method and assumptions
- Fidelity uses Kolmogorov–Smirnov and total-variation complements per column, plus correlation and contingency similarity for every column pair.
- Privacy combines exact-copy detection, nearest-neighbour distance against a real-to-real baseline, a per-row distance ratio, membership inference, and the singling-out, linkability and attribute-inference attacks.
- Utility trains gradient-boosted models on the synthetic data and evaluates them on held-out real data, measured against the same model trained on real data and against chance.
- Attack metrics used the 1,800-row holdout you supplied as the control group, on the stated basis that those records were excluded from your generator's training data. The reported risks are therefore residual risk: exposure attributable to the synthetic data rather than to what any attacker could guess.
Verification
Scores are bound to the exact files measured by their SHA-256 fingerprints. Hash your copy of a file to confirm it is the one this report describes — the fingerprints outlive the source data, which is deleted.
- Synthetic dataset SHA-256
- 3d6d11ebae2c83ff652b6b653a412e7e584ffd1d6c1cdebf627f566131cd610b
- Real sample SHA-256
- 253d0117d2164d584658d5d4db502d33a1548730495b2da27dc8d3e05eb795cc
- Holdout sample SHA-256
- c1766cf2f6f1f31a29e874a3dd1d28fc33e1409e811565961789dad20b24b33b
- Metrics engine
- v0.6.0
- Benchmark snapshot
- not applicable — benchmarking not enabled
- Source data
- deletion pending
This report is independent third-party evidence intended to support your Data Protection Impact Assessment or anonymisation assessment. It is not a legal determination, a certification, or a statement that any dataset is compliant with the GDPR or any other regulation. Scores are reproducible from the metrics engine version stamped above.