Method
How a dataset is scored.
Published methods, named libraries, and stated limits. Everything on this page is checkable against a report you receive — and every claim here is one we would rather state plainly than be asked about in diligence.
- Data types
- Tabular — CSV, Parquet
- Distributions
- SDMetrics · KS and total-variation complements, per column
- Correlations
- Every column pair — correlation and contingency similarity
- Privacy attacks
- Anonymeter · singling out, linkability, attribute inference
- Leakage
- Exact-copy detection, nearest-neighbour distance, membership inference
- Utility
- TSTR · gradient boosting, scored above chance against a real-data baseline
- Report
- PDF certificate + interactive dashboard, immutable once issued
- Findings
- What is wrong and where — generator-agnostic, never how to tune yours
- Provenance
- SHA-256 of every input file, printed on the certificate
- Versioning
- Metrics engine and benchmark snapshot stamped on every report
- Retention
- Source data deleted as the final step of every job, on every plan
The four pillars
Three pillars need a real sample to compare against. The fourth is structural and runs on anything.
Fidelity
30% of the overall scorePer-column shape tests (Kolmogorov–Smirnov complement for numerics, total-variation complement for categoricals) and every pairwise relationship — correlation similarity for numeric pairs, contingency similarity where a category is involved. Shapes and pair trends are weighted equally.
SDMetrics skips column pairs whose real-data association falls below a threshold. We drop that threshold to zero and score every pair, because a correlation the generator invented — one that is not in the source at all — is a finding, not noise to be filtered out.
Privacy
40% of the overall scoreExact-copy detection, distance to closest record against a real baseline, membership inference, and the three Article 29 Working Party criteria via Anonymeter: singling out, linkability, and attribute inference. Attacks run at 500 attempts each.
Privacy carries the heaviest weight because it is the pillar a buyer cannot self-assess and the one a DPIA turns on. Exact copies saturate the metric fast: a 5% match rate drives that component to zero.
Utility
30% of the overall scoreTrain on synthetic, test on real. Histogram gradient boosting over up to three auto-detected target columns, task type (classification or regression) inferred per target, scored with ROC-AUC or F1 for classification and R² for regression.
Measured above chance, not from zero. A model that only ever predicts the majority class is not 70% useful — it is 0% useful, so the score is taken relative to a prior-strategy baseline on the same split.
Structural integrity
Reported separately, not averaged inDuplicate rows, missing-value rates, constant columns, extreme outliers, and collapsed categories — the common generator failure modes that are visible without any reference data.
This is the only pillar that can run without a real sample, so it is what a Mode 2 (synthetic-only) report is built from. It is scored and can raise a critical finding, but it is not part of the weighted headline.
From pillars to a grade
The headline score is a weighted mean of fidelity, privacy and utility — but a mean alone would let one excellent pillar carry a dataset that fails on the pillar that matters.
Grades
- A
- 90 and above
- B
- 75 – 89
- C
- 60 – 74
- D
- below 60
Critical findings
A pillar below its floor is a disqualifying finding, not an average-able weakness. It caps the overall grade at D no matter how the other pillars scored, and the reason is printed on the certificate.
- Privacy
- 25
- Structural integrity
- 25
- Fidelity
- 10
- Utility
- 10
The floors are not uniform, and the asymmetry is deliberate. A dataset that reproduces real records verbatim scores 100 on fidelity and beats an honest generator on utility — so privacy keeps a strict floor, because that is the outcome an independent validator must never grade well. Weak fidelity or utility means the data is not much use, which the weighted average already expresses; capping the grade on top of that would tell you less than the pillar score itself does.
Findings
A score tells you a pillar is weak. A finding tells you what is wrong and where. Both are in the report.
Three tiers, and silence when there is nothing to say
Findings are graded disqualifying, material, or worth knowing. A disqualifying finding is the one that capped the grade — the report opens with it rather than burying it, naming the pillar, its score, the floor it missed and what that means in a sentence. The thresholds are set so a clean dataset produces no findings at all: a section that always fires is one readers learn to skip, which costs us the times it matters.
Lost relationships are told apart from invented ones
A column pair that scores badly can be wrong in two opposite ways, and the similarity score alone cannot distinguish them — the generator either lost a relationship the source had, or fabricated one it did not. We report the correlation on both sides, so the finding reads "age × income collapsed 0.62 → 0.11" rather than a bare score. A fabricated relationship is the more dangerous of the two: it is not missing signal, it is false signal, and a model trained on it learns something the real world does not contain.
Privacy findings name the driver, not just the risk
Where the attack reports it, the finding says which part of the schema carries the exposure: the column with the highest attribute-inference risk, the split of quasi-identifiers that makes records linkable, the count of synthetic rows that are byte-for-byte copies of a real record. Article 29 framing is carried through to each one.
Findings survive zero retention, by construction
Findings are derived from the measurements and the shape statistics, never from retained records — the module takes the scored result as its only input and cannot reach the data even in principle. That is why they can appear on a report whose source data was deleted before the PDF was written, why the deletion timestamp in the footer does not come with an asterisk, and why it is enforced by a test rather than left to convention.
What a finding will never say
Diagnosing is auditing. Prescribing is consulting. We cannot be both and stay worth citing, so the line is drawn in the engine rather than left to judgement:
- No generator-specific tuning advice. Never “set epsilon to X in this tool”. We do not know your pipeline, we have not tested that change, and an auditor who tells you how to score better on their own audit is not an auditor.
- No offer to fix it for you. Remediation is a different business with a different incentive. The moment we are paid to raise a score we stop being independent of it.
- No promised outcome from a remedy. Where a general technique is worth naming, it is named as a technique. We can report the driver we measured; we cannot promise what a change would score without re-running on data we have already deleted.
What this does not claim
The limits are part of the method. A validator that only publishes its strengths is doing the thing we exist to correct.
Without a holdout, attack results are upper bounds
Anonymeter measures residual risk only when it can compare against real records the generator never saw. If you do not upload a holdout, we report naive risk — labelled on the certificate as an upper bound — and membership inference is reported as not tested rather than passed. Splitting your real sample in half does not fix this: both halves stand in the same relation to the generator.
Attacks are stochastic, so privacy is a measurement with error
The attack-based metrics sample; run the same dataset twice and the privacy score moves slightly. Treat a privacy score as a measurement with a tolerance, not a fixed property of the file. Below 500 real rows the attacks are noisy enough that the report says so explicitly rather than publishing a confident number.
This is evidence, not a legal determination
The report is designed to be evidence you can put in front of a DPO, an auditor, or a client, and the Compliance report maps its language to the Article 29 framing a DPIA uses. It is not a certification of GDPR compliance, and we will never describe it as one. That determination is your data protection officer's to make, and ours to support with measurements.
Scores describe the files you uploaded
Every input is SHA-256 fingerprinted and the digests are printed on the certificate. A score is bound to exactly those bytes — it does not transfer to a regenerated dataset, a different sample, or a later version of the same pipeline.
Want to see it applied? Read a real report — published unedited, including the pillar the generator failed.
Get the first certificate
We're opening validation to a first group of teams. Leave your email and we'll get in touch when your category is ready.
Or read a full sample report first — it's a real run, published unedited.