Train on synthetic, test on real (TSTR)

TSTR is the standard way to measure whether synthetic data is useful — not whether it looks right, but whether a model that learns from it works on real data. It is the core of the utility pillar in synthetic data validation, and it is the metric most likely to expose a generator that got the statistics right and the substance wrong.

Definition

Train a predictive model on the synthetic dataset. Evaluate it on real data the model has never seen. Compare that performance to a model trained on real data and evaluated on the same held-out real rows.

The first is TSTR (train synthetic, test real). The comparison baseline is TRTR (train real, test real). The utility of the synthetic data is how close TSTR gets to TRTR. If a model taught only by synthetic rows performs almost as well on real rows as a model taught by real ones, the synthetic data preserved whatever structure that task depends on.

The intuition: fidelity metrics check that the synthetic data has the right shape; TSTR checks that it has the right content — the relationships that actually let you predict something.

How it's computed

The procedure is simple; the choices inside it are what make results comparable or not.

1. Choose a target column — the thing to predict. In a validation setting you usually don't know the user's downstream task, so pick several plausible targets automatically: categorical columns with a sensible number of classes for classification, numeric columns for regression. Reporting across up to a handful of targets, and averaging, is more robust than committing to one.
2. Split the real data into a training portion and a held-out test portion. The held-out portion is used for evaluation only, in both arms.
3. Train two models with the same algorithm and settings — one on the synthetic data, one on the real training portion. Simple, well-understood models (gradient-boosted trees, logistic regression) are preferable to tuned deep models: the point is to measure the data, not the modelling.
4. Evaluate both on the held-out real rows with a task-appropriate metric: AUC or F1 for classification, R² or RMSE for regression.
5. Report the gap. Two ways are common. The ratio TSTR / TRTR (1.0 = no loss). Or, more carefully, skill above chance: how much of the improvement over a trivial baseline (predicting the majority class or the mean) the synthetic-trained model retains. Skill above chance is better because it doesn't reward tasks that were trivially easy to begin with.

A validation engine will typically fix the algorithm, use skill above chance, average across auto-detected targets, and report per-target results so a bad aggregate can be traced to its cause.

What a bad score usually means

A low TSTR score with high fidelity is a very specific symptom, and it almost always has one of a small number of causes.

Under-represented minority classes. The most common by far. The real data has 2.1% positive cases for fraud; the synthetic data has 0.4%. Column-level fidelity barely notices — the marginal is "close" — but any classifier trained on the synthetic data has seen a fraction of the positive examples and is blind to the pattern. Look at per-class counts, not just distributions.

Collapsed dependencies. The generator learned each column but not the relationships between them, so the synthetic data has the right marginals and no signal. TSTR on any target that depends on those relationships falls to near-chance. This will usually show up in correlation-preservation metrics too; if it doesn't, the dependency was non-linear or conditional and only TSTR caught it.

Lost tails and outliers. Regression targets are often driven by extremes the generator smoothed away. R² drops because the model never saw the high-value cases.

Target leakage in the real data, not the synthetic. Occasionally TSTR is low because TRTR is unrealistically high — the real data contains a column that trivially predicts the target and the generator, correctly, didn't reproduce that artefact tightly. Worth checking before blaming the generator.

The remedy in each case is generator-agnostic: condition or stratify generation on the minority class, use a model that captures multivariate structure, don't clip tails, and re-run TSTR per target to confirm the fix landed where the loss was.

Reading TSTR alongside the other pillars

TSTR is powerful and it is task-dependent. A dataset can score well for one target and poorly for another. That is not a flaw in the metric — it is information about what the generator preserved. It does mean:

• Never read a single aggregate utility number without the per-target breakdown.
• Excellent TSTR does not imply privacy. A generator that memorised its training data has near-perfect TSTR (it is training on real rows in disguise) and no privacy at all. Utility must be read next to the privacy pillar, and a critical privacy failure should cap the overall grade regardless of utility.
• TSTR requires real data to test on. In a synthetic-only validation run, TSTR cannot be computed and should be reported as not tested, not as passed.

Worked example

The published sample report is a run of exactly this procedure: the UCI Adult census dataset, 4,200 rows × 15 columns, synthesised with SDV Gaussian Copula and scored against a real sample plus an 1,800-row holdout on metrics engine 0.4.0.

Target	Task	TRTR	TSTR	Skill retained
class	Classification (ROC-AUC)	0.895	0.737	60.0%
sex	Classification (ROC-AUC)	0.916	0.689	45.3%
native-country	Classification (macro F1)	0.081	0.040	0%

Averaged across the three auto-detected targets, that is a utility score of 35.1/100 — by a wide margin the weakest of the four pillars in that report.

The worst target is native-country, where the synthetic-trained model falls below the chance baseline and skill retained clamps to zero; the findings put it down to collapsed categorical structure rather than a missing minority class, naming native-country among ten column pairs that no longer co-occur as they do in the source. Note also that TRTR is only 0.081 macro F1 against a 0.040 chance baseline — there was little signal in the real data for the generator to preserve, which is why the per-target breakdown matters more than the average.

Running the same engine over a deliberately memorising generator on the same dataset shows why utility alone proves nothing: it retains 69% of TRTR skill — beating the real-data model outright on class, 0.981 against 0.895 — and still grades D, because its membership-inference attack succeeds with an AUC of 1.00 and the resulting privacy finding caps the grade.

The full report for this run is at /sample.

How SynthProof reports it

The utility pillar in a SynthProof report is TSTR skill above chance, averaged over up to three auto-detected targets, with a fixed model family so results are comparable across datasets and generators. Per-target results are shown, and when a target scores poorly the findings section names the likely cause — an under-represented class, a collapsed dependency — with a general remedy. In synthetic-only mode utility is marked not tested. A utility score below the critical floor caps the overall grade; so does a critically low privacy score, so high utility never carries a dataset that leaks.

Related metrics

Correlation preservation usually fails alongside TSTR when dependencies collapsed. Column shape (KS / TVD) can look fine while TSTR fails — that gap is itself a finding. Membership inference is the counterweight: the metric that catches the memorising generator TSTR would reward.

Back to the guide: Synthetic data validation: the complete guide.