Train on synthetic, test on real (TSTR)

TSTR is the standard way to measure whether synthetic data is useful — not whether it looks right, but whether a model that learns from it works on real data. It is the core of the utility pillar in synthetic data validation, and it is the metric most likely to expose a generator that got the statistics right and the substance wrong.

Definition

Train a predictive model on the synthetic dataset. Evaluate it on real data the model has never seen. Compare that performance to a model trained on real data and evaluated on the same held-out real rows.

The first is TSTR (train synthetic, test real). The comparison baseline is TRTR (train real, test real). The utility of the synthetic data is how close TSTR gets to TRTR. If a model taught only by synthetic rows performs almost as well on real rows as a model taught by real ones, the synthetic data preserved whatever structure that task depends on.

The intuition: fidelity metrics check that the synthetic data has the right shape; TSTR checks that it has the right content — the relationships that actually let you predict something.

How it's computed

The procedure is simple; the choices inside it are what make results comparable or not.

1. Choose a target column — the thing to predict. In a validation setting you usually don't know the user's downstream task, so pick several plausible targets automatically: categorical columns with a sensible number of classes for classification, numeric columns for regression. Reporting across up to a handful of targets, and averaging, is more robust than committing to one.
2. Split the real data into a training portion and a held-out test portion. The held-out portion is used for evaluation only, in both arms.
3. Train two models with the same algorithm and settings — one on the synthetic data, one on the real training portion. Simple, well-understood models (gradient-boosted trees, logistic regression) are preferable to tuned deep models: the point is to measure the data, not the modelling.
4. Evaluate both on the held-out real rows with a task-appropriate metric: AUC or F1 for classification, R² or RMSE for regression.
5. Report the gap. Two ways are common. The ratio TSTR / TRTR (1.0 = no loss). Or, more carefully, skill above chance: how much of the improvement over a trivial baseline (predicting the majority class or the mean) the synthetic-trained model retains. Skill above chance is better because it doesn't reward tasks that were trivially easy to begin with.

A validation engine will typically fix the algorithm, use skill above chance, average across auto-detected targets, and report per-target results so a bad aggregate can be traced to its cause.

What a bad score usually means

A low TSTR score with high fidelity is a very specific symptom, and it almost always has one of a small number of causes.

Under-represented minority classes. The most common by far. The real data has 2.1% positive cases for fraud; the synthetic data has 0.4%. Column-level fidelity barely notices — the marginal is "close" — but any classifier trained on the synthetic data has seen a fraction of the positive examples and is blind to the pattern. Look at per-class counts, not just distributions.

Collapsed dependencies. The generator learned each column but not the relationships between them, so the synthetic data has the right marginals and no signal. TSTR on any target that depends on those relationships falls to near-chance. This will usually show up in correlation-preservation metrics too; if it doesn't, the dependency was non-linear or conditional and only TSTR caught it.

Lost tails and outliers. Regression targets are often driven by extremes the generator smoothed away. R² drops because the model never saw the high-value cases.

Target leakage in the real data, not the synthetic. Occasionally TSTR is low because TRTR is unrealistically high — the real data contains a column that trivially predicts the target and the generator, correctly, didn't reproduce that artefact tightly. Worth checking before blaming the generator.

The remedy in each case is generator-agnostic: condition or stratify generation on the minority class, use a model that captures multivariate structure, don't clip tails, and re-run TSTR per target to confirm the fix landed where the loss was.

Reading TSTR alongside the other pillars

TSTR is powerful and it is task-dependent. A dataset can score well for one target and poorly for another. That is not a flaw in the metric — it is information about what the generator preserved. It does mean:

• Never read a single aggregate utility number without the per-target breakdown.
• Excellent TSTR does not imply privacy. A generator that memorised its training data has near-perfect TSTR (it is training on real rows in disguise) and no privacy at all. Utility must be read next to the privacy pillar, and a critical privacy failure should cap the overall grade regardless of utility.
• TSTR requires real data to test on. In a synthetic-only validation run, TSTR cannot be computed and should be reported as not tested, not as passed.

Worked example

UCI Adult census, 4,200 rows × 15 columns, SDV Gaussian Copula, metrics engine 0.8.0. Utility pillar 35.1/100, averaged over three auto-detected targets.

Target	Task	Metric	TRTR (real)	TSTR (synthetic)	Chance	Skill retained
class	classification	ROC-AUC	0.895	0.737	0.5	60.0
sex	classification	ROC-AUC	0.916	0.689	0.5	45.3
native-country	classification	F1-macro	0.081	0.040	0.040	0.0

The "skill retained" column is measured above chance, and native-country shows why that matters. Its raw TSTR score of 0.040 is not 49% of the real model's 0.081 — it is exactly chance (0.0399). A model trained on the synthetic data predicts native-country no better than guessing the prior, so the retained skill is zero, not fifty percent. Scored from zero instead of from chance, that column would have read as a partial success.

native-country is the worst target, and the finding on it is about the sample rather than the synthetic data: "native-country: 37 classes have fewer than 300 examples in the real sample (the rarest has 1); utility results for these classes are unreliable." It is a 40-plus-category column dominated by one value, and in a 4,200-row sample most of its categories have only a handful of rows — note that the real-data model only reaches 0.081 itself. A zero on a target this thin says the sample cannot judge it, not that the generator failed it; how much data you need covers why.

The memorising-generator control scored 69.3 on the same pillar, with ROC-AUC of 0.981 on class and 0.990 on sex — beating the real training data at 0.895 and 0.916, because it is training on real rows in disguise. Its overall grade was still capped at D: privacy scored 3.8, membership inference returned AUC 1.0, and all 4,200 rows were exact copies. Utility alone would have ranked it first.

How SynthProof reports it

The utility pillar in a SynthProof report is TSTR skill above chance, averaged over up to three auto-detected targets, with a fixed model family so results are comparable across datasets and generators. Per-target results are shown, and when a target scores poorly the findings section names the likely cause — an under-represented class, a collapsed dependency — with a general remedy. In synthetic-only mode utility is marked not tested. A utility score below the critical floor caps the overall grade; so does a critically low privacy score, so high utility never carries a dataset that leaks.

Related metrics

Correlation preservation usually fails alongside TSTR when dependencies collapsed. Column shape (KS / TVD) can look fine while TSTR fails — that gap is itself a finding. Membership inference is the counterweight: the metric that catches the memorising generator TSTR would reward.

Back to the guide: Synthetic data validation: the complete guide.