How much data do you need to validate synthetic data?

Short answer: far less than you think. A random sample of 10,000–100,000 rows of your real data, and a similar amount of synthetic data, is enough for every metric a validation engine runs. Uploading a full 40-million-row production table doesn't sharpen the scores — it just makes the upload slow and the privacy attacks expensive.

This page explains why samples suffice, how to draw one safely, the one situation where naive sampling goes wrong, and how big a holdout should be.

Why samples are enough

Every validation metric is statistical. A Kolmogorov–Smirnov distance between two distributions, a correlation matrix, an attack success rate with a confidence interval, a train-on-synthetic-test-on-real performance delta — all of these are estimates whose precision improves with sample size at a steeply diminishing rate. By tens of thousands of rows, the estimates have stabilised: running the same metrics on ten times more data moves the scores by less than their own confidence intervals.

There is a practical corollary on the cost side. The attack-based privacy metrics — nearest-neighbour searches, singling-out and linkage attacks — scale badly with dataset size. Past ~100K rows you are paying quadratically more compute for statistically indistinguishable answers. That is why file-size limits on validation services exist: they are methodology, not upsell.

Two honest exceptions where more data genuinely helps:

• Very rare classes or categories. Statistical stability is per-estimate; an estimate about something that appears 30 times in your sample is not stable. See the sampling caveat below.
• High-cardinality structure. If the analysis you care about lives in fine-grained combinations (hundreds of categories × hundreds of categories), a larger sample keeps the joint cells populated. Even then, 100K–500K rows is usually the ceiling of usefulness, and per-file limits are why the cap is a size, not a row count.

How to sample

Randomly, and that's nearly all of it. A uniform random sample preserves distributions, correlations, and class balance in expectation. Do not filter, do not take the first N rows (they are usually sorted by time or ID, which biases everything), do not deduplicate first.

One-liners:

# pandas
python -c "import pandas as pd; pd.read_csv('data.csv').sample(100_000, random_state=7).to_csv('sample.csv', index=False)"

# shell (keeps the header row)
(head -1 data.csv && tail -n +2 data.csv | shuf -n 100000) > sample.csv

-- SQL (engines vary; this is the portable pattern)
SELECT * FROM my_table ORDER BY random() LIMIT 100000;

For Parquet, pd.read_parquet(...).sample(...) works the same way.

The one caveat: rare classes. If your data has a class or category you specifically care about — fraud at 0.5%, a rare diagnosis, a small region — check that it survives sampling in useful numbers. At 0.5% prevalence, a 100K sample keeps ~500 examples: fine. At 0.01%, it keeps ~10: not fine — the utility metrics for that class will be noise, and the report may flag a thin minority class that is your sample's fault, not your generator's. The fix is not filtering (that biases everything else); it is either a larger sample, or a stratified sample that keeps class proportions while guaranteeing a floor — at least a few hundred examples of the class you care about:

# stratified: proportions preserved, rare class guaranteed
df.groupby("label", group_keys=False).apply(lambda g: g.sample(max(min(len(g), 300), int(len(g)*0.02))))

Sample the real and synthetic data the same way. If the synthetic dataset is huge too, sample it with the same method and a similar size. Comparing a 100K real sample against 40M synthetic rows is valid but wasteful; 100K against 100K is the sweet spot.

How big should the holdout be?

The holdout — real records your generator never saw, which is what turns attack-based privacy metrics from upper bounds into measurements — needs enough rows for the attacks to have statistical power. A few thousand rows is usually plenty; a few hundred is thin. Draw it randomly from the same population as the training data, before generation, and keep it with the same care as the training data — it is real data. More on why it matters: why privacy attacks need a holdout.

What this means in practice

For a typical validation run: a 10K–100K-row random sample of the real data, a similar-sized synthetic sample, and a few-thousand-row holdout. Total upload usually under 100 MB, minutes of processing, and scores that would not change if you uploaded everything.

If your file is much bigger than that, the right move is almost never a bigger plan — it is a sampling one-liner.

Related

Synthetic data validation: the complete guide · Why privacy attacks need a holdout · TSTR