Synthetic data platform

Sensitive data has an awkward property: you need it to build anything useful, and you are not allowed to have it. Health data especially. The moment real records leave the place they live, you have a problem that no amount of good intentions fixes.

Synthetic data is the way around that. You generate a dataset that behaves like the real one, with the same distributions, the same correlations, the same awkward edge cases, except that none of the rows belong to anybody. You can hand that to a model, to a demo, to a test suite, to a team that is not cleared for the real thing.

The generator

The generating half lived in its own service, with its own database and its own deploy, so it could be worked on and scaled without touching the rest of the platform. GAN models did the heavy lifting on the structured side: train on the shape of the real set, then produce rows that hold the same statistical properties without reproducing anybody in particular.

The other half was prompted, not trained. That part is quick to change and quick to get subtly wrong, which turned into the interesting problem on this project.

Telling whether a change was an improvement

Editing the instructions that drive generation takes about ten seconds. Knowing whether the edit made things better takes a lot longer, and eyeballing a handful of rows tells you almost nothing. The output looks plausible either way. That is the whole trouble with generated text: plausible is the default, not the goal.

So I built an orchestration of LLMs whose entire job was to judge the output. A run would come out the other end and go to a panel of models, each scoring it against a named axis rather than a vague sense of quality. Patient satisfaction. Doctor satisfaction. The rest of the things we actually cared about the generated conversations getting right.

Every set of instructions was a version, and every version carried its scores. A new one did not replace the old one because it read better to me on a Tuesday afternoon. It replaced it when it beat it on the numbers, on the same inputs, across the axes. When it did not, the change went back in the drawer, which happened more often than you would hope.

Two things came out of that which I did not really expect. The scores made regressions visible, so a prompt that quietly got worse at one thing while getting better at another stopped being invisible. And arguing about the axes was more useful than arguing about the prompts, because naming what you want out of the output is most of the work of getting it.

How it was put together

• FrontendReact, TypeScript, Hero UIThe app people actually used. Upload a dataset, describe what you needed out of it, watch the run, download the result.
• Dataset serviceGAN models, PostgresIts own service, with its own database and its own deploy. It trained on the shape of the real data and produced a synthetic set that kept the statistics and dropped the people.
• ScoringAn orchestration of LLMsA panel of models reading the output of a run and scoring it, so a change to the generation instructions could be compared against the version before it.
• InfrastructureAWS, Docker, CI/CDContainers per service, a pipeline per environment, and SSL and domains configured on the way up rather than by hand afterwards.
• Marketing siteAstro, React islandsA separate, mostly static site, with React only where a piece of the page genuinely had to move.

The services were containerised and each had its own pipeline, with development, staging and production as separate targets and validation in front of every deploy. Domains and certificates were configured by the same automation that brought the machines up, because the alternative is remembering to do it at three in the morning eleven months later.

What stayed with me

The GANs were the part that sounded impressive and the scoring was the part I think about. Anything generated needs something pointed at it that is not your own optimism, and once you have that, you can change things without being frightened of them :)

← Back