The loophole AI uses to train on our data

Hi, and happy Tuesday.

The “don’t train on my data” checkbox in ChatGPT, Claude and other chatbots may not work the way we think it does.

There are a lot of distracting AI headlines at the moment - such as labs “pacing the frontier” - or slowing down -  as they reach recursive self-improvement, i.e. AIs models that can build their own AI models.

But I think the story you’ll want to hear more about - though less sensational on the surface - is OpenAI’s claim that its AI has solved Navier–Stokes, one of mathematics’ most prestigious unsolved problems.

Why?

Because it appears the OpenAI solution to Navier–Stokes may have come from its AI training on the data of mathematicians who were already on the verge of solving it themselves.

But what is Navier–Stokes and why is it important?

Navier-Stokes equations are the fundamental laws of physics that describe how fluids move - where "fluid" means any liquid or gas. They take Newton’s second law of motion (F = ma) and apply it to fluid flow, accounting for forces like pressure, friction (viscosity), and turbulence.

These equations underpin everything from aircraft design to weather forecasting and blood-flow modelling.

Even though engineers use these equations every day using supercomputers to calculate approximations, mathematicians still don't fully understand the math behind them.

Specifically, we cannot prove whether the equations always yield smooth, reasonable solutions, or if they can break down at some point. 

To highlight how deep this mystery is, the Clay Mathematics Institute named it one of the seven Millennium Prize Problems in 2000, offering a $1 million reward for a definitive mathematical proof.

The mathematicians who almost solved it

Mathematicians Tristan Buckmaster and Levent Alpöge had been making significant progress on the problem since the start of the year. They had spent months developing their approach using Claude and OpenAI’s Codex, uploading drafts and discussing their work inside those systems.

By August, their progress had become common knowledge in academic circles.

Realizing this Millennium Prize problem may be solvable,  OpenAI innocently decided to direct approximately 10,000 AI agents at it - to have a go themselves.

After 88 hours, 2.7 million messages and 130 billion output tokens, they produced a Navier–Stokes solution. Estimates place the compute cost in the eight figures.

Yes, they probably spent $10-$40 million pursuing a $1 million prize.

But what concerned Tristan Buckmaster was that OpenAI’s model had pursued a “forced-blowup” route similar to the one his team had quietly been exploring. He described “forced” as a bright red flag: almost nobody was working on that approach, and he did not believe a model would naturally arrive there within days.

The telling phone call

Text messages were exchanged between between the mathematicians and OpenAI’s Sébastien Bubeck. In those, Bubeck says:

The call happened. Buckmaster writes of it afterwards:

The reply was, “Why would you ruin your career?”

I replied that I’m an academic.

The reply was: “If you don’t want me to be nice, then I don’t have to be nice.”

There is no definitive proof that OpenAI used Buckmaster and Alpöge’s data, and OpenAI claims their specific conversations were not accessed, including for training their models.

Yet the controversy exposes a much larger problem.

When we select “don’t train on my data,” we naturally interpret that as:

Nothing learned from this conversation will be used to train future models.

We did a deep dive into the OpenAI and Anthropic terms of service and this is not necessarily what the words promise.

Even with “don’t train on my data” checked, the AI companies are not prevented from creating derivative content from your content - such as summaries, extracted facts and synthetic conversations preserving the substance of the original content.

Given each new model produced by the labs is even larger than the previous and requires even more data, and everything on the Internet had already been exhausted by GPT-3, I’d always wondered what the source of the training data was for the newer models.

We may now have an answer: It appears that, while our original conversations may not be used verbatim, their intellectual ghosts may be used to train those newer models.

So what can we do?

(1) Run open-source models locally for proprietary work.

For summarization, classification, extraction and routine analysis, today’s open models are already highly capable. Local hosting is the only strong technical guarantee that your data never reaches somebody else’s infrastructure.

We already do this ourselves, and many of the systems use these as a key component. They’re fairly easy to set up.

(2) Compartmentalize across models.

Give different providers separate pieces of a problem. One model can research the literature; another can organize public evidence. Keep any one provider from seeing the complete picture, so they can “connect the dots.”

(3) Distinguish convenience from invention.

Asking a cloud model to summarize previous attempts at Navier–Stokes reveals little.

Explaining your novel route to solving it reveals everything.

The takeaway?

AI is becoming too powerful to ignore. Refusing to use it entirely would be like insisting on a horse and cart while everyone else has discovered the automobile.

Instead, we have to be smart in how we use this technology. 

A lot of what we do at Prescouter - and what you probably do too - is “building the map.” While these outputs can be quite sophisticated,  they are often not introducing anything particularly novel to the model.

However, how those maps are used - to produce the buried treasure - warrants consideration, such as through using locally hosted models.

Best,

Dino