I have a similar story, but perhaps even stranger.

I work for a startup. We often bring a wooden arcade with us to conferences as a marketing gimmick.

The arcade runs a single side-scrolling video game. You're running from a monster and dodging obstacles. The goal is to survive as long as possible, and your result is measured in meters.

There are always a few competitive guys who spend the entire conference taking turns to play it. And every single time, the same thing happens.

Say the current high score is around 200m. Everybody fails somewhere around that number: 190m, 186m... Maybe someone manages 210m. And the high score moves up at a snail's pace.

Then, a new guy shows up and gets something like 500m on his third try. From their next turn on, everybody easily does 450 or more, even though they were struggling to get past 200 just one turn ago.

What makes it stranger is that the game is dead simple. It's not like the new guy discovered a move that unlocked this capability. And it wasn't a lack of motivation either - they'd all been playing for an hour already. They just started performing better after seeing it was possible. There has to be a name for this phenomenon.

> I think giving someone hope that an answer exists might as well be the same thing as giving them the answer these days.

If you read the history of major scientific discoveries, this has been the case for a long time. There are many things that were independently discovered by different people at nearly the same time. Once people know something is solved or solvable, it gets a relentless amount of focus.

You see this in many things. Look at climbing for instance, or running. Once the first v17 had been done, suddenly many people did them. Or running with a sub 2 hour marathon.

Not the hope, but telling them that an answer does exist. This ofcourse means now we are going to go into an even more darker cave next time we are looking for some gold and pathbreaking discoveries will become even more rare. Add to that, the fear of people not winning against ai and having fewer rewards, then fewer people even enter those fields or attempt problems over the next generation. AI erodes skills not at individual but at civilisational level.

It's something that happened before LLMs - multiple discovery. Calculus is a classic example.

If you believe the totality of the document, there was more shadiness in how OpenAI acted than just timing. Save other things they are accused of, the progression from rumors to replication would attract much less scrutiny. With those in mind timing begins to look suspicious at best.

Would anyone be surprised if major model companies had tagged the accounts of competitor employees for extra tracking? Given the concerns about distillation and bench marking it hardly seems irrational, but how it is used matters quite a lot.

That's the other thing. Even if it somehow magically turns out that they didn't plagiarize Buckmaster and Alpöge, that an independent audit goes through all their processes and finds that they could not possibly have stolen anything, and ignoring their outrageous attempts to force a coauthor off the author list, and ignoring that their employees act like small children on social media, this is a company whose culture is “we heard of a breakthrough being possible, let's scoop it”.

Normally, when you tell a coworker that you're wrapping up a result, unless they're some kind of sociopath, their natural inclination would not be to try to steal it from you.

I’m not a mathematician so take this with a grain of salt. Apparently terry tao commented that the approach used for the Euler paper can “probably” be used for solving NS but it’s still technically challenging and can probably be done with an LLM with a lot of compute. To me the crux of the issue is whether the insight were stolen so that the problem becomes something that is in the domain of LLMs. This is much different than LLMs coming up with the insight. OpenAI wants everyone to think the LLM came up with the insight and solved the thing by itself even though they have perhaps an army of researchers.

Having the chat logs enter the training data and having them have a meaningful influence on the ultimate result the model produces are very different things.

The text for all the Goosebumps books are certainly in the training data and to some small amount influenced the solve. But their contribution was so vanishingly small it would seem absurd to say R L Stein should have recourse for contibuting to the solve.

It's almost as if it was actually found by manually written brute-force algorithm running on OpenAI's massive computer cluster.

It’s fishy though that they heard one of seven problems was about to be solved and threw perhaps 15 million bucks at the right one.

[Edit: they said "two of": "On Tuesday, September 1, we heard rumors that two Millennium Prize problems had been resolved. .. we launched an effort ... on all open Millennium Prize problems".]

This is also what I've been thinking. The result itself is amazing but it's not like this was completely unexpected. There has been a huge amount of progress on the problem in the last 10 years without which it seems unlikely today's full resolution would have been possible. It is not clear what strategy was taken but it sounds like it borrowed heavily from the two spanish mathematicians. Experts will scrutinize the proof and it will be interesting to see if anything truly original or unexpected was done, outside of known techniques, a move 37.

If an entirely new problem, within a new field of mathematics were to appear tomorrow, I highly doubt a human mathematician would be useful at all on their own.

Extreme temperature levels (>2.0) can push a GPT of its manifold, essentially producing predictions barely distinguishable from random noise (it flattens the probability distribution of the next token). In theory this could predict anything including the next field of mathematics (infinite monkey theorem) but realistically that would never happen.

However, how to we know the next field of mathematics isn't a novel combinations of several other sub-fields? That level of mathematics would be indistinguishable from magic to most people and so in their eyes the GPT did something truly inventive.

The kind of humans who invent entire fields of science on their own come by a few times a generation. It’s fine to say AI isn’t anywhere as close to them in intelligence, but instead is comparable to the “average” mathematician who is building on the work done by others and taking it a bit further.

A lot of what humans do is combining old ideas.

And an LLM could in theory also stumble upon entirely new ideas: there's randomness in how they generate their reasoning and answers after all.

I suspect that we are seeing a lot of advances coming from the combination of existing but somewhat obscure knowledge coming from LLMs at the moment, because LLMs are really good at this. At least compared to humans.

Even before our AI friends became good, they were already known for having read approximately every paper and every textbook published in any language. You only need to increase intelligence a fairly small amount from there to get to something like the 'convex hull' of human knowledge.

Compare https://slatestarcodex.com/2016/11/17/the-alzheimer-photo/

The gist is that basically whenever anyone comes up with a new method you get a big burst of activity of picking up all the now lower hanging fruit, that was previously out of reach.

LLM can't be trained that easily. More like actual human are checking your logs and stealing valuable things from you.

Maybe just a rumor of a high value target having their API keys accidentally consumed in the context...

People talk. If you have a breakthrough solving one of the most famous problems outstanding, you're going to tell people.

You'll say, "Don't tell anyone", which they will ignore because they get a rush and perceived status by sharing it. So then they tell someone, along with "Don't tell anyone", etc.

It's a small enough world (both in academic math, one at Anthropic) that you get to OAI in very few hops.

They weren't working in secret?

> one should not use LLM services for confidential or proprietary information

That’s obvious, isn’t it? Just like you wouldn’t upload your confidential documents to an online spellchecker, or your proprietary code to an online compiler?

Or if you're going to trust one of them, maybe it shouldn't be OpenAI

Or use Lumo from Proton. Are there any other privacy first companies offering LLMs?

That they do it is just concerning to me in that it says that home-ran models just aren't good enough. Surely researchers like this have the processing power to run them at home, they just don't have the processing power to train models of comparable level.

This is something I feared would happen and where open source would be left behind. Maybe they can do something with crowd-sourcing computational power from volunteirs. They were after all able to get Leela Chess Zero to be comparable to AlphaZero by training from volunteer processing power but it seems to me we live in a world now where the best models keep their stuff closed.

In imagine generation too. I'm not sure how well Stable Diffusion can compete in following instructions with all those advanced models that are kept secret.

Their privacy policy for normie subscribers says in plain English they use your Personal Data for research. I think it’s pretty unreasonable to use the service and expect otherwise.

> as they all seem to be run by assholes.

Wait until you learn how many other SaaS and web 2.0 and cloud based things are also run by assholes.

You know this metaphor? https://en.wikipedia.org/wiki/Turtles_all_the_way_down

But instead of turtles, it's assholes.

But more seriously, no, none of what I wrote above is an attempt to excuse or play down the specific role of assholes in large AI companies.

That's a good question.

It is able to contribute code, but maybe not good code.

It's the same in math: it's able to solve problems, but not necessarily in a good way with a human readable code.

Math papers are a lot like software:

- theorems are like API

- lemmata like internal/private function API

- definitions are like types

- the proofs are the implementation

The proofs of ChatGPT are not necessarily readable or maintainable.

Math problems are often stated in a way that makes it possible to automatically verify if a solution is correct. Which means a loop that speculates an approach (LLM and/or prompts), implements it (LLM), then checks (automated) can work. You still need to have a very good LLM, and probably very good prompts with interesting research directions otherwise you can probably loop forever.

No one knows if it's actually LLM doing the heavy weight. It could be just human written brute force algorithm running on their massive computer cluster.

Notably all the major announcements so far are counterexamples or formalizations of existing results to my knowledge. Not necessarily something you can just brute force, but areas with high return on elbow grease.

You don't need a GOOD proof, just A proof.

search, verifiability and compute

Isn't it also a simple idea that a model designed to recall relevant information from its training data, which is also known to have been trained on data from user transcripts, would, in fact, reproduce directly relevant work by leading experts in the field? Seems like Occam's razor would apply to that situation as well.

We know that LLMs are trained to recall relevant info. We know AI vendors are using user transcripts to train models. Two plus two equals four, right? I mean, an LLM that failed to recall the transcripts of those researchers would be a bad model.

Given that the solution took a somewhat “unusual” approach, I find it even more unlikely that an AI model would have come up with this on its own.

Funny how they did not solve any of the other problems, just the one where there was already solutions to the NS with some restrictions in their chats, and their solution seems to derive from those.

Why is that simple or plausible? Why is simpler or more plausible than lifting an almost-finished solution from a researcher's account?

There's also an elephant here in this room: Quite "coincidental" one of the coauthors worked for the competition. Of course GPT would know this fact. And OAI has every incentive to particularly monitor those accounts.-

I agree with Occam.

That 100x step up from using 100 agents for Euler to 100_000 agents for Navier-Stokes, in a single day seems a bit sus.

I also think this is not ethical behavior. This is at least academic dishonesty, kind of a plagiarism or intellectual theft.

That’s why they wanted to credit Tristan and to give $1M award to him. But again, they acted unethically in that process as well. They wanted him to remove Levent (Anthropic affiliation) as co-author and threatened Tristan to “end his career”. Their behavior is actually telling, their work was not completely independent from Tristan&Levent’s unpublished work.

This is not the simplest explanation.

The most direct line from problem to proof is OpenAI building off of conversations the mathematicians had with their AI.

On the contrary, a level-headed summary that gathers information from all the different sources is necessary.

Yes, you have ability to get information very quickly, but not everyone does.

The breakthrough was on August 15th, but Tristan and Levent had been working towards it (with the help of various models) for the best part of a year.

I personally doubt that their work influenced the OpenAI result - OpenAI themselves say "While unlikely, we cannot rule out that..." - but that "we cannot rule out" is exactly the problem.

If even OpenAI "cannot rule out" the influence of their usage of ChatGPT on this layer result then my discomfort at not understanding how my own usage of ChatGPT affects its training is magnified.

I would assume a lot of codex data goes back into training. A well steered session is extremely valuable data.

AI companies use heuristics to filter sessions, then llms to further filter, then use various techniques too anonymize the session, then process it and add it to various datasets for further selection and refinement. they don't need huge storage for this.

The last time I checked there was a loophole — if you provide feedback in-session (responding to “how are we doing” or “which prompt is better”) then they can use that feedback + relevant context. Relevant context for chatgpt might include memories / other sessions. That may not be the only loophole.

That in itself is a dark, dark pattern. There should at the very least be explicit warnings for users who have checked “do not train”; or they should not be presented with such dialogs.

That's because what people enter into LLMs is the last gold there is out there. Everything else is already scraped or ensloppified.

Maybe next step is to filter your input client side through an unknown number of obfuscators where you ask LLMs to rephrase your question (onion router idea) such that no single provider can be certain that this is human input and not some slop feedback loop.

Are you kidding/trolling? The P=NP problem is FAR more fundamental, and if proven true, would basically be a proof that e.g. public key crypto can be broken (NOT a description of how to though).

Basically, it would be a proof that all the REALLY hard (combinatorial) problems out there, have a much simpler solution, if we were able to find it.

EDIT:

NS is used daily in engineering and gas/fluid modeling. We sort of “know it works”. The smoothness proof is “just” formalizing what practitioners assume is true (very coarsely said, no intention to diminish the result!)

It’s a bit like the Collatz function IMO, empirical evidence isn’t proof, but we’ve got a huge amount of evidence for the behavior we’re trying to prove.

I believe P vs NP is a different beast entirely. We don’t even know which way the answer should go.

What makes you think they didn't try?

That one is way more difficult.

Put down the pitchfork. It's a toggle in their settings.

Or, using anonymised data, search for anyone seriously trying to tackle this problem - probably about 10 people in the entire world and use their ideas as a starting point. They wouldn’t even need to be watching specific accounts or using de-anonymised data if they know what they’re looking for.

It seems completely trivial to feed sessions to their own LLM and ask it to look for various things in them, from detecting problematic use cases to finding interesting mathematical work.

The claim that they 'cannot rule out' using Buckmaster's data isn't true in principle. Trivially, imagine that the model had been trained on data only since Jan 1, but a discovery was published on Jan 2.

Whether it's true in practice that's a matter of how their systems' functioning & its current state. If they really care about proving this, it's worth an audit. In fact, if anything, they may be able to prove that it hadn't used Buckmaster's inputs, but wouldn't be able to prove that it had.

I can give you some context. 1. Terence Tao's mastodon explains the way this problem was solved does not in itself contribute much. LLMs (and in this case) produce massive, often unintelligible proofs that do not further understanding. It is often that in pursuit of solving these problems, many other discoveries are made. 2. There is a more serious question about scooping. If OAI is using chat data from researchers to make discoveries, essentially every researcher who chats with an LLM can get scooped. You could be 80% of your way to solving a problem, and LLM could solve the remaining 20%, and get all the credit. Years of your work could be scooped in an instant. If you're a PhD student, this is even worse. Here it's a world famous problem. But imagine you're a PhD student, working on your small but extremely career/progression critical problem, and you get scooped by an AI you talk to. No one is even going to care.

They could explain how they use data from their users "to improve our models".

In the absence of that, how can we make informed decisions about how to use their tools?

(And "you can opt out" is a weak answer IMO, because I can't guarantee that everyone I email about something has also opted out.)

A very simple "these two pipelines don't connect up in our architecture, here's our internal high level network diagram combined with our data ingestion opt-out feature flag that we will stand by in court" as opposed to "yeah, we don't even entirely know how our own customer facing systems are connected to our training pipeline, but it probably didn't happen".

If they have zero-retention, then it is not possible.

So what they are saying is that they don't have zero retention.

If you’ve done functional programming and is familiar with mathematical proofs it should be quite intuitive to read the language

Fortunately, there's plenty of competition between AI labs. Not all of them are even in Silicon Valley.

the proofs were verified in Lean, so unlikely.

But we all know that is code for 'something wonky happened, and it wasn't me'.

Oh please. This whole “I’m not accusing them of anything, but this sure seems suspicious” doesn’t work past the 6th grade. He knew exactly what he was doing.

SQL is so ubiquitous and the use case so obvious, there's no way they have not already been tracking performance and benchmaxxing on SQL queries for years.

But I don't think openai will bother to release a competitor, the real threat is that anyone with a decent LLM and a harness to try a few queries will land at the same or a better query within minutes.

No one wants to “steal” your vibe coded crap. Your prompts aren’t as unique or valuable as you think.

It's extremely unlikely that there is anything interesting or novel in the optimisation of a small SaaS SQL

The answer to this is obvious: the effect on the multi-billion dollar valuation in the imminent IPO.

How is that any different from any other academic research? Every PhD candidate solves problems essentially nobody cares about. They don't even get $1M, they get nothing.

There are two benefits though.

One is recognition. Cred. The PhD candidate gets to put a ", Ph.D." behind their name, opening doors to future academic employment or other endeavors where people value titles. The AI lab gets to say their tech solved sth that humanity wanted bad for a long time. Both cases with substantial financial upside (higher income for Mr. PhD and higher company valuation for the AI lab).

The other one is that this is how scientific progress works. $1M or not. That number was just a PR campaign by the math community to point to some goals. It's clear that it would cost more than $1M to get there.

This has ~never not been the case for any research in any modern sense.

If you want a level headed technical discussion instead of pitchforks and drama you have come to the wrong forum.

OAI's side of the story is that they discovered the approaches (and indeed solved problems - Euler equations vs NS equations) differed. They then offered Buckmaster lead authorship of OAI's proof, without Alpöge. But they never demanded that Alpöge be stripped of coauthorship on resolving the regularity of the Euler equations. At least that's the claim.

https://xcancel.com/SebastienBubeck/status/20973794116915163...

Authorship is weird.

Kind of a tangent, but in some fields authorship is actually more of a formal rank: once you're in the group you appear on every paper. A good fraction of the "authors" won't even know that a paper is being published in their name, and the vast majority haven't read a word of it.

In these fields it can be quite a challenge to not appear on a paper.

Yes, there is no doubt about scientific misconduct. I think the entire math community agree on that.

This article is the tldr.