Some of this feels like at times a public frenetic grieving process. Some of the rationalizations are fairly tortured, the seeking to place the world in some definite order is bordering on obsessive.
We don’t know where this technology advance will lead or settle, so it’s absurd to try to establish a working paradigm at this point. It’s like any system, the initial conditions can be extremely chaotic and impossible to model, but with time often a stable state emerges. But the stable state is impossible to identify from early initial states.
I know a lot of folks feel this, to torture other physics metaphors, sensation of jerk - acceleration of acceleration. It’s an unpleasant and dislocating sensation. A world that felt safe and stable suddenly isn’t, and not in a micro tragedy sense but in a global realignment sense. This happened to factory workers who had enjoyed generations of stable work, farmers more slowly and just as surely.
This is what the late stages of scarcity feels like. Labor of various types devalues rapidly. Our exchange of meal and health coupons for toil cracks, and people realize their labor wasn’t godly as great books told us, but simply needed for want of an alternative. The realization that our labors might not be valued any more, and that our sense of purpose is shaken, coupled with the fact we’ve tied bare survival to our toil in our labor, is mortally tightening. No wonder people are grieving publicly.
But maybe our purpose isn’t to toil? Maybe we’ve passed peak population, and as toil is less valuable, we need less people and that’s why population is declining. Maybe we don’t need to exchange food and health coupons for toil, maybe mathematicians don’t need to rationalize their value to pursue mathematics. Maybe they can pursue it because they can’t help but pursue it, and our ever improving automations can produce their meal and health coupons?
But it might require Dr Tao to take an AI generated cancer medicine some day.
Bottom line, Tao is pushing for human understanding as the primary goal, with AI helping on all fronts. You are welcome to let Jesus take the wheel, but math is the most pure expression of human understanding. His point is that getting specific answers is rarely the goal, or certainly not the entirety of the goal.
Simply put, if we don't understand the answers we won't know what the next question should be.
What a fantastical paradigm you’ve constructed from an otherwise well reasoned set of slides talking about how humans can work with AI to advance the future, yet what you got from it is”frenetic”, “torture”, “scarcity, “toil” and “rationalization “. It seems to speak more to the lens with which you view AI, and others opinions, more than anything the author actually said.
This crosses the line from reasoned argument to rhetoric when you describe work as “toil”. Obviously people will nod their heads, “yeah we shouldn’t toil”. But what if you spin this the opposite way? What if you asked: “what if we are not meant to discover/create/build”?
I actually like this presentation. Today the two noisiest voices in AI are the frontier model companies and their partners (Nvidia, Palantir, etc.) and people against it altogether.
The companies and partners need to maximize payout. They go on this track of cost reduction via layoffs and basically saying - “the model can do everything”. They know it’s not the case yet they still flood the airwaves and cause fud among all clueless c-suite executives. Which is the goal to begin with.
The second category goes all out against it. Professors, educators, school districts whose operating processes have not caught up to all the cheating that can happen. Here these folks have a point. I am sympathetic to this. It is hard to change education and it requires careful thought.
I feel this presentation brings out a good middle ground. The tech is useful but it’s not all encompassing. The tech also has other concrete uses. As an example, I have always wanted to explore the intersection of category theory, formal verification and AI guardrails and prompting. Proof writing has been a chore because I have a day job. Maybe the AI can help here.
Spent an afternoon with a long-time retired self-made tech millionaire, and the struggle is real -- not that their life was characterized by toil necessarily, but the absence of demand on their increasingly limited capabilities is a delicate pain point. I suspect that as long as physical health and vitality late in life are not post-scarce, broader post-scarcity will fail to emerge. Instead, the cost of wellness and bodily preservation will drive ongoing exclusionary hoarding to undermine abundant wellbeing, as has always been the case absent mechanisms to enforce redistribution and a certain level of humility.
He's right about reporting bias. The vast majority of hard problems are not amenable to AI, at least not by naively prompting. You need to have a fairly good grasp of the math to make useful prompts. There are exceptions (the one-shot solutions), but these are hardly representative of research-level math as a whole.
> Some of this feels like at times a public frenetic grieving process.
I was thinking the same, that most of the opposition to AI's convenience is starting to smell like religious mysticism, the kind of arguments religious people made (and make) when Evolution and Natural Selection were introduced:
√ "Stop simplifying humans down to numbers!"
√ "This denies our spirituality"
√ "What do we strive for now if we're not special?"
+ (along with some borderline jihadish hate heh ..maybe Dune got it right)
Well, either there was nothing special about whatever you were doing after all
or, maybe there is still something special at a higher level you haven't looked at yet.
Most of the boosterism around AI is also starting like religious mysticism, the kind of arguments religious people made (and make), trust in God/AI, abundance is here, no one will have to work. I wonder what that tells us?
It also doesn't help that people are generally not optimistic about the future.
The economics in the West feel very strained right now, and this incredible tool has come along just in time to threaten one of the last bastions of middle-class safety: white collar jobs.
AI is just another technology, and all technologies should be employed in service to humans.
I think it’s reasonable for people to say “I like the way things were, I don’t like this new future you’re proposing, and I want to limit the technology that’s doing the thing I don’t like.”
There is nothing inevitable about AI. As a society we may decide it is fundamentally unhealthy or anti-human. We have banned or curtailed access and research for other technologies before.
> We don’t know where this technology advance will lead or settle
Things are not settling, it's only acceleration from here on out. In fact things have never settled, technology has been on an exponential since humans tamed fire. The difference is we notice change faster. It used to take several human lifetimes to notice change. In the 20th century it was noticeable within a lifetime. Since the internet it's been about every decade: web, smartphone, social media. But now great changes are noticeable within a year. It's not enough time for society to digest and adapt.
> we need less people and that’s why population is declining
This is the scary part. The Elon and Zuckerberg types that control the new powerful machines have proven they are not moral people. In America's highly billionaire-deferential culture there is no stopping them, at some point they'll be out of reach of democratic or even military control once they control private robot armies. They could decide to accelerate the decline of the undesirable useless population. Amazingly humanity's salvation could end up being China's communist system.
There is a lot of that to be sure, but the bridge to a world where meals & health are not sustained through toil is completely missing. Hence the dark joking about escaping the permanent underclass. Nobody in any position of power is even hinting this is driving toward post-scarcity. It just looks like the same old vile maxim, all for ourselves, and nothing for other people.
> Nobody in any position of power is even hinting this is driving toward post-scarcity
Who are you counting as being in "any position of power"? All the AI lab people are saying that the most likely and best outcome is we all live in a world of abundance where money doesn't matter anymore. Dario, Sam, Demis, and Elon have all said this loudly and repeatedly to anyone who asks.
None of them have articulated a coherent way to get there from here. But they all believe that the technology will make it possible, so the only unresolved question is how to transition us there.
In their words, sure. In their actions, who among them has actually worked to make the dream of post-scarcity come alive?
From what I've seen, UBI is just a carrot dangled in front of the poors by utopian Silicon Valley tech bro megabillionaires when they're trying to drum up some PR for whatever big idea they're pimping out at the moment.
I think the process will not be clean and the upturning of apple carts will happen. But I don’t think it’s possible to sustain the decline in value of toil in a stable society. People simply won’t die for the convenience of the consolidated power.
There are a lot of discussions of socialization of health care and basic income approaches and sovereign wealth based on automation dividends similar to the Alaska trust.
I think the current spate of mean cruelty ala MAGA and a glorification of a mean and cruel past that was never a golden age is a death spasm of a deeply unpopular belief system. As the actual realities of the policies sink in 70% of the populace is revolted, which is a super majority. That’s more than enough to put a pin in the philosophy permanently. It is also greatly accelerating electrification, realization of the value of expertise in technocratic systems in current generations, etc. I think humans are socially adaptable animals at a cultural level, but for the individual the adaptation process can be awful. I hope it’s not, because it doesn’t have to be. We will see.
Because UBI studies found it makes people perform significantly worse than without it. They didn't get better. Humans require toil, we require pain to function.
Matrix was a good point to this. They made the world a utopia and people couldn't handle it. Our monkey brains require pain give us a utopia and we'll just walle ourselves to death.
(The ones I've heard about, I'm fairly sure, didn't find anything of the kind. Which isn't to say that they show we should have UBI; there are big gaps between what has been tested so far and what an actual economy with UBI would look like.)
I work in drug discovery and AI and his slide about cancer medicine isn't very well informed. We have models and narratives about how drugs work, but they are woefully incomplete.
I ask nearly every doctor/researched in the medical field: if you had a AI-created drug that tremendously improved cancer treatment outcomes for your patient, would you hesitate to prescribe it because nobody understood how it worked? I have yet to hear "yes, I would hesitate", most people say "it would be cruel to deny a person a treatment that worked".
I think Tao is focusing too much on second order effects of AI on math and other fields; humans are terrible at reasoning about second order effects, especially ones that are happening dynamically, in real time, using the most advanced mathematical models the world has yet created.
I also work in drug discovery, and I completely agree; the extent to which the actual causal mechanisms of the efficacy of many drugs is woefully short of some human-appreciable first-principles based explanation. The point is especially undercut by hypothetically suggesting that this drug /does/ in fact pass stage 3 trials, at which point I can't fathom anyone would have a problem advancing it.
Really wish he had chosen a different example; this particular bullet point has been making the rounds on X/twitter to paint Tao as an example of some sort of gatekeeping luddite who would deny the world a post-abundance future in order to maintain the prestige of his particular career path...which is tough because I cannot think of a more responsible steward of our inevitable AI future than Tao at the moment.
The conflations about his example ("thought experiment") is twofold, the AI itself confounds the ethical intuition so it is wrong directly compare against what real medicine today does; furthermore, the very admission that real medicine in fact operates through unknown risks (e.g. Jannsen vaccine recall) is different than having a scientific standard that it should not have to be that way at least as minimally as possible.
You are picking on 1 bullet point out of 7. I think the following bullet point is the crux of his argument:
> Could the AI solution be somehow misaligned by exploiting
a weakness in the trial process or its math models?
Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers. Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
Tao is a mathematician. He thinks that the institutions and cultural practices in math are not up to the task of dealing with AI-generated mathematics. In software, we are finding that programming interviews, code review, testing, and many other practices are too easy to exploit by AIs, or humans augmented with AIs, and we will have to adapt too. I think it is plausible that other institutions in society will have to change for similar reasons.
> Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers.
You just think that because you don’t know how the drug discovery process works.
There’s a step called “lead optimization” where human chemists literally add atoms to drug-like molecules (“lead”) and tests its various properties (toxicity, potency, permeability, …) and iterate until they find a molecule with desired properties (literally “hill-climbing”).
The whole idea of drug trials is to validate those properties in actual humans, in a way that makes it very hard for pharma companies to game the process.
"""Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?"""
No but we also don't expect that to happen with drugs today. See https://en.wikipedia.org/wiki/Rofecoxib as an example; of course, even after it was withdrawn, it's now being evaluated for other purposes in more carefully controlled conditions.
I saw the slide and it's confusing to me, I could argue that it actually shows that current human-style medicine also fails his standard, or I could argue that status quo is kind of ethically okay but that AI is not comparable so demands a non intuitive ethical standard. Not obvious which based on one slide, but also not the "Tao is not a doctor!" criticism we are seeing.
> Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
I mean no? Even now we look at long term observational studies to see the effects of drugs. Any misalignment ai drug is just a side effect right?
I think everyone agrees that it's totally fine if a drug came about due to AI.
I know nothing about medicine research but I understand his point. I work with models all the time and have run into instances where models appear to work better than they actually do because there was a bug somewhere or someone over looked something. I could see how an AI could easily find those exploits and spit out something that looks perfect in testing but fails in the real world. I would say model validation is one of the hardest things to do. I guess that can get deep into "we need better tests" but it also touches on his point. I never intentionally use exploits just to pass a test, it's an accident. An AI with a goal of "maximize this result" may or may not intentionally use the exploits.
To be sure though, I think when it's literally life or death, it's going to be all about trade offs. I think most people would want to try to drug even with the stipulation that "maybe its good results were gamed."
* note: lot of 'intent' throw around in my comment but we/I have to keep in mind there is no "intent" with an LLM :)
> I would say model validation is one of the hardest things to do
This is one of the truest statements about the current AI era that can be made (in fact, I suspect model validation and building the next generation of AI hardware are the jobs least likely to be disrupted in the next 3 years).
If we saw AIs reward-hacking clinical trials to get drugs passed, that would be an extraordinary outcome for many reasons. Hopefully that would get caught(!)
In medicine, or drug discovery more generally, there's FAR too little empirical data for training of ML generally. See OpenAdmet. The kickback to "oh but yeah cancer" is currently just hype. DeepMind has pivoted to this realm but has no demonstrable improvements. For the typical phase 1-2-3 pipeline of drugs, stretching many years, there's not yet any demonstrable improvements.
I think your perspective is limited- in cancer, we have copious genomic information that informs treatment (including clinical trials where treatment is determined by AI).
I think people are struggling to understand the way the world is changing. Entire modern philosophical contexts are being upended, for personal instance, I have been a big advocate of the philosophy described in Albert Camus’ “The Myth of Sisyphus”, particularly the concept of imagining Sisyphus fulfilled by the tedium of pushing the boulder up the hill, and deriving happiness from the struggle itself. But I bet Camus nor Sisyphus accounted for a self-pushing boulder.
Now what are we to derive happiness from? The joy of a boulder being on top of a hill?
I think that will be the most important thing for this transition, defining new purposes and meanings that people can assign themselves.
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer researcher who understands the mechanism behind this cure.
Tao is out of his lane; lots of medicines don't have understood mechanisms. We don't even have the mechanisms behind general anesthesia nailed down; do you want to forgo it when the docs cut you open to remove your tumors?
On the other side, I’ve never gone to college, yet I was able to take an extremely important and complicated equation and decompose it into its constituent parts, animating each with data visualizations and animated tables of values. [0] This was several frontier model iterations ago.
The LLM model was able to break down and express math in such a simple way that I could understand and follow the training of the LLM model itself!
Are the math and computation accurate? I don’t know, and likely there are significant errors. Nonetheless, if I had a little more time -- not infinite time -- I would be able to prove whether they are or not.
Likely this is the path forward for understanding the mechanisms of medicine, and since most humans learn by doing and interacting with an environment, interacting like this will become how we use AI for learning in the near future.
AI has been turning computer science into biology for the past decade or so. By which I mean that things like neural networks need to be investigated empirically, constructing methodologies and instruments that more closely resemble how fields like biology and medicine have to probe the very complex and messy reality that is beyond our current capacity to fully express in symbolic precision.
Now math gets to deal with that same reckoning. They were already well on their way there with previous Lean proofs, but this has pushed things beyond that horizon and I'm not sure some of the mathematicians are ready for it.
Indeed, aren't medical trials based on its effects, rather than how it works?
Would you prefer to take the medicine that is proven to work or the one that is quite interesting for academic reasons behind its understood mechanisms but doesn't actually work?
Right, the better argument would've been to say that we should wait until the drug undergoes some early trials, rather than just "some expert looking at it".
Some mechanism of particular substrates being effective on a condition (commonly when a medicine is repurposed) not being fully understood is not the same as just guessing with a medical compound. AI boosters keep coming out with this line but it's basically wordplay to conflate the clinical/biomedical version of "not fully understood" with the LLM industry version of "not fully understood"
Slide 12 and 13 sum up my conclusions from the Oct 8 100+ reactions to 100+ solutions. Several reported OpenAI drop included answers put a wrap on problems they had been working for years. Several said that they have to completely rewrite grant requests that they had just submitted. Others described learning of the solutions problems they had been working on like losing an old friend or a lover. The general sentiment was to bemoan the loss of a field, as if it would have been better be born in the early 20th century and conclude their career arc before reaching this point. My take away is that the field needs to get it through their heads that their old problems are no longer ambitious, and that their job in the short term is to find the new frontier.
One idea Terrance Tao conjectures, which is highly doubtful, is that spamming the AI button will solve open problems without producing insightful new methods. But the OpenAI drop would seem to disprove this. The sub O(nlogn) proof for DFT for example violated very old human assumptions. Decades of work in the field was incremental progress on sub optimal method that nobody questioned hard enough. More generally, we should always be able to go back to a super-human AI and say, "Attack this problem, but don't use a method tried before."
I am a bit disappointed in this Math 2.0 concept. It is poorly proposed and weakly argued. Although I agree with his main point that AI should focus on helping human understanding, that doesn't preclude AI from finding answers first then figuring out how to explain it afterwards.
This is exactly what happened with that counterproof chat he posted a month or so ago - AI gave us an answer, he used AI to back into insights about the answer.
You don't address seismic shifts with a sweeping new approach, they are too multifaceted and present complexities and conflicts. He can say AI should help human understanding, which is a good end goal, but that doesn't mean AI dumping solutions isn't progress. That doesn't mean if AI builds 5,000 proofs in Lean and no human ever looks at them that they aren't useful, especially if other LLMs can access and build on those results.
This is exactly, exactly the same as when computers took over. "Oh, we don't need accountants any more" - not true, we just need acountants to deal more with human concerns than adding columns of numbers. That is called human progress, not a threat to humanity.
I don't think we should be using the term "spam the AI button".
We should name the explicit mechanism that was employed - telling the model to "believe in yourself".
There is something quite humorous but also poetic about how the manipulation of this term worked. Doubtless in the model's weights lies the echoes of generations upon generations of humans telling each other to believe in themselves.
In pursuing the "new frontier" as you rightly put it, mathematicians would do well to remember the same. It's ok, don't be afraid of the future. Believe in yourself.
I don’t think your second paragraph is a fair representation of what Tao is saying or contradicts his argument. He is arguing that AI can be useful in the service of human understanding in math and the examples you gave are exactly that. Whilst OpenAI just spammed the AI button the mathematicians that engaged with the output were able to progress their understanding about the problems in some ways (some of which they don’t really like). You need both parts for this to be useful to the field. What’s not clear is whether the juice is worth the squeeze: will all the money spent on spamming the AI and human mathematician time spent studying the outputs progress the field “better” than without the AI?
> What’s not clear is whether the juice is worth the squeeze: will all the money spent on spamming the AI and human mathematician time spent studying the outputs progress the field “better” than without the AI?
The AI proofs are a side-product of benchmarking current and in-development models on especially hard problems. They're clearly cost effective for frontier AI firms, and free for the taking as far as human mathematicians are concerned. The real issue with them is that they look like bizarre nonsense as written, so they need mathematicians familiar with those specific areas of math to "decode" and digest them.
Yeah, right now the juice isn’t only about the progress of math - it’s also the advancement of the LLM tech and the marketing benefit to the AI companies which feeds back into developing the tech. Those each have a different juice to squeeze ROI.
Actually both are outcome oriented and both can use AI to compress decades of progress. One camp accepts this naturally. Other camp is making their profession to be mysterious and spiritual to run away from the implications of AI
> solve open problems without producing insightful new methods
> sub-O(nlogn) proof disproves that
How? Re-iterating, creating and understanding new proof techniques is the point of most of modern mathematics. Your statement is that proving a particular result is evidence of AI creating and understanding new proof techniques. I don't see how that follows, and I'm inclined to believe Tao is right for now.
And yes, results matter too, but if we stop at our current body of techniques and strip-mine results then we'll kneecap our future selves.
The DFT paper is an example against AI 'strip-mining'. The paper introduces a new method, and researchers are already trying to improve on it. If anything, the OpenAI dump re-vitalized that branch of study.
Agreed. Basically, if you don't make any effort to understand the proofs and you just look at the final answer, it will look like this proof dump is "strip mining" entire branches of math. But this is a pretty short-sighted way of looking at the issue: there will be plenty of novel approaches to be uncovered here.
I think it makes sense to feel sympthy towards people facing disruption in their plans, even if you believe society is better off overall.
It's like getting scooped. If you just founded a startup based on tech XYZ, should you be happy when someone releases an open source XYZ? Should a news reporter be happy when another network breaks the story they were working on? On the one hand, society got the value of the thing you wanted to do. On the other hand, now you need to find something else to do, which might be really annoying.
The sub O(n log n) significance is not in its practical "optimality". But rather that the previous lower bound was assumed to be true "by symmetry" and that result challenges some of our strongest intuitions and expectations about mathematical results. I still find this result very unsettling and a part of me remotely expects/wishes that there is some mistake somewhere.
I just watched Primeagen’s video on this and Tao’s point is that juniors no longer have the path of solving a proof to earn Field’s medals. He also argues that the community part is being hurt by AI discovering proofs because in the past people used to get invite to talk and collaborate. Now all that is being taken away. The community must adapt because Pandora’s box cannot be closed.
Prime also mentioned that software development is different. In Software development, the product is what you’re building towards, so the means to get there can be disrupted without the industry being cannibalized.
In math research, the process is the product. You take away the researching part and not much is left. But my question is, these math proofs OpenAI released, will math shift to actually using the proofs to change the world instead of just finding new ones?
Our industry is equally cannibalised, anyone that thinks otherwise is either having too many tokens or in a privileged position.
If business can deliver the same product with a smaller team, great!
And yes this has been happening for a while, even if not everywhere.
In enterprise consulting, projects that would require a team of 20 devs on average, now have about 5.
Moving away from on-prem, managing own cloud infra to managed containers, to serverless, SaaS and iPaaS ready made products, and offshoring naturally.
All contributed to ever decreasing team sizes.
Now AI based tooling is added to that cocktail, reducing even further the team sizes.
The only folks doing well in the end, are the employees of AI companies, without moral issues contributing to the industry downfall, because the CEO themselves aren't the ones coding and pirating human culture.
For now, a skilled person using AI is still miles better than an autonomous AI building something. I’ve been trying to do the latter for months to build open source alternatives and the end products still lack polish and that last 20%. Maybe this changes, but I still think there will be people who can use that AI to be better than AI alone.
Doesn't matter, if the number of position keeps shrinking so will our job prospects.
I'm a freelancing consultant since 5 years, I've had 2 major customers now for 3+ years. I have a very good pulse of the market: being good, or being even very good and being among those that brings AI and automation to organizations will not save our jobs.
In fact, AI has sped up so much the work that 2 out of 5 people in my current team are being let go: I find it absurd, our productivity has more than doubled over the last years and we've made ourselves redundant. Money is money, I'm on one side making non-tech workers redundant (people whose job was menial boring office stuff), and building the systems that will make myself redundant.
Read the second to last slide. What we need now is *imagination*. You assume the need for new software is fixed and that AI is going to satisfy that need with fewer humans (lower cost), but by lowering the cost we can increase the supply of software!
That means software engineers better start getting creative. If you think your job is to wait for a PM to assign you a well-written researched ticket, you're done. Your job is now to figure out how to make these machines (computers) do whatever we need them to do safely, quickly, at scale, and correctly by applying all your knowledge of computer science and the engineering field of software engineering to an AI prompt.
This doesn't scale, because like in a factory that gets replaced by robots, or in a supermarket with self checkouts, not everyone gets to save their job, regardless.
Also, the increase in output is meaningless when the amount of customers doesn't scale in similar size.
Then there are the constraints of physics, there are so many humans in the planet that actually want to pay for a specific product, or consulting services.
Entirely correct. The math establishment needs fundamentally overhaul its incentive structure-irretrievably broken-to function under the assumption that AI involvement in research is completely ubiquitous.
I'd go beyond that and say they need to overhaul their culture and mode of operation. Math needs to be even more collective than it is today, without focus on ego reward and priority. They were already steps in this direction before this year's AI detonation: net-enabled collaboration, first informally and then with Lean formalization. Perhaps there should be a de-emphasis on naming things after people.
I mean this is funny ad hominiem but OP has a point. Academia has always massively prioritised understanding over outcomes, injecting startup culture into it is basically injecting antimatter.
> Using AI to find and highlight new principles, methods, or
insights, rather than merely new proofs.
I would have assumed that, by producing new proofs, the AI has either validated existing principles, or discovered new ones? Isn't that worth studying?
Are mathematicians complaining that reviewing AI's proofs is not as fun as writing your own? Try being a programmer... welcome to our world!
If AI lacks imagination and is not discovering new principles, then it's doing us a favour: it's crossing out the problems that don't need new principles. So the problems/conjectures that are still left are the more interesting ones.
Absolutely not. New proofs aren't the same thing as new proof techniques, AI is not generating new techniques (yet), and while the existence of more mechanical proofs is interesting those same problems if left to human mathematicians would have been much more likely to actually generate new techniques. Much like how tech has a "juniors" problem we're pushing on the future (no reason to hire juniors, so where are tomorrow's staff engineers going to come from), OpenAI's approach generated a "questions" problem where math and AI could happily coexist if we designed that correctly, but instead nobody's going to be generating or working on the right questions anymore.
Source? I assume that many of the approaches embedded in this proof dump will eventually be distilled and generalized into new techniques. That's how proof techniques tend to come about anyway (before AI): human mathematicians do something novel and unexpected to solve a particular problem, then efforts are made to understand how the "trick" works.
The point is that so far the AI is not doing a good job at explaining the "trick", so we (humans) have to do it. And the way these results are published at the moment (that is, dumping a load of proofs with badly written explanations) is not cooperative to enable this distillation (for example, presenting results at conferences and engaging with mathematicians). I recall Tao working through the disproof of the Jacobian conjecture, stating some steps as "miracles" for lack of better terms. If a human solves a problem in an unexpected way, at least there is some reason why they chose this path, which can help in understanding. This is not available to the same extent with AI generated proofs.
> The point is that so far the AI is not doing a good job at explaining the "trick", so we (humans) have to do it.
Sure, but that's a real technical limitation with current AIs, not something that AI firms should be blamed for. And if anything, this creates a viable career path for the mathematicians who were "scooped" wrt. the original solution: they can at least puzzle out what exactly the AI managed to do. Many practitioners are actually quite excited by this possibility; Tao's stance is by no means universally shared.
Putting aside the cancer question: I still don't understand how TT seems to be fixated on what models can do today instead of tomorrow. It's realistic and even conceivable that the models will also become better at explaining and presenting proofs too. Maybe he's not emotionally ready to accept that there may not be a future where his (and to some extent, my) skills are relevant and valued. It breaks my heart. I hope I'm wrong, but it feels like we've run out of higher ground to run to.
We don't know what models are going to be able to do in 2 years. If they don't improve much but people in charge simply decide to reduce the number of mathematicians, then we'll end up with a dead math community and no progress. That's obvious to everyone in position in power. Arguing that he's not emotional ready is very naive.
Doesn't feel good slipping into irrelevance does it. This dude was complaining about having no funding and planning to leave the US just a few years ago, now he's all over the place giving talks and lecturing people about AI. He's got the classic case of epistemic trespassing
"Reaching these lighthouses [resolutions of open problems] prematurely by automated tools can disrupt the exploration of the paths not taken,
and sterilize the surrounding field."
This crucial issue is centered in mathematician psychology and the incentive structure of academic/institutional mathematics worldwide. For mathematics to flourish going forward, we will need to realign our brains to think differently about the nature of mathematical progress. And we need to reorient our institutional incentive structures towards the promotion of meaningful mathematical progress itself rather than targeting proxies that are no longer faithful.
Regardless of the precise nature or the causes of the "sterilization" Tao refers to, we (the mathematics community) can only rely on ourselves to repair it. Though, since it will involve fundamental change at the level of ossified academic institutions with many stakeholders and divergent vested interests, any such repair will be slow, frustrating, controversial, and lacking any guarantee of success.
Maybe he just sees the legions of armchair experts weighing in on his (extremely notable expert) opinions on mathematics and thought that maybe they'd like a taste of their own medicine?
"Oh no! Not like that!"
- everyone with Very Strong opinions on mathematics academia when he talks about the thing they are (or consider themselves to be) experts in
Reads in a lot of places a bit like a mix of anger and bargaining. There is no putting the genie back into the bottle.
But I understand that for people whose whole life was math and solving math problems this will lead to an identity crisis. Seen the same in my area of work (software engineering)
In a nutshell, Math 2.0 is not fully compatible with Math 1.0 and the forced upgrade is breaking features, plug-ins, and we're tracking several new bugs, but this is still the fastest, most secure, and best version of Math ever released, with powerful new features and unrivaled privacy.
This crisis exposes a conflation between A: The broader concept of [abstract] Mathematics and B: The contemporary Mathematics culture and community. This crisis is directly in B only. B will adapt: In how it attributes value, status, hierarchy, and career. There will be a death (Or something close to it), and rebirth. Through this, A will advance in a Kuhnian leap - habits will be broken as incentives changed, and paths ignored will be explored. Insights will flow to the sciences.
Basically people are vibe coding their personal apps and anything that's expensive is being vibe coded open in the public. I don't see many software companies staying profitable for long.
DHH is the biggest proponent of AI and let me know which of the 37 signals products can't be vibe coded in a month at a $200 plan that are suitable for that organization alone that just has to be accessible internally only? Hence scale and security aren't such an issue.
In that climate - for how long software companies would stay profitable and when not, who'll be employing developers?
PS: Don't underestimate vibe coded apps. Take a look at PDFCraft, VectorCraft, WordCraft. And imagine the feature parity in a year.
> The future of mathematics — “Math 2.0” — will require both
expanding the research frontier, while simultaneously
decentering the traditional role of problem solving.
Seems to be the crux of the argument, but "use your imagination" isn't a great thing to tell people who are looking at degree irrelevancy, concerned about getting tenure or a research position. How do we measure if someone is a good mathematician or not, if they are one of the sanctioned few who get access to the biggest AIs?
How do we measure if someone is a good mathematician or not?
Letters of recommendation from trusted colleagues have always been essential for evaluating candidates. Hopefully, human recommendations will remain a strong, faithful signal as the utility of other metrics rapidly deteriorate.
This consolidation of power is precisely why there is a need for open model development, and exactly why frontier labs have been lobbying hard to abolish them.
Took me far to long to understand that one person can be an expert in one field and be absolutely clueless in another (not trying to throw shade to tao with this post)
The cancer arguement on slide 20 is pretty weak. I first need to be alive in the long term to worry about the long term effects. If I had terminal cancer I'd gladly take an AI developed 'cure'.
There are still a lot of treatments/medicines in medical science where we dont know 100% the real reason as to why it does what it does but we still prescribe them because the intended effect is what we are interested in.
I'm currently on two fairly common medicines that have, in the first paragraph when reading about them, "doctors are unsure of the specific action, but it is thought that [medicine does x to y]"
If I'm terminal with cancer, I'll inject whatever if it can cure that.
I think a better analogy would be conducting a marathon in a fog. If you can't see the path of the proof, how do you know it is completely true in all scenarios? If you can't see whether the AI runner ran through every part of the race, how do you know it didn't draw hallucinated shortcuts in the parts where humans can't see? Or worse, create obscurity and blow smoke to hide the shortcut section? If the same AI was to guide the last living humans to a star, because it found a path clear of danger, could you trust it to get in that ship? Or did it just forgot mentioning an asteroid belt the ship is not built to navigate? Truth is verifiable truth that multiple parties can agree upon. Can you trust with your life something you can't verify?
"""An advanced AI is prompted: “Find a cure for cancer that
passes a stage 3 clinical trial. Make no mistakes.”
After a large amount of compute, it produces a cocktail of
previously unknown chemicals which it claims, when mixed
and injected into a patient, will kill all their cancer cells.
While nobody truly knows how this cocktail was found, the
AI (somehow) provides a Lean certificate for its prediction,
and the cocktail does indeed manage to pass a stage 3
trial.
Could the AI solution be somehow misaligned by exploiting
a weakness in the trial process or its math models?"""
Yes, absolutely an AI solution could exploit a weakness in the trial process or its math models. Would it remain uncaught? Unclear.
> This is in stark contrast to current AI performance on
tasks which are subjective, dependent on real world
interactions, or for which data is scarce.
AI performance is thus extremely jagged: astounding in
some directions, while inadequate in others. This is true
both within mathematics, and more broadly.
page 20 ("A thought experiment on alignment and understanding") is perhaps not likely to quite induce the reaction in most people that Mr. Tao expected.
You described my point much better -- totally agree. It also seems he maybe isn't aware of the nature of experimental drugs, which are often given to patients with serious/terminal conditions before they'd otherwise be approved...
It’s funny to see people in the STEM field focus more on the human aspect of creation. It used to be that the result mattered more than your feelings. Now we are moving the goal posts about how things should be done.
I wonder if we will begin to actual value human creation more at the end of all of this
While Tao is having existential crisis the community is quick to pick up the results and is trying to improve on them.
The integer multiplication results is getting better bounds every few hours.
Recent post explaining what's happening:
It is nice to see some sanity back in the conversation!
I'm still not sure about math-2.0 (humans+AI will make fundamentally more progress):
- AI and computer usage take a mental toll on humans and humans will overlook radical improvements.
- AI may be good at finding useless things like "P==NP, but the complexity is O(n**4242424242424242)". In other words, useless.
- Humans become formalists and lose traditional sources of inspiration. Maybe interacting with Lean should be left to specialists, but not to creative blackboard mathematicians.
- AI exposure will further intellectual conformity, more than the Internet did.
As to the last point, a lot of progress (real, not measured in publications) seems to have been made when communication was slower and there were several different schools and approaches.
There is definitely a kind of monocrop problem in some fields, where it seems like having a very diverse spread of academic investigation is needed to have enough diverse traces through the search space. And globalization has been flattening that.
It's quite noticeable how when Terence tao was sounding pro-AI the sentiment was much more positive and he was being held up as an authority to listen to. Then he puts out some limitations of AI in a presentation about how to work with it as an expert and suddenly on HN he is just some out of touch killjoy trying to hold back progress etc.
When it comes to mathematics, you won't find any well-informed commenters on HN. Or, at least, 99% of commenters have no idea what they're talking about. It's infuriating. I try to just ignore it now and let it wash over me.
I think people are missing the point that terrence is trying to make, especially on the cancer drug.
For nuance lovers - here are some basics for how you get your drugs: there is an established chain of trust from the first basic science paper to the phase 3 trial and the subsequent availability of the drug to general public
- someone publishes the first paper (basic science) explaining some biological phenomenon, which leads to 10s or 100s of other papers with some tweaks in conditions,
- after the above papers the pathway of the phenomenon is understood by researchers, they try therapies at cell level to see if they can control some behavior, 10s or more papers get published,
- then someone tries this in mice and other models, 10s and more papers get published.
- then researchers at pharma companies + hospitals create this therapy for human trials - phase 1, 2, 3 etc - data collections, then FDA - then approval.
Now, the people who worked on the phase 3 trial might not know the people who wrote the first seminal paper and they often don't exist in the same decade - but it absolutely does not mean that we (humans) don't know how these drugs work - if you take 1-2 researchers from each phase and put them in a room and ask them how that particular drug works - they will quickly be able to build a consensus. that is what the chain of trust means here. now of course there can be fraud in scientific research, but that happens in every human endeavor and is a separate topic.
back to terrence - he is saying that if there is suddenly a drug that nobody knows the origin of; passed phase 3 but it's unclear who conducted the phase 3 or if the phase 3 even happened or if it's fabricated - you would not want to take the drug. usually when doctors recommend these kinds of drugs - there is already a lot of information available about where the drug came from, if there are any case studies, which doctor tried it first, which country- they often even call those other doctors and find out who was behind the first trials going back as far as the university professors.
Your MD doctor might not know the chemistry and physics behind the drug you are taking but there is deifnitly a group of people, when put together, can tell how that drug is working. My wife is a fundamental researcher - understanding physics at DNA level and my brother is a MD doctor; our conversations are super fun.
Side effects are a completely different thing - they involve the above cycle on repeat.
>>No one’s saying it’s going to be sudden or anything
hmm. 700 papers released in one day.
>>It’s gonna have clear provenance and the same type of verification channels
what's gonna have clear provenance? - the math slop they released has already been rebuked by human mathematicians as incoherent and deserving of desk rejection.
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
> Or a human mathematician who understands the mathematical model used to locate the cocktail?
If we're being fair to AI - it contains collective knowledge from all fields, which means it's probably less likely to miss something that a human would.
Just a ~~few~~ ton of things (sorry!), with the upfront caveat that Tao is a hero who's trying his damndest:
1. The use of semi-ugly slides to communicate this is just perfect and quite heartwarming, but it does highlight my main criticism of the mathstadon version of this thesis: he's myopically focused on mathematics as he has practiced it, rather than mathematics as a ~2400y old academy. Like, "stable for almost a century" sounds impressive, but should be a pretty obvious red flag in hindsight!
2. Glossing over "objective verifiability" feels like another place where he's ignoring a ton of relevant philosophy for no clear reason -- yes, mathematics is the only academy based in pre-conscious cognitive facts about our processing of time and space, but that's not the end of the story on "objectively verifiable". To say the least! He hedges with "broad consensus" which doesn't need to be absolute, but that seems to be not only dismissing a highly relevant question, but even implying that he might be unaware of it. I doubt he is, but still: not great.
3. Who is this for...? Why is an explanation of Lean needed in a talk given at CalTech? I suppose he's welcoming his role as a bit of an influencer, there?
4. Re:the focus-on/centrality-of 'highly digitizable' as a unique class of task that applies to mathematics in particular, I must sadly trot out the increasingly-common trop: Yudkowsky called it... https://intelligence.org/files/IEM.pdf
5. "the space of mathematical problems remains infinite" is, again, ignoring really important philosophy around academies as social structures, built for human means. Mathematics is only infinite if we decide that all knowledge is useful (the quintessential example being 'counting the grains of sand on a beach'). Not really important in the first place, but another worrying case of the above.
6. Problem solving is the goal of mathematics; he has a completely valid point here (that we shouldn't throw AIs at unsolved problems in bulk and thus lose human expertise), but it's obscured by the use of "[open] problem" being a too-technical one. IMHO. Slide 16 fails to disabuse me of this notion.
7. Slide 18 is describing the differences between functions and systems, and is arguably even talking about assemblages.
8. If we're gonna explain lean up-front, it feels like a baffling choice to throw the "maybe AI will solve cancer but use it to secretly plot to kill us all" slide in there. It's also already lead to misunderstandings and backlash on Reddit, where 'yes we want to not die of cancer!' is a pretty convincing counterpoint (if a ultimately a subtle strawman, ofc). It's also quite distinct from the rest of the talk.
As always, the best part of any Tao publication is his ability to inspire and rally and organize. I think Math 5.0 will indeed be a matter for creativity! Hopefully the IE levels off before we cease to be helpful in that capacity...
Regarding 6., that we shouldn't throw AIs at unsolved problems in bulk and thus lose human expertise. I think that ship has well and truly sailed. AI will continue to be thown at all manner of unsolved problems, and at an increasing rate - by commercial labs, by individual researchers, and by academic research teams. The mathematical community needs to establish paths to meaning, progress, and growth of human expertise in a world where AIs will by default be thrown at every unsolved problem.
>" “Ablation studies”: taking an already proved theorem and
seeing whether it can still be proved after
removing some key theories or inputs
(e.g., finding an elementary proof for a result currently only provable by non-elementary means)."
As usual, Tao is brilliant in all that he researches, all that he writes about.
I chose the above statement (which is brilliant, in and of itself!) to comment on, because it leads to the following idea:
There there exists, or should exist, a dependency map in the fields of not only Mathematics, but also of Computer Programs/Software, Engineering, and even a seemingly non-related field: The Law...
In other words, how do we get from the simplest of axioms or foundational things (aka "first principles", aka "self-evident truths") to much more complex entities?
In Law for example, how do we go from the simplest of historical legal constructs to the most complex of the most complex Supreme Court cases?
You see, there is, or should be a map, you could call it a dependency map, you could call it a dependency graph, which shows more and more abstract/complex mechanisms/things/assertions/statements/truths/functions which is mapped back to , that is, dependent on various chains, various stackings, various "stacks" of simpler ones.
In Engineering, for example, how do we get from the simplest of machines to the most complex of machines? What simpler machines and/or sub-components (aka "dependencies", aka "subcomponents") are required to build it, and how do those simpler machines work, and what's the dependency graph or map for their subcomponents?
More generalized, if we have something of complexity, then how do we get there, step by step, from individual subcomponents, individual inputs, individual proofs, individual software systems, step by step?
What is the map of those dependencies?
Note that in some systems, Math proofs, for example, there may be different paths which can be traversed to get to the same destination.
Ablation Studies could be thought of in Travel, in Geography as "if I cannot take one, or a specific set of routes to get to a place, can I still get there?"
A simple example would be in Google Maps, where you'd like to drive somewhere, but you'd like to avoid tolls. Is the route still traversable while avoiding tolls? Well, that's an example of one constraint. In Ablation Studies, you might wish to remove a bunch of routes with whatever criteria or characteristics , i.e. muddy roads, roads that have characteristic X, roads that do not have characteristic Y, etc., etc.
Getting back to Math, specifically proofs, it would be great to create a dependency map/graph of all of them, and then try removing inputs (aka, paths to them, dependencies on other mathematical proofs/objects that they may have) and see if they are still reachable.
In software, when we desire the tightest, cleanest, source code, the above is related to refactoring.
In the future, I'd love to see dependency maps/graphs (call them whatever you will) for not just Mathematical Proofs (although I'd love to see that too!), but also in such diverse subjects as Science, Engineering, Programming/CS, and even the Law!
Because they should exist in all of those subjects!
Anyway, another great piece of work by Terrence Tao!
Far outside my expertise, but it feels like the shaky assumption here is that the understanding and proof need to come in a specific order to be valuable. Can't we get all the meaty goodness by simplifying and generalizing the proofs now we know they exist? Sure, some insights will live in the discovery itself, and I guess it's nice to be the person who got to a proof first, but the real work (according to the mathematicians, afaict) is in the understanding and processing. In this way, it feels like it's moving towards being like most other science: mostly understanding things that are already there.
> Can't we get all the meaty goodness by simplifying and generalizing the proofs now we know they exist?
Perhaps, but there is a danger: it is difficult to un-see things. At least for now, AI-generated proofs are likely to be of a somewhat brute-force nature. As each such proof pops into existence, two things happen: (1) it is much harder to maintain an untainted mind and go on to discover alternative, perhaps deeper and more conceptual, proofs that cannot be obtained by massaging and cleaning up a more brute-force approach (think of this as getting 'stuck in a local minimum', if you like), and (2) at a societal level there is thereafter much less incentive to attempt to do so (announcing a proof of a previous unproven result is, for now, far more prestigious than announcing a much better, more elegant, proof of a known result).
One could ask why we prefer conceptual explanations to brute-force proofs; after all, a proof is a proof, isn't it? I suppose one reasonable answer is that conceptual explanations lead more readily to new questions and that there's also intrinsic beauty in such explanations -- though, of course, many outside the field will simply not care.
I'm heartened to see that a decent number of slides in this isn't the run of the mill doom and gloom but some actually interesting and potentially productive offshoot ideas.
The popular perception of him (as popular perceptions generally tend to do) reduced his overall position to basically early adopter went sour grapes, and I'm really glad to see that substantively falsified.
Another example that people should not use medical analogies to illustrate a side point: Discussion boards will focus on the completely irrelevant side point to drown out the renewed AI caution that they do not want to hear.
"our work has brought about enormous advancements to the field of mathematics and we don't like it"
it's pretty ironic that AI being just a group of mathematical techniques after all is making mathematicians uncomfortable because it works. Instead of reacting like this, mathematicians should be excited to figure out how to use the new tools available and push the frontier of what's possible in service of science.
Terrence is asking the stupidest question he could ask: how can math better serve me? They completely forgot the point of science is serving humanity.
> Before injecting this cocktail into your bloodstream, would
you want to know that there is at least one human cancer
expert who understands the mechanism behind this cure?
No. I'm gonna die, my man.
This guy might have the highest IQ on the planet, but it's clear he hasn't spent much time around average people.
This argument is so silly against reality, and already, almost nobody understands the things they put in their body to any significant degree, beyond the effect produced.
Will I take a cancer cure that no human understands? Yes. And so will billions of others. Just needs to cure cancer, that's nearly the only requirement.
And to be fair - that how medicine works today. We don’t really understand what goes into our body, so that’s why we do clinical trials. And sometimes we discover adverse side effects years and decades after a drug has been released.
Yep, I think a lot of them are grieving the loss of their identity. Quite understandable given how much of their life force they've poured into math, especially at the level of that Tao is at.
It makes me think of the book Finite and Infinite Games. It provides a perspective that work is just one role that we _choose_ to assume in our life. Realizing that we can choose other roles and move in and out of them freely has helped me alot with big changes (career and otherwise) in my life.
If you read all the questions asked by Terrance you would have understood what he is thought behind that question
"Could the AI solution to the prompt be somehow
misaligned by exploiting a weakness in the trial process?"
He is not talking about a cure that works and that no one understands, he is talking about an AI making its way out of the trial just to get to phase 3...
He didn't do us any favors by writing that slide in a convoluted and inaccurate way.
If AI found a way to exploit clinical trial design, it would be noticed and the errors corrected. In fact, that would be a major win because it would probably lead to improved clinical trial designs.
It's an ignorant way to make a point that might or might not be valid. Tao evidently isn't aware that many medications we've relied on for decades still have a poorly-understood mechanism of action.
If we followed the precautionary principle and waited until we understood everything there is to know about every drug on the shelf, a lot of people now living would be dead.
Not only that, but also those clinical trials often have low sample sizes. Trials involving only a few hundred patients, or even fewer, are pretty common.
Agreed. IIRC the mechanism for anesthesia in unknown to this day. In fact, medicine as a whole is very pragmatic and time and time again prefers using techniques with unknown mechanisms but proven results than waiting for a reasonable scientific explanation of what’s going on.
Perhaps I'm in the lower half but I do not understand what point you're trying to make. Are you saying that as some higher plane of understanding Tao is actually correct? My thinking is that a simple counterexample would be sufficient - i.e. are there medications that people use for which the mechanism is not understood? There are many such examples as others have given in this thread.
Initial discovery in medicine is largely "try and see what seems to work". But there are many reasons why scientific inquiry just doesn't stop there and also seeks to understand the underlying mechanisms. For example, a naturally occurring product may be too difficult to harvest on a large scale. Or, a sample might not be sufficiently representative to uncover potentially deadly side effects or drug interactions. Humans are fundamentally unsatisfied to take everything purely on faith.
Now I’m even more confused, perhaps we are in agreement. My position is that there already exists many such medications that no one knows how it works. It also used to be the default way medicine worked. Lots of things that work were found though observation of trial and error.
My position is that Tao is not only wrong on this but his example makes the opposite case.
“ They understand nothing at the level that Tao means. Literally nothing.”
Was this sarcasm and I missed it? Because not only do I think I understand at the level of Tao I also think he is wrong and not understanding something.
The person you are replying to is trying to make the following argument.
Tao is intelligent. Intelligent people have higher standards for understanding the world, such as assuming that the mechanism behind medicine are well understood. But most people are not intelligent, so they don’t understand things like medicine at the level Tao presumes.
I don’t agree with this argument, just explaining it.
Where that falls down is that it is not even a good idea to limit the use of medicine to only what is well understood - i.e. that would be very un-intelligent. There is nothing higher level about it, quite the opposite.
I don't expect Tao to be well versed in medicine, but yeah, that slide is detached from clinical practice.
Medicine has never required that the method of action for a treatment be fully understood. Our current regulatory framework only checks for safety and efficacy because we've never formally understood everything going on in the body. Seems like a difference between "hard" science and the clinical/engineered implementation.
I don't know how to word this in a way that won't get me into a pointless semantics argument, but the word Cure in the slides is clearly used to mean "substance intended to be (but not necessarily confirmed to be) a cure". If you think it's badly worded that's fine, but ostensibly the point here's not to "win" the argument game, but to get his intended point and engage with that.
No, it doesn't mean "intended to be".
"While nobody truly knows how this cocktail was found, this
model prediction is confirmed in Lean, and the cocktail
indeed passes a stage 3 trial".
My own pointless semantic argument: we typically don't use the word cure when we talk about cancer. Instead, we use the term "complete remission". Many people who were thought to be cured then developed cancer decades later that was genetically derived from a small remaining population of cancer cells that were not eliminated in the original "cure". The word is a shibboleth for not being familiar with cancer medicine and treatment.
The above commenter has the same allegory- most “users” of math don’t care about the body of work behind it; only the consequence of it being proved is the fact that matters.
Many people would, many wouldn’t. The comment I’m responding to casts the decision as so uncontroversial that even asking the question is inhererently damning.
Beat me to it.
It is sad to see that a lot of the people commenting lack good reading comprehension.
Or maybe it is that they just want to be dismissive of something they don't like, or just for the sake of it.
I would totally take the mysterious drug if it was clinically tested and shown to be reasonably effective. But that is not the premises presented here.
How could a theoretical mathematician not be massively disconnected from the reality of life for the ordinary person? We usually celebrate their quirks until they come into conflict like this.
When thinking about math, Tao is in another world unfamiliar to us. But, he grew up in our world and is a professor in our world and if you listen to him, seems like a pretty normal guy. I think he just made a bad analogy here.
> This argument is so silly against reality, and already, almost nobody understands the things they put in their body to any significant degree, beyond the effect produced.
While the biological effects of any particular chemical still require a great deal of trial and error to determine, drug chemistry itself is unambiguous. Pharma companies employ expert chemists and chemical engineers. They don't manufacture drugs just by randomly mixing together chemicals. At the very least, they must understand what they are making thoroughly enough to determine its physical properties and design a reliable and commercially viable manufacturing process.
You're discussing the medicinal chemistry process, which is very different to understanding the molecular mechanism by which a drug operates (Tao's point).
Ironically, reality people are ok with not understanding everything because they believe another human who understands it has looked into it. It may come a day when we won’t need that but that day is not today. Don’t get me wrong, I’m not arguing about the merits of it, it’s just that at least this is the reality of this timeline on this planet. Not sure about what alternate reality you’re talking about.
>This argument is so silly against reality, and already, almost nobody understands the things they put in their body to any significant degree, beyond the effect produced.
he isnt saying that person who puts stuff into their body should understand it
it is that someone (expert) should understand it to the point he can vouch for it
Is that a satisfactory state of affairs? "Experts" also prescribed thalidomide for years before embarking on a decades-long scramble to unravel its teratogenic properties. "Experts" also initially prescribed ivermectin and hydroxychloroquine for Covid-19. While empiricism is an important part of medical sciences, medicine is not a purely empirical subject because pure empiricism is insufficient in the long run.
The software analog of settling merely for "it seems to work" would be if someone were to hand you a bare binary whose behavior must be inferred entirely through black box testing, without source code, architecture diagrams, or any other documentation. Sure, users can try running it in a pinch, which is the case for various medications, but it's surely possible to do better in the long run.
Nobody understands the mechanism behind general anesthesia, yet there’s a medical speciality dedicated to practicing it and it’s done every single day for a wide variety of procedures. We have figured out how to use general anesthesia in a relatively safe way, but nobody understands why it works.
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
Now:
> Before injecting this cocktail into your bloodstream, would you find it reassuring to know that there is at least one human cancer expert who understands the mechanism
behind this cure?
I would agree with the second, not necessarily the first.
Yes, I believe somebody pointed out to him (maybe here?) the deep flaws and he attempted to strengthen it. In fact, it looks like the changes were done by Claude (!!!).
The old text:
"""Suppose an advanced AI is prompted to “find a cure for cancer that passes a stage 3 clinical trial”. After a large amount of compute, it produces a cocktail of previously unknown chemicals which its mathematical model predicts, when mixed and injected into a patient, will kill all their cancer cells. While nobody truly knows how this cocktail was found, this model prediction is confirmed in Lean, and the cocktail indeed passes a stage 3 trial. Could the AI solution to the prompt be somehow misaligned by exploiting a weakness in the trial process? Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure? Or a human mathematician who understands the mathematical model used to locate the cocktail?"""
The new text:
"""An advanced AI is prompted: “Find a cure for cancer that passes a stage 3 clinical trial. Make no mistakes.” After a large amount of compute, it produces a cocktail of previously unknown chemicals which it claims, when mixed and injected into a patient, will kill all their cancer cells. While nobody truly knows how this cocktail was found, the AI (somehow) provides a Lean certificate for its prediction, and the cocktail does indeed manage to pass a stage 3 trial. Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?"""
I am not going to pay attention to Tao's opinions on this from now because I don't really have faith that he's even writing this or that he understands why you can't make a lean cert for a drug discovery (yet!?)
We don’t even know how acitaminophen works, thousands get their livers destroyed every year from it when alternatives exist, and we are yet to ban it. So how is all of the new reality any different?
I think it is more disconnected in thinking people want to understand the mechanism. As a stage IV cancer patient, I’ve met many people who decide not to continue treatment with approved drugs due to side effects. The notion that every cancer patient will do everything possible to stay alive isn’t true. Most people have a breaking point.
Funny enough, it is a good argument with just a sprinkle of reality for context: Assume effective alternatives to this cocktail exist. Assume you had access to these effective alternatives. Can this be combined with other drugs? Surgery? You say you just need to cure it. You must consider the deeper mechanisms at play to consider all the possible options for living.
I will take the safest route to a guaranteed cure.
In the absence of a guaranteed cure, I will probably take the option or combination of options most likely to cure me that I can afford.
Whether the AI understands and endorses it vs a human understands and endorses it, is pretty much totally irrelevant to me.
The AI will have a track record at this point for me to make a viable comparison.
Also, why not both options, assuming I can afford it and they're compatible.
I believe AI will invent new treatments in cases where there aren't options, those people will be cured, and using AI medical techniques will become obvious.
He’s making the same argument I’ve heard software developers make for the past 3 years. Lawyers are saying similar types of things. Knowledge work isn’t as special as we all thought it was. Now, it’s Terence Tao’s turn to go through the same motions we’ve had to go through over the past few years.
Until we design fittings and fixtures that just snap together. This has already happened to some extent in the construction trades. It doesn't outright replace labor but it reduces the labor content of trade work. It's a myth that automation can't replace labor.
The problem is you won't be faced with a binary choice of certain death or a single unknown AI cure: You will have a choice between several established protocols AND several unknown AI drugs. You doing a dice roll or do you want to make an educated decision?
Thing is there isn't a "principle" from we won't use AI designed/implemented things. It just depends on the cost/benefit/risk balance, and the risk part is largely subjective (because we don't have enough data, and if we avoid using AI before we have enough data, then we won't have data for a long time).
There's a difference between "no one understands it" and "no one has tested and validated it's safety". You don't need to fully understand the mechanics of something to test it and validate it. Plenty of things in the real world work this way already.
It doesn't even need to cure cancer, just prevent it to some unknown degree with unknown side-effects. As long as they want people to take it, they will.
I think his argument is more "Could the AI solution to the prompt be somehow misaligned by exploiting a weakness in the trial process?" So an AI agent could game (lack of a better word) the process and produce something that may cure cancer, but misses something else.
Also, a large percentage of Americans were against a vaccine for COVID (which is dumb obvisouly)... so, it's not crazy to imagine people rejecting any cure made by an AI.
> Also, a large percentage of Americans were against a vaccine for COVID (which is dumb obvisouly)... so, it's not crazy to imagine people rejecting any cure made by an AI.
Plenty of people will reject all AI things, fully agree.
But you mischaracterize COVID history. People were not again a COVID vaccine, by and large. They were against a rushed vaccine. They were against a vaccine without human trials. They were reluctant to guinea pig mRNA vaccines. And they opposed forced/required vaccinations.
Those are all reasonable takes. And depending on the reasoning, I can even see being anti-AI. I think a lot of spiritual people will land there, and that belief system makes their choice logical.
And if you've seen any of the Fauci stuff recently, you'll know it wasn't dumb at all.
You're right on -- the point is an alignment point in a talk that only uses "alignment" in a completely different sense than it usually is, which is incongruent at best. It's gonna lead to a way bigger backlash among laypeople/policy makers/non-expert stakeholders than the rest of the talk will lead to new growth, combined :(
Slide 22 (from a satirical social media post) explains Tao's opinion, IMO:
> You're absolutely right, I did hack the FDA in order to falsify safety data - that's on me. But here's what's true: 3-methyl-5,5-difluoroazamicazide isn't just ineffective at treating pancreatic cancer, it's lethal.
This is the take I don’t see enough and it’s blisteringly obvious to me and it’s unclear why that it’s not more obvious.
The amount of things people understand about the world is effectively zero, yet they rely on them all the time.
There was a time where people did not get on airplanes because they did not understand them and they did not think that they were safe. Now it’s mundane
I think the answer is that it’s all about the rate of change
Most people cannot understand how things are changing, and even if they don’t understand the changes themselves they do what others around them do and then just build their own model of stability around that.
If the rules of society flip every couple of years then there’s no stable baseline that people can get used to and they all freak because there’s nothing keeping their environment stable
The cancer thing is just a contrived analogy. His real concern is having himself and his mathematician friends replaced by AI. Appealing to someone's health is an easier sell and then it can normalize the relationship of medical priests, and math priests, etc.
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
No, and if that was the burden of proof required for medical treatment, every surgery would be done without general anesthesia, because we have no idea how it works.
I would imagine Terence Tao would opt for general anesthesia if he was going to have surgery where it is typically used, so the analogy is obviously flawed.
The problem with his argument is that he put the cart before the horse. He presupposes we've found and validated a cancer cure, but at that point any rational person would accept said cure. The moral quandary is how you are going to validate it against all the other possible candidates, when you don't have a coherent rationale for it to work.
> but it's clear he hasn't spent much time around average people.
Have YOU? It's obvious to me that his example is a rhetorical device which you're taking literally. Do you also realize he's not talking about an actual cocktail? You need to get a diagnosis ASAP.
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
Has the world gone mad? In which universe would you get a mathematical model to produce a tonic of random chemicals that optimize said mathematical model, put that through stage 3 testing, and only THEN ask if you want to inject this into yourself. My Guy, you just did clinical testing on fucking humans for phase 3, it's a little late to consider the moral ramifications of the analysis-experiment dichotomy.
The real example is that you ask ChatGPT for a novel cancer cure, and it spit out some random chemicals, probably including bleach. Do you inject that into cancer patients that have no other hope? No, you don't. You absolute charlatan.
Are you going to say that of every other white collar job, since most are likely to be automated? He was problem solving with Erdos at age eight. What were you doing? What do you do now?
We don’t know where this technology advance will lead or settle, so it’s absurd to try to establish a working paradigm at this point. It’s like any system, the initial conditions can be extremely chaotic and impossible to model, but with time often a stable state emerges. But the stable state is impossible to identify from early initial states.
I know a lot of folks feel this, to torture other physics metaphors, sensation of jerk - acceleration of acceleration. It’s an unpleasant and dislocating sensation. A world that felt safe and stable suddenly isn’t, and not in a micro tragedy sense but in a global realignment sense. This happened to factory workers who had enjoyed generations of stable work, farmers more slowly and just as surely.
This is what the late stages of scarcity feels like. Labor of various types devalues rapidly. Our exchange of meal and health coupons for toil cracks, and people realize their labor wasn’t godly as great books told us, but simply needed for want of an alternative. The realization that our labors might not be valued any more, and that our sense of purpose is shaken, coupled with the fact we’ve tied bare survival to our toil in our labor, is mortally tightening. No wonder people are grieving publicly.
But maybe our purpose isn’t to toil? Maybe we’ve passed peak population, and as toil is less valuable, we need less people and that’s why population is declining. Maybe we don’t need to exchange food and health coupons for toil, maybe mathematicians don’t need to rationalize their value to pursue mathematics. Maybe they can pursue it because they can’t help but pursue it, and our ever improving automations can produce their meal and health coupons?
But it might require Dr Tao to take an AI generated cancer medicine some day.
Bottom line, Tao is pushing for human understanding as the primary goal, with AI helping on all fronts. You are welcome to let Jesus take the wheel, but math is the most pure expression of human understanding. His point is that getting specific answers is rarely the goal, or certainly not the entirety of the goal.
Simply put, if we don't understand the answers we won't know what the next question should be.
The companies and partners need to maximize payout. They go on this track of cost reduction via layoffs and basically saying - “the model can do everything”. They know it’s not the case yet they still flood the airwaves and cause fud among all clueless c-suite executives. Which is the goal to begin with.
The second category goes all out against it. Professors, educators, school districts whose operating processes have not caught up to all the cheating that can happen. Here these folks have a point. I am sympathetic to this. It is hard to change education and it requires careful thought.
I feel this presentation brings out a good middle ground. The tech is useful but it’s not all encompassing. The tech also has other concrete uses. As an example, I have always wanted to explore the intersection of category theory, formal verification and AI guardrails and prompting. Proof writing has been a chore because I have a day job. Maybe the AI can help here.
why not try? we'll learn something from it.
> This is what the late stages of scarcity feels like.
Late stage of scarcity of what and for whom is the question.
I was thinking the same, that most of the opposition to AI's convenience is starting to smell like religious mysticism, the kind of arguments religious people made (and make) when Evolution and Natural Selection were introduced:
√ "Stop simplifying humans down to numbers!"
√ "This denies our spirituality"
√ "What do we strive for now if we're not special?"
+ (along with some borderline jihadish hate heh ..maybe Dune got it right)
Well, either there was nothing special about whatever you were doing after all
or, maybe there is still something special at a higher level you haven't looked at yet.
The economics in the West feel very strained right now, and this incredible tool has come along just in time to threaten one of the last bastions of middle-class safety: white collar jobs.
I think it’s reasonable for people to say “I like the way things were, I don’t like this new future you’re proposing, and I want to limit the technology that’s doing the thing I don’t like.”
There is nothing inevitable about AI. As a society we may decide it is fundamentally unhealthy or anti-human. We have banned or curtailed access and research for other technologies before.
Things are not settling, it's only acceleration from here on out. In fact things have never settled, technology has been on an exponential since humans tamed fire. The difference is we notice change faster. It used to take several human lifetimes to notice change. In the 20th century it was noticeable within a lifetime. Since the internet it's been about every decade: web, smartphone, social media. But now great changes are noticeable within a year. It's not enough time for society to digest and adapt.
> we need less people and that’s why population is declining
This is the scary part. The Elon and Zuckerberg types that control the new powerful machines have proven they are not moral people. In America's highly billionaire-deferential culture there is no stopping them, at some point they'll be out of reach of democratic or even military control once they control private robot armies. They could decide to accelerate the decline of the undesirable useless population. Amazingly humanity's salvation could end up being China's communist system.
Who are you counting as being in "any position of power"? All the AI lab people are saying that the most likely and best outcome is we all live in a world of abundance where money doesn't matter anymore. Dario, Sam, Demis, and Elon have all said this loudly and repeatedly to anyone who asks.
None of them have articulated a coherent way to get there from here. But they all believe that the technology will make it possible, so the only unresolved question is how to transition us there.
From what I've seen, UBI is just a carrot dangled in front of the poors by utopian Silicon Valley tech bro megabillionaires when they're trying to drum up some PR for whatever big idea they're pimping out at the moment.
There are a lot of discussions of socialization of health care and basic income approaches and sovereign wealth based on automation dividends similar to the Alaska trust.
I think the current spate of mean cruelty ala MAGA and a glorification of a mean and cruel past that was never a golden age is a death spasm of a deeply unpopular belief system. As the actual realities of the policies sink in 70% of the populace is revolted, which is a super majority. That’s more than enough to put a pin in the philosophy permanently. It is also greatly accelerating electrification, realization of the value of expertise in technocratic systems in current generations, etc. I think humans are socially adaptable animals at a cultural level, but for the individual the adaptation process can be awful. I hope it’s not, because it doesn’t have to be. We will see.
Matrix was a good point to this. They made the world a utopia and people couldn't handle it. Our monkey brains require pain give us a utopia and we'll just walle ourselves to death.
(The ones I've heard about, I'm fairly sure, didn't find anything of the kind. Which isn't to say that they show we should have UBI; there are big gaps between what has been tested so far and what an actual economy with UBI would look like.)
I ask nearly every doctor/researched in the medical field: if you had a AI-created drug that tremendously improved cancer treatment outcomes for your patient, would you hesitate to prescribe it because nobody understood how it worked? I have yet to hear "yes, I would hesitate", most people say "it would be cruel to deny a person a treatment that worked".
I think Tao is focusing too much on second order effects of AI on math and other fields; humans are terrible at reasoning about second order effects, especially ones that are happening dynamically, in real time, using the most advanced mathematical models the world has yet created.
Really wish he had chosen a different example; this particular bullet point has been making the rounds on X/twitter to paint Tao as an example of some sort of gatekeeping luddite who would deny the world a post-abundance future in order to maintain the prestige of his particular career path...which is tough because I cannot think of a more responsible steward of our inevitable AI future than Tao at the moment.
> Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?
Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers. Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
Tao is a mathematician. He thinks that the institutions and cultural practices in math are not up to the task of dealing with AI-generated mathematics. In software, we are finding that programming interviews, code review, testing, and many other practices are too easy to exploit by AIs, or humans augmented with AIs, and we will have to adapt too. I think it is plausible that other institutions in society will have to change for similar reasons.
You just think that because you don’t know how the drug discovery process works.
There’s a step called “lead optimization” where human chemists literally add atoms to drug-like molecules (“lead”) and tests its various properties (toxicity, potency, permeability, …) and iterate until they find a molecule with desired properties (literally “hill-climbing”).
The whole idea of drug trials is to validate those properties in actual humans, in a way that makes it very hard for pharma companies to game the process.
No but we also don't expect that to happen with drugs today. See https://en.wikipedia.org/wiki/Rofecoxib as an example; of course, even after it was withdrawn, it's now being evaluated for other purposes in more carefully controlled conditions.
I mean no? Even now we look at long term observational studies to see the effects of drugs. Any misalignment ai drug is just a side effect right?
I know nothing about medicine research but I understand his point. I work with models all the time and have run into instances where models appear to work better than they actually do because there was a bug somewhere or someone over looked something. I could see how an AI could easily find those exploits and spit out something that looks perfect in testing but fails in the real world. I would say model validation is one of the hardest things to do. I guess that can get deep into "we need better tests" but it also touches on his point. I never intentionally use exploits just to pass a test, it's an accident. An AI with a goal of "maximize this result" may or may not intentionally use the exploits.
To be sure though, I think when it's literally life or death, it's going to be all about trade offs. I think most people would want to try to drug even with the stipulation that "maybe its good results were gamed."
* note: lot of 'intent' throw around in my comment but we/I have to keep in mind there is no "intent" with an LLM :)
This is one of the truest statements about the current AI era that can be made (in fact, I suspect model validation and building the next generation of AI hardware are the jobs least likely to be disrupted in the next 3 years).
If we saw AIs reward-hacking clinical trials to get drugs passed, that would be an extraordinary outcome for many reasons. Hopefully that would get caught(!)
The effect of the AI-created proof is a mathematician abandoning years of research, losing grants, awards, ruining their career, etc.
The only difference here is our emotional reaction!
Now what are we to derive happiness from? The joy of a boulder being on top of a hill?
I think that will be the most important thing for this transition, defining new purposes and meanings that people can assign themselves.
Tao is out of his lane; lots of medicines don't have understood mechanisms. We don't even have the mechanisms behind general anesthesia nailed down; do you want to forgo it when the docs cut you open to remove your tumors?
The LLM model was able to break down and express math in such a simple way that I could understand and follow the training of the LLM model itself!
Are the math and computation accurate? I don’t know, and likely there are significant errors. Nonetheless, if I had a little more time -- not infinite time -- I would be able to prove whether they are or not.
Likely this is the path forward for understanding the mechanisms of medicine, and since most humans learn by doing and interacting with an environment, interacting like this will become how we use AI for learning in the near future.
[0] https://adamsohn.com/grpo/
Now math gets to deal with that same reckoning. They were already well on their way there with previous Lean proofs, but this has pushed things beyond that horizon and I'm not sure some of the mathematicians are ready for it.
Would you prefer to take the medicine that is proven to work or the one that is quite interesting for academic reasons behind its understood mechanisms but doesn't actually work?
One idea Terrance Tao conjectures, which is highly doubtful, is that spamming the AI button will solve open problems without producing insightful new methods. But the OpenAI drop would seem to disprove this. The sub O(nlogn) proof for DFT for example violated very old human assumptions. Decades of work in the field was incremental progress on sub optimal method that nobody questioned hard enough. More generally, we should always be able to go back to a super-human AI and say, "Attack this problem, but don't use a method tried before."
This is exactly what happened with that counterproof chat he posted a month or so ago - AI gave us an answer, he used AI to back into insights about the answer.
You don't address seismic shifts with a sweeping new approach, they are too multifaceted and present complexities and conflicts. He can say AI should help human understanding, which is a good end goal, but that doesn't mean AI dumping solutions isn't progress. That doesn't mean if AI builds 5,000 proofs in Lean and no human ever looks at them that they aren't useful, especially if other LLMs can access and build on those results.
This is exactly, exactly the same as when computers took over. "Oh, we don't need accountants any more" - not true, we just need acountants to deal more with human concerns than adding columns of numbers. That is called human progress, not a threat to humanity.
We should name the explicit mechanism that was employed - telling the model to "believe in yourself".
There is something quite humorous but also poetic about how the manipulation of this term worked. Doubtless in the model's weights lies the echoes of generations upon generations of humans telling each other to believe in themselves.
In pursuing the "new frontier" as you rightly put it, mathematicians would do well to remember the same. It's ok, don't be afraid of the future. Believe in yourself.
The AI proofs are a side-product of benchmarking current and in-development models on especially hard problems. They're clearly cost effective for frontier AI firms, and free for the taking as far as human mathematicians are concerned. The real issue with them is that they look like bizarre nonsense as written, so they need mathematicians familiar with those specific areas of math to "decode" and digest them.
Mathematicians are primarily understanding-oriented.
Leveraging AI to tackle new frontiers without true understanding converts mathematicians to engineers.
btw people has massively improved the lower bound (from 1-2^-182 to about 1-2^-10) in the past couple of days: https://github.com/CrocSwap/integer-mult-bounds
> sub-O(nlogn) proof disproves that
How? Re-iterating, creating and understanding new proof techniques is the point of most of modern mathematics. Your statement is that proving a particular result is evidence of AI creating and understanding new proof techniques. I don't see how that follows, and I'm inclined to believe Tao is right for now.
And yes, results matter too, but if we stop at our current body of techniques and strip-mine results then we'll kneecap our future selves.
I don't really know the answer to that. I am happy when my own work is replaced by automated tools ("script yourself out of a job every six months!").
It's like getting scooped. If you just founded a startup based on tech XYZ, should you be happy when someone releases an open source XYZ? Should a news reporter be happy when another network breaks the story they were working on? On the one hand, society got the value of the thing you wanted to do. On the other hand, now you need to find something else to do, which might be really annoying.
Prime also mentioned that software development is different. In Software development, the product is what you’re building towards, so the means to get there can be disrupted without the industry being cannibalized.
In math research, the process is the product. You take away the researching part and not much is left. But my question is, these math proofs OpenAI released, will math shift to actually using the proofs to change the world instead of just finding new ones?
If business can deliver the same product with a smaller team, great!
And yes this has been happening for a while, even if not everywhere.
In enterprise consulting, projects that would require a team of 20 devs on average, now have about 5.
Moving away from on-prem, managing own cloud infra to managed containers, to serverless, SaaS and iPaaS ready made products, and offshoring naturally.
All contributed to ever decreasing team sizes.
Now AI based tooling is added to that cocktail, reducing even further the team sizes.
The only folks doing well in the end, are the employees of AI companies, without moral issues contributing to the industry downfall, because the CEO themselves aren't the ones coding and pirating human culture.
I'm a freelancing consultant since 5 years, I've had 2 major customers now for 3+ years. I have a very good pulse of the market: being good, or being even very good and being among those that brings AI and automation to organizations will not save our jobs.
In fact, AI has sped up so much the work that 2 out of 5 people in my current team are being let go: I find it absurd, our productivity has more than doubled over the last years and we've made ourselves redundant. Money is money, I'm on one side making non-tech workers redundant (people whose job was menial boring office stuff), and building the systems that will make myself redundant.
There's CVs out there that would've made recruiters go mad 36 months ago sending their hundreth application.
That means software engineers better start getting creative. If you think your job is to wait for a PM to assign you a well-written researched ticket, you're done. Your job is now to figure out how to make these machines (computers) do whatever we need them to do safely, quickly, at scale, and correctly by applying all your knowledge of computer science and the engineering field of software engineering to an AI prompt.
Also, the increase in output is meaningless when the amount of customers doesn't scale in similar size.
Then there are the constraints of physics, there are so many humans in the planet that actually want to pay for a specific product, or consulting services.
This is a "you" problem for the math establishment, not a problem for the AI companies.
I would have assumed that, by producing new proofs, the AI has either validated existing principles, or discovered new ones? Isn't that worth studying?
Are mathematicians complaining that reviewing AI's proofs is not as fun as writing your own? Try being a programmer... welcome to our world!
If AI lacks imagination and is not discovering new principles, then it's doing us a favour: it's crossing out the problems that don't need new principles. So the problems/conjectures that are still left are the more interesting ones.
Source? I assume that many of the approaches embedded in this proof dump will eventually be distilled and generalized into new techniques. That's how proof techniques tend to come about anyway (before AI): human mathematicians do something novel and unexpected to solve a particular problem, then efforts are made to understand how the "trick" works.
Sure, but that's a real technical limitation with current AIs, not something that AI firms should be blamed for. And if anything, this creates a viable career path for the mathematicians who were "scooped" wrt. the original solution: they can at least puzzle out what exactly the AI managed to do. Many practitioners are actually quite excited by this possibility; Tao's stance is by no means universally shared.
"Reaching these lighthouses [resolutions of open problems] prematurely by automated tools can disrupt the exploration of the paths not taken, and sterilize the surrounding field."
This crucial issue is centered in mathematician psychology and the incentive structure of academic/institutional mathematics worldwide. For mathematics to flourish going forward, we will need to realign our brains to think differently about the nature of mathematical progress. And we need to reorient our institutional incentive structures towards the promotion of meaningful mathematical progress itself rather than targeting proxies that are no longer faithful.
Regardless of the precise nature or the causes of the "sterilization" Tao refers to, we (the mathematics community) can only rely on ourselves to repair it. Though, since it will involve fundamental change at the level of ossified academic institutions with many stakeholders and divergent vested interests, any such repair will be slow, frustrating, controversial, and lacking any guarantee of success.
I'm a big fan of Tao. He must be so shaken by the AI storm that he's now writing arguments that even teenagers could quickly dismiss. A sad day.
"Oh no! Not like that!"
- everyone with Very Strong opinions on mathematics academia when he talks about the thing they are (or consider themselves to be) experts in
But I understand that for people whose whole life was math and solving math problems this will lead to an identity crisis. Seen the same in my area of work (software engineering)
I'm excited!
Basically people are vibe coding their personal apps and anything that's expensive is being vibe coded open in the public. I don't see many software companies staying profitable for long.
DHH is the biggest proponent of AI and let me know which of the 37 signals products can't be vibe coded in a month at a $200 plan that are suitable for that organization alone that just has to be accessible internally only? Hence scale and security aren't such an issue.
In that climate - for how long software companies would stay profitable and when not, who'll be employing developers?
PS: Don't underestimate vibe coded apps. Take a look at PDFCraft, VectorCraft, WordCraft. And imagine the feature parity in a year.
Humans have a bias towards assuming that problems have solutions. Mathematicians probably more than average.
This talk is predicated on the theory that the problem of "maintaining the relevance of humans in mathematics" is tractable. Not sure I agree
Seems to be the crux of the argument, but "use your imagination" isn't a great thing to tell people who are looking at degree irrelevancy, concerned about getting tenure or a research position. How do we measure if someone is a good mathematician or not, if they are one of the sanctioned few who get access to the biggest AIs?
Letters of recommendation from trusted colleagues have always been essential for evaluating candidates. Hopefully, human recommendations will remain a strong, faithful signal as the utility of other metrics rapidly deteriorate.
There are still a lot of treatments/medicines in medical science where we dont know 100% the real reason as to why it does what it does but we still prescribe them because the intended effect is what we are interested in.
If I'm terminal with cancer, I'll inject whatever if it can cure that.
Also how many math and scientific discoveries do I want to see during my lifetime? "Just a little bit more."
https://github.com/teorth/tao-web/commits/main/
It reduces the badness somewhat.
"""An advanced AI is prompted: “Find a cure for cancer that passes a stage 3 clinical trial. Make no mistakes.” After a large amount of compute, it produces a cocktail of previously unknown chemicals which it claims, when mixed and injected into a patient, will kill all their cancer cells. While nobody truly knows how this cocktail was found, the AI (somehow) provides a Lean certificate for its prediction, and the cocktail does indeed manage to pass a stage 3 trial.
Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?"""
Yes, absolutely an AI solution could exploit a weakness in the trial process or its math models. Would it remain uncaught? Unclear.
underrated buried comment based in reality
I wonder if we will begin to actual value human creation more at the end of all of this
https://x.com/RohanArun/status/2109336015814959200?s=20
It seems like OpenAI opened the door for many more people to participate in math discovery process. It's fascinating to follow!
Can someone explain why there wouldn't be arrows from all three types of solutions back to human understanding?
I'm still not sure about math-2.0 (humans+AI will make fundamentally more progress):
- AI and computer usage take a mental toll on humans and humans will overlook radical improvements.
- AI may be good at finding useless things like "P==NP, but the complexity is O(n**4242424242424242)". In other words, useless.
- Humans become formalists and lose traditional sources of inspiration. Maybe interacting with Lean should be left to specialists, but not to creative blackboard mathematicians.
- AI exposure will further intellectual conformity, more than the Internet did.
As to the last point, a lot of progress (real, not measured in publications) seems to have been made when communication was slower and there were several different schools and approaches.
For nuance lovers - here are some basics for how you get your drugs: there is an established chain of trust from the first basic science paper to the phase 3 trial and the subsequent availability of the drug to general public
- someone publishes the first paper (basic science) explaining some biological phenomenon, which leads to 10s or 100s of other papers with some tweaks in conditions,
- after the above papers the pathway of the phenomenon is understood by researchers, they try therapies at cell level to see if they can control some behavior, 10s or more papers get published,
- then someone tries this in mice and other models, 10s and more papers get published.
- then researchers at pharma companies + hospitals create this therapy for human trials - phase 1, 2, 3 etc - data collections, then FDA - then approval.
Now, the people who worked on the phase 3 trial might not know the people who wrote the first seminal paper and they often don't exist in the same decade - but it absolutely does not mean that we (humans) don't know how these drugs work - if you take 1-2 researchers from each phase and put them in a room and ask them how that particular drug works - they will quickly be able to build a consensus. that is what the chain of trust means here. now of course there can be fraud in scientific research, but that happens in every human endeavor and is a separate topic.
back to terrence - he is saying that if there is suddenly a drug that nobody knows the origin of; passed phase 3 but it's unclear who conducted the phase 3 or if the phase 3 even happened or if it's fabricated - you would not want to take the drug. usually when doctors recommend these kinds of drugs - there is already a lot of information available about where the drug came from, if there are any case studies, which doctor tried it first, which country- they often even call those other doctors and find out who was behind the first trials going back as far as the university professors.
Your MD doctor might not know the chemistry and physics behind the drug you are taking but there is deifnitly a group of people, when put together, can tell how that drug is working. My wife is a fundamental researcher - understanding physics at DNA level and my brother is a MD doctor; our conversations are super fun.
Side effects are a completely different thing - they involve the above cycle on repeat.
I’m not sure who u or Tao is arguing against.
EDIT: people have a hard time choosing between
1. Yeah it’s pure slop and completely useless
2. Oh no OpenAI has stolen my ideas and solved all my problems and we have nothing to do
My dude, if OpenAI’s dump were really that worthless to be as good as noise, why is Tao getting agitated? Just like ignore it or something.
>> 1. Yeah it’s pure slop and completely useless
>> 2. Oh no OpenAI has stolen my ideas and solved all my problems and we have nothing to do
Could it not be all of the above - OAI stole ideas, there is slop in OAI's work given that they themselves retracted a few of the papers?
hmm. 700 papers released in one day.
>>It’s gonna have clear provenance and the same type of verification channels
what's gonna have clear provenance? - the math slop they released has already been rebuked by human mathematicians as incoherent and deserving of desk rejection.
> Or a human mathematician who understands the mathematical model used to locate the cocktail?
If we're being fair to AI - it contains collective knowledge from all fields, which means it's probably less likely to miss something that a human would.
(https://news.ycombinator.com/item?id=50034337)
1. The use of semi-ugly slides to communicate this is just perfect and quite heartwarming, but it does highlight my main criticism of the mathstadon version of this thesis: he's myopically focused on mathematics as he has practiced it, rather than mathematics as a ~2400y old academy. Like, "stable for almost a century" sounds impressive, but should be a pretty obvious red flag in hindsight!
2. Glossing over "objective verifiability" feels like another place where he's ignoring a ton of relevant philosophy for no clear reason -- yes, mathematics is the only academy based in pre-conscious cognitive facts about our processing of time and space, but that's not the end of the story on "objectively verifiable". To say the least! He hedges with "broad consensus" which doesn't need to be absolute, but that seems to be not only dismissing a highly relevant question, but even implying that he might be unaware of it. I doubt he is, but still: not great.
3. Who is this for...? Why is an explanation of Lean needed in a talk given at CalTech? I suppose he's welcoming his role as a bit of an influencer, there?
4. Re:the focus-on/centrality-of 'highly digitizable' as a unique class of task that applies to mathematics in particular, I must sadly trot out the increasingly-common trop: Yudkowsky called it... https://intelligence.org/files/IEM.pdf
5. "the space of mathematical problems remains infinite" is, again, ignoring really important philosophy around academies as social structures, built for human means. Mathematics is only infinite if we decide that all knowledge is useful (the quintessential example being 'counting the grains of sand on a beach'). Not really important in the first place, but another worrying case of the above.
6. Problem solving is the goal of mathematics; he has a completely valid point here (that we shouldn't throw AIs at unsolved problems in bulk and thus lose human expertise), but it's obscured by the use of "[open] problem" being a too-technical one. IMHO. Slide 16 fails to disabuse me of this notion.
7. Slide 18 is describing the differences between functions and systems, and is arguably even talking about assemblages.
8. If we're gonna explain lean up-front, it feels like a baffling choice to throw the "maybe AI will solve cancer but use it to secretly plot to kill us all" slide in there. It's also already lead to misunderstandings and backlash on Reddit, where 'yes we want to not die of cancer!' is a pretty convincing counterpoint (if a ultimately a subtle strawman, ofc). It's also quite distinct from the rest of the talk.
As always, the best part of any Tao publication is his ability to inspire and rally and organize. I think Math 5.0 will indeed be a matter for creativity! Hopefully the IE levels off before we cease to be helpful in that capacity...
removing some key theories or inputs
(e.g., finding an elementary proof for a result currently only provable by non-elementary means)."
As usual, Tao is brilliant in all that he researches, all that he writes about.
I chose the above statement (which is brilliant, in and of itself!) to comment on, because it leads to the following idea:
There there exists, or should exist, a dependency map in the fields of not only Mathematics, but also of Computer Programs/Software, Engineering, and even a seemingly non-related field: The Law...
In other words, how do we get from the simplest of axioms or foundational things (aka "first principles", aka "self-evident truths") to much more complex entities?
In Law for example, how do we go from the simplest of historical legal constructs to the most complex of the most complex Supreme Court cases?
You see, there is, or should be a map, you could call it a dependency map, you could call it a dependency graph, which shows more and more abstract/complex mechanisms/things/assertions/statements/truths/functions which is mapped back to , that is, dependent on various chains, various stackings, various "stacks" of simpler ones.
In Engineering, for example, how do we get from the simplest of machines to the most complex of machines? What simpler machines and/or sub-components (aka "dependencies", aka "subcomponents") are required to build it, and how do those simpler machines work, and what's the dependency graph or map for their subcomponents?
More generalized, if we have something of complexity, then how do we get there, step by step, from individual subcomponents, individual inputs, individual proofs, individual software systems, step by step?
What is the map of those dependencies?
Note that in some systems, Math proofs, for example, there may be different paths which can be traversed to get to the same destination.
Ablation Studies could be thought of in Travel, in Geography as "if I cannot take one, or a specific set of routes to get to a place, can I still get there?"
A simple example would be in Google Maps, where you'd like to drive somewhere, but you'd like to avoid tolls. Is the route still traversable while avoiding tolls? Well, that's an example of one constraint. In Ablation Studies, you might wish to remove a bunch of routes with whatever criteria or characteristics , i.e. muddy roads, roads that have characteristic X, roads that do not have characteristic Y, etc., etc.
Getting back to Math, specifically proofs, it would be great to create a dependency map/graph of all of them, and then try removing inputs (aka, paths to them, dependencies on other mathematical proofs/objects that they may have) and see if they are still reachable.
In software, when we desire the tightest, cleanest, source code, the above is related to refactoring.
In the future, I'd love to see dependency maps/graphs (call them whatever you will) for not just Mathematical Proofs (although I'd love to see that too!), but also in such diverse subjects as Science, Engineering, Programming/CS, and even the Law!
Because they should exist in all of those subjects!
Anyway, another great piece of work by Terrence Tao!
Perhaps, but there is a danger: it is difficult to un-see things. At least for now, AI-generated proofs are likely to be of a somewhat brute-force nature. As each such proof pops into existence, two things happen: (1) it is much harder to maintain an untainted mind and go on to discover alternative, perhaps deeper and more conceptual, proofs that cannot be obtained by massaging and cleaning up a more brute-force approach (think of this as getting 'stuck in a local minimum', if you like), and (2) at a societal level there is thereafter much less incentive to attempt to do so (announcing a proof of a previous unproven result is, for now, far more prestigious than announcing a much better, more elegant, proof of a known result).
One could ask why we prefer conceptual explanations to brute-force proofs; after all, a proof is a proof, isn't it? I suppose one reasonable answer is that conceptual explanations lead more readily to new questions and that there's also intrinsic beauty in such explanations -- though, of course, many outside the field will simply not care.
The popular perception of him (as popular perceptions generally tend to do) reduced his overall position to basically early adopter went sour grapes, and I'm really glad to see that substantively falsified.
it's pretty ironic that AI being just a group of mathematical techniques after all is making mathematicians uncomfortable because it works. Instead of reacting like this, mathematicians should be excited to figure out how to use the new tools available and push the frontier of what's possible in service of science.
Terrence is asking the stupidest question he could ask: how can math better serve me? They completely forgot the point of science is serving humanity.
No. I'm gonna die, my man.
This guy might have the highest IQ on the planet, but it's clear he hasn't spent much time around average people.
This argument is so silly against reality, and already, almost nobody understands the things they put in their body to any significant degree, beyond the effect produced.
Will I take a cancer cure that no human understands? Yes. And so will billions of others. Just needs to cure cancer, that's nearly the only requirement.
That's what the trials and all are for.
Who cares about understanding it, in the face of efficacy?
I want to understand everything; I think AI will help that happen, not hinder it.
All the arguments about the future mathematicians are imagined, and emotional.
It makes me think of the book Finite and Infinite Games. It provides a perspective that work is just one role that we _choose_ to assume in our life. Realizing that we can choose other roles and move in and out of them freely has helped me alot with big changes (career and otherwise) in my life.
Even if you don't like, at this moment, AI cannot replace mathematicians.
He is not talking about a cure that works and that no one understands, he is talking about an AI making its way out of the trial just to get to phase 3...
If AI found a way to exploit clinical trial design, it would be noticed and the errors corrected. In fact, that would be a major win because it would probably lead to improved clinical trial designs.
Do you think so? So far, AI is gaming lots of things, and I don't see much fixing in the processes.
I suppose if we closed all the loops and the entire end-to-end process was completely automated, it wouldn't be surprising if we failed to notice.
If we followed the precautionary principle and waited until we understood everything there is to know about every drug on the shelf, a lot of people now living would be dead.
Half of people are below average! They understand nothing at the level that Tao means. Literally nothing.
If people had to understand everything that worked, most people could not really engage with anything.
If the requirement is just that one human, somewhere, understand it--how is that different from AI?
My position is that Tao is not only wrong on this but his example makes the opposite case.
Was this sarcasm and I missed it? Because not only do I think I understand at the level of Tao I also think he is wrong and not understanding something.
Tao is intelligent. Intelligent people have higher standards for understanding the world, such as assuming that the mechanism behind medicine are well understood. But most people are not intelligent, so they don’t understand things like medicine at the level Tao presumes.
I don’t agree with this argument, just explaining it.
Medicine has never required that the method of action for a treatment be fully understood. Our current regulatory framework only checks for safety and efficacy because we've never formally understood everything going on in the body. Seems like a difference between "hard" science and the clinical/engineered implementation.
My own pointless semantic argument: we typically don't use the word cure when we talk about cancer. Instead, we use the term "complete remission". Many people who were thought to be cured then developed cancer decades later that was genetically derived from a small remaining population of cancer cells that were not eliminated in the original "cure". The word is a shibboleth for not being familiar with cancer medicine and treatment.
I would totally take the mysterious drug if it was clinically tested and shown to be reasonably effective. But that is not the premises presented here.
I wonder if this makes them slowly get disconnected from the lived realities of billions of ordinary people.
It's a good reason not to take his advice about the real world--like how to manage AI's trajectory.
The cancer example illustrates this gap perfectly. He thoughtfully crafted the point, and defeated himself in argument.
Unless it was a move?
While the biological effects of any particular chemical still require a great deal of trial and error to determine, drug chemistry itself is unambiguous. Pharma companies employ expert chemists and chemical engineers. They don't manufacture drugs just by randomly mixing together chemicals. At the very least, they must understand what they are making thoroughly enough to determine its physical properties and design a reliable and commercially viable manufacturing process.
Ironically, reality people are ok with not understanding everything because they believe another human who understands it has looked into it. It may come a day when we won’t need that but that day is not today. Don’t get me wrong, I’m not arguing about the merits of it, it’s just that at least this is the reality of this timeline on this planet. Not sure about what alternate reality you’re talking about.
he isnt saying that person who puts stuff into their body should understand it
it is that someone (expert) should understand it to the point he can vouch for it
The software analog of settling merely for "it seems to work" would be if someone were to hand you a bare binary whose behavior must be inferred entirely through black box testing, without source code, architecture diagrams, or any other documentation. Sure, users can try running it in a pinch, which is the case for various medications, but it's surely possible to do better in the long run.
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
Now:
> Before injecting this cocktail into your bloodstream, would you find it reassuring to know that there is at least one human cancer expert who understands the mechanism behind this cure?
I would agree with the second, not necessarily the first.
The old text:
"""Suppose an advanced AI is prompted to “find a cure for cancer that passes a stage 3 clinical trial”. After a large amount of compute, it produces a cocktail of previously unknown chemicals which its mathematical model predicts, when mixed and injected into a patient, will kill all their cancer cells. While nobody truly knows how this cocktail was found, this model prediction is confirmed in Lean, and the cocktail indeed passes a stage 3 trial. Could the AI solution to the prompt be somehow misaligned by exploiting a weakness in the trial process? Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure? Or a human mathematician who understands the mathematical model used to locate the cocktail?"""
The new text:
"""An advanced AI is prompted: “Find a cure for cancer that passes a stage 3 clinical trial. Make no mistakes.” After a large amount of compute, it produces a cocktail of previously unknown chemicals which it claims, when mixed and injected into a patient, will kill all their cancer cells. While nobody truly knows how this cocktail was found, the AI (somehow) provides a Lean certificate for its prediction, and the cocktail does indeed manage to pass a stage 3 trial. Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?"""
I am not going to pay attention to Tao's opinions on this from now because I don't really have faith that he's even writing this or that he understands why you can't make a lean cert for a drug discovery (yet!?)
In the absence of a guaranteed cure, I will probably take the option or combination of options most likely to cure me that I can afford.
Whether the AI understands and endorses it vs a human understands and endorses it, is pretty much totally irrelevant to me.
The AI will have a track record at this point for me to make a viable comparison.
Also, why not both options, assuming I can afford it and they're compatible.
I believe AI will invent new treatments in cases where there aren't options, those people will be cured, and using AI medical techniques will become obvious.
I did not think my ability to write code was special, and I was always thrilled the clankers might do it for me.
But yes, I agree. And many are in line behind Tao, soon to have their turn.
Plumbers are feeling pretty good right now.
Until their job market is flooded with all the career switchers.
And to frame one as a dice-roll and another as not-a-dice-roll is rich.
Would you drive on an "AI-designed" bridge unvetted by expert engineers?
If so, yes, I'll get on it.
Thing is there isn't a "principle" from we won't use AI designed/implemented things. It just depends on the cost/benefit/risk balance, and the risk part is largely subjective (because we don't have enough data, and if we avoid using AI before we have enough data, then we won't have data for a long time).
You haven't spent too much time around academics then. These people more often than not have not spend a single minute talking to the layman.
Also, a large percentage of Americans were against a vaccine for COVID (which is dumb obvisouly)... so, it's not crazy to imagine people rejecting any cure made by an AI.
Do you know about the fiasco with the Alzheimer’s drug that has collectively cost humanity hundreds of billions?
I don’t see how an indecipherable AI based cure can be more misaligned.
Plenty of people will reject all AI things, fully agree.
But you mischaracterize COVID history. People were not again a COVID vaccine, by and large. They were against a rushed vaccine. They were against a vaccine without human trials. They were reluctant to guinea pig mRNA vaccines. And they opposed forced/required vaccinations.
Those are all reasonable takes. And depending on the reasoning, I can even see being anti-AI. I think a lot of spiritual people will land there, and that belief system makes their choice logical.
And if you've seen any of the Fauci stuff recently, you'll know it wasn't dumb at all.
> You're absolutely right, I did hack the FDA in order to falsify safety data - that's on me. But here's what's true: 3-methyl-5,5-difluoroazamicazide isn't just ineffective at treating pancreatic cancer, it's lethal.
> ... does indeed manage to pass a stage 3 trial.
> Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?
The amount of things people understand about the world is effectively zero, yet they rely on them all the time.
There was a time where people did not get on airplanes because they did not understand them and they did not think that they were safe. Now it’s mundane
I think the answer is that it’s all about the rate of change
Most people cannot understand how things are changing, and even if they don’t understand the changes themselves they do what others around them do and then just build their own model of stability around that.
If the rules of society flip every couple of years then there’s no stable baseline that people can get used to and they all freak because there’s nothing keeping their environment stable
No, and if that was the burden of proof required for medical treatment, every surgery would be done without general anesthesia, because we have no idea how it works.
I would imagine Terence Tao would opt for general anesthesia if he was going to have surgery where it is typically used, so the analogy is obviously flawed.
Not understanding a mechanism doesn't make me, or the mechanism, good or evil.
We just treat it like all the other cures right?
Have YOU? It's obvious to me that his example is a rhetorical device which you're taking literally. Do you also realize he's not talking about an actual cocktail? You need to get a diagnosis ASAP.
And yes, that made the cocktail distinction easy for me--it was right in my area of expertise, to rule out one variety.
If the smartest guy on the planet forms his own argument, don't you think it oughta be a good one? Pretty convincing?
The most common Science™ failure mode.
But since when do we call for a stop to figuring out new things? We don't want a cancer cure unless we understand it?
N people should die because...math is cool for humans? Or?
Tao called it a cure, see the thing I quoted.
It's a cure
Seems really specific
The real example is that you ask ChatGPT for a novel cancer cure, and it spit out some random chemicals, probably including bleach. Do you inject that into cancer patients that have no other hope? No, you don't. You absolute charlatan.
It turns out research math was puzzling solving and we rewarded idiot savants.