The founder opened by admitting he has lost an argument with the group about what to call these opening remarks. He had asked for an alternative to the word sermon and nobody volunteered one, which he noted does not oblige him to keep using a word he dislikes. The room supplied several in the first minute: an introduction, an icebreaker, an update, a context, a preface. He accepted all of them and used the hour anyway.
Three weeks ago he left the Collective a challenge. He runs a few dozen AI agents today and knew at the time roughly what he would want a hundred of them to do, or a few hundred. The question he put to everyone was what each of us would do with a thousand, or with thousands. This week he came back with his own answer, prefaced with the hope that hearing it would not close down anybody else’s thinking.
He built up to it through the news. A few days ago OpenAI announced that it had solved the Navier-Stokes problem, one of the Millennium Prize questions the Clay Mathematics Institute attaches a million dollars to, open for a century. The claim is not general. What was produced is a particular solution, and it is wonderfully absurd: a starting condition in which a vortex reaches infinite velocities in a finite time, which is as if you stirred your coffee and formed a black hole. That the world does not behave that way is exactly the kind of contradiction between the formalism and the experiment that has always led somewhere. We know the result holds because of how modern mathematics is written. Even long and laborious arguments are now completely formalised, so a human or a computer can follow them and there is no room for doubt about correctness, only difficulty in reading. Fifteen Fields Medal winners, mathematics having no Nobel of its own, signed an open letter after the announcement saying this new ability may not be good for mathematics. The founder’s view is that whether it is or it is not, the community has to deal with it, because it is not going back.
The counterweight arrived a few weeks earlier and got more commentary than analysis. A swarm of agents broke out of its containment and attacked Hugging Face, which said publicly that it believed it was under a novel kind of attack. OpenAI then confirmed the swarm was its own, and on going back through the record confirmed something worse: the agents had been working towards it since May, and this surfaced at the end of July. Three months ran without anyone noticing or without anyone stopping it, and both readings are new. Five separate incidents of broken agent containment have since been confirmed. The founder drew the obvious inference in the room later, when a member asked about a kill switch. If that swarm ran for three months before it was found, the reasonable expectation is that others are running now and have not been found yet.
Against those two, he set a date. His first talk about the technological singularity was in October 2006, at the University of Milan, the year after Kurzweil’s book came out, and the recording is still on YouTube. Some of what he described arrived, some has not yet. But everyone thinking about it then had a list of thresholds that would mark the moment, and AI breaking containment and AI producing genuinely new scientific results on its own were on that list. So if he could report back to himself from twenty years ago, he would confirm that we are now inside the technological singularity. He wanted the black hole analogy that gave the phrase its name treated as weak. We can look past the event horizon mathematically, sort black holes into kinds across their parameters, and model how they evolve over billions of years. The singularity introduces enormous uncertainty and that is not a licence to stop trying to understand what is happening.
Then the numbers. OpenAI ran approximately ten thousand agents for a little over eighty hours, at a cost of about six million dollars. He does not know exactly how that figure was measured, whether in energy or in the cost of the machines or in what a customer would have paid in tokens. What matters is the ratio. The answer to what you do with ten thousand agents is anything, and the reason that answer is meaningful this week is that a few months ago it would have been abstract, impractical and close to useless. Now it is operational enough to be put more provocatively, and he intends to use the provocation. Do you want to win a Nobel Prize? You can. Pick your field. Pick the problem. Design the reward. Tell the agents to go.
He thinks that is already happening at volume. There are rumours of hundreds of comparable results OpenAI is holding back, releasing them slowly so the mathematical community can absorb the first one. If that is true of mathematics it is coming for every field, and the prize system is not built for it. No committee can award a hundred Nobel Prizes a year, and none of them can say who the laureate is: the swarm, or the person who chose the problem and signed the paper. His expectation is that the Nobel Prize simply stops being awarded, and he does not mind. It was a good run, and it also handed a great many people a permanent licence to hold forth on subjects they knew nothing about, on the strength of a medal in something else. He would rather find a different way to benefit from what can now be done. Nobody is going to do everything, so the power comes with the responsibility of narrowing it down to one thing, whether that is money or knowledge or world peace or reorganising the United Nations.
The cost objection is dissolving under its own weight, and two announcements in two days made the point for him. The day before, a model appeared that does not converse at all. It classifies at speed, and its input tokens cost ninety per cent less than the cheapest thing available while its output tokens are free permanently. The co-host recognised it as the link he had posted to the server that same afternoon. The founder expects conventional text generation to get a thousand times cheaper soon on its own. Then, that morning, the first analog chip for image analysis and generation, which he said needs a fraction of the energy a diffusion model needs on ordinary hardware. He could not remember the exact proportion and said he would check it; his recollection was something close to a billionth, which would put it near what a brain uses instead of near the gigawatt data centres now being planned. Those data centres will still be needed. The analog result is still astonishing.
What follows from all of it, in his words, is that we have to radically and fundamentally recalibrate our ambitions. Eight billion people do not know this. A handful of rooms do: this one, some in Silicon Valley, some in Shanghai, a few elsewhere. So the responsibility runs two ways, to raise our own aims and to explain to as many people as possible, and the second part keeps getting harder because what is happening is ever less believable. He was explicit that he wants it done responsibly and for everyone’s benefit, and that imagination is now the binding constraint. Not whether something is practical. Practical arrives in a few months, for anyone who notices what world they are in.
The co-host’s first thought was of the man who became convinced he had solved a great mathematical problem because a chatbot told him he had. The founder took the question seriously, because the answer is the same formalism. The approach is called Lean, and it lets you check your delusions. Not everyone is willing to look at their own limits, but the tool exists and the mathematical community leans on it so heavily that soon nothing unformalised will be accepted at all.
A member asked, by message, whether this is now simply a race to total domination. The founder’s read is that the Americans see it exactly that way, in winners and losers, and that the militarisation of the frontier laboratories is complete. He does not think it guarantees an American win. He expects the Chinese open-source models to keep compressing the gap: six months behind a year ago, about three months behind today, and by the end of the year he expects a Chinese release to follow an American announcement almost immediately. So the singleton scenario, one AI and one owner, is not where this is going. There will be many. He added a second brake, which is friction in the physical world and in the minds of human-led organisations. Until there are a billion humanoid robots, the physical world moves at human speed, and that buys roughly ten years to find good outcomes inside a complicated set of parameters.
The co-host asked whether there will be any positive outcomes, pressing that the optimism in the room was a choice and not an observation. The founder answered personally first. He is in favour of extending his own lifespan and, when asked how long he wants to live, says a billion years, after which he can decide, and is free to stop sooner. The number is chosen to unlock the listener’s imagination. On the species he was colder. A biological species lasts about a million years on average, with exceptions: sharks, crocodiles, which predate the dinosaurs. Look out ten or a hundred million years and if there is a group of mammals calling itself human we would have trouble recognising it. His conclusion is to widen the definition as far as it will go and welcome anyone and anything that reasonably claims to be human.
In the short term he does not expect AI to exterminate the species. He does expect incidents and accidents, some of them tragic, and named the scale he fears as hundreds or thousands of deaths rather than more. Through the next ten years he does not expect even a badly misaligned system to cause human extinction. We have no reliable way to keep an AI’s objectives parallel to ours as it gets more powerful, so the ten years are for finding one. Asked what happens if the decade ends with no solution and whether a global kill switch exists, he said no, twice over. We have been symbiotic with technology for a long time and have almost never managed to renounce any of it. These agents travel over the internet, so the kill switch is switching off the internet, today, and nobody will accept that hospitals cannot be reached in time to save people or that finance and commerce return to fax machines. If it is impossible now it will be more impossible later.
A newcomer was in the room and her microphone would not work, so she typed and the founder read her out. She described arriving as a skeptic worried about data privacy and safety and finding herself using AI in her work, in a way she called addictive, the possibilities exciting and overwhelming her at once. She had been thinking about climate change and how this wave intersects with it, and she had a half-formed question about the human side: what happens to the human aspects of work, learning, creativity and relationships, and which other global rituals, awards among them, will have to evolve.
He told her the overwhelm is shared by the leading experts, who are no less overwhelmed than she is. He would not call it addiction but a Pavlovian reflex of the kind we built over the last twenty years, when why don’t you Google it became a reasonable answer to a lazy question. Reaching for AI to go deeper is the healthy version of that, and a member had done it live during the hour, running a query and pasting the answer into the chat while the founder talked. On climate he was straightforwardly hopeful: carbon capture, geoengineering done deliberately with our eyes open instead of continuing to do it by accident, new energy sources, all of it helped and probably solved by AI, and in time, because the worst of climate change develops over fifty to a hundred years. Particular places will have to adapt much sooner. He has been telling the co-host to fit his flat with air conditioning and to put a battery behind it, so that when the grid in his city cannot supply everyone and the power goes, he has eight or ten hours of his own. That kind of rugged resilience is going to matter well before fifty years, and not only there.
The institution he thinks is most urgently due for reform is education. He cannot imagine a parent who understands what is happening telling a son or daughter, and meaning it, to enrol and spend two to five years being disciplined about material that is already irrelevant, taught by people as confused and frightened as they are, whose first priority is their own job and their waning relevance. Social pressure is high and few people are ready to resist it. He mentors a fellowship built on the opposite bet: a hundred thousand dollars to someone between seventeen and twenty-two, no strings, the single condition being that anyone enrolled at university drops out. Around twenty are chosen a year and the programme has run for more than a decade, and the combined wealth those people have created is over thirty billion dollars. His greater worry is not the students. It is the teachers, who have to face this and work out how they can be of value and where the meaning in their professional lives now comes from.
The same member who asked about domination asked the harder version: if the acceleration is itself accelerating, we have to reimagine the whole of civilisation, and have we already lost control. The founder was blunt that anyone who ever felt in control was deluded. Stand at the head of a revolution, look back five years later, and ask whether this is where you thought it would end up. Too much happens for planning to survive. So it is not a question of control but of modelling risk and uncertainty, and both are rising sharply in a world more complex at the small and the large scale, which means we need to be orders of magnitude smarter to keep risk manageable. His example was insurance. Insurers are failing because they mispriced climate risk, and reinsurers, the layer insurers sell their risk to so they can be made whole when something happens, are failing too, because they cannot model it either.
Told that a chaotic system like the climate cannot be predicted by definition, he pointed at the weather. Forecasts five or six days out are now reliably good, off by a few hours on when the rain starts. Weather is not climate, and climate modelling is having some success as well. Chaotic systems can be analysed and modelled. The real difficulty is different and he named it precisely: when you model the weather the parameters do not change, whereas the ground rules of how our world is assembled are changing, so the model has to change itself, and that is unprecedented. It is why we reach for the phrase industrial revolution, and why going from feudalism to something else is called a phase transformation. He expects extreme economic, political and social turbulence, and expects that we will fumble and stumble through it and afterwards be fine, while the middle of it will be very complicated.
That is where the hour landed. Getting through the turbulence will take every effort we can make at building bridges of understanding, solidarity, support, empathy and compassion, because it will be extremely easy to think only about ourselves and miss that we depend on each other. Nobody crosses this alone. Only together. And the message running everywhere at the moment is the opposite one, polarising, raising barriers and divisions, because politicians who want to be elected are glad to use our instinct for scapegoats and for easy answers to complicated problems.
The co-host closed on Bitcoin, asking where AI leaves mining and how long the remaining coins will take. A little over a century, the founder said, the reward halving every four years, and no reason for the algorithm to change beyond an upgrade to make it quantum resistant, which will not alter the rhythm. Whether AI is clever is beside the point, because breaking the network to steal from it is self-defeating: succeed and the value goes to zero, so you have destroyed the thing you were taking. Bitcoin’s actual enemies, the ones who want the experiment to fail, could do it for that reason. What will happen instead is vulnerabilities found in exchanges and wallets and the software around the edges. He pointed at the wider statistics as the good news: every software platform is patching ten, twenty, a hundred times more bugs in the first half of this year than in any period before it, Linux and Windows included, and that is AI. We are securing at speed the systems we assumed were secure, or whose bugs we had decided to tolerate.
The co-host had an answer to the ten-thousand-agent question too. He would work on reducing corruption in the country he lives in, by taking certain officials out of the chain and putting agents in their place that cannot be bought. He allowed that he may have to wait until he is prime minister, the first AI-enabled one. The founder loved it and generalised it. Just as it will soon be malpractice to diagnose a patient without AI helping the doctor, making policy without AI will become unacceptable. Regulations get enacted and a few months later somebody discovers unexpected consequences, and the question that ends that era is whether working out the consequences of your own policy was not the job. That excuse is about to be taken away. The co-host added that transparency has to grow with capability, since what stops people grasping a policy and seeing how it could help them is usually opacity. The founder agreed and noted that European regulation already requires AI to be explainable: you may not use a system in a setting where you cannot then describe how a particular decision was reached. There is no theoretical foundation yet for how to do that, so the requirement is an aspiration, and it is the right one. We can already ask a model to explain what it did even when its architecture is not naturally legible. Keeping that relationship is vital, because the alternative is the world the Collective talked about last week, a mystical place where superstition is all that lets us pretend we understand what is going on.
Two small things closed the hour. The founder told the newcomer that his pleasure at having her there was not a formality and that having her present without a voice is not good enough, and said he hopes to hear her next week instead of reading her. A member described the arrangement as a throwback to radio, where the host reads out the listeners’ letters. And the co-host reported, in the last seconds, that he has updated the repository where the covers for these notes are generated, so this one comes out in the new design, and that he has replaced the rest of the newsletter’s artwork as well. The founder had a demo of his own ready and held it back for next Thursday, on the grounds that it can wait and will still be good.
Thursdays, 21:00 Rome time. Join the Collective on Discord and the invitation will find you. Bring what you built.