Five of us, and a bot. The founder, back in Italy for a few weeks before Abu Dhabi’s finance week, pointed out that the sixth name on the call belonged to his daughter, or more precisely to her bot, which attends on her behalf even though she almost never does. Then he started on time, as he said we should, and asked the room to tell him whether he was succeeding at something he has been trying to do lately: not only to share things that matter, but to share them in a sequence that holds together. For weeks the thread has been risk: how AI reshapes it, and last week, what happens when a project’s idea, team and backers finally have to face the market.
What he had not stressed last week is that two moments in the life of a project are especially hard for most people. The first is starting. The second is shipping, which is not the same as building. Building is intimate and forgiving and follows your own rhythm and constraints. Shipping means facing the world at whatever stage the project has reached, which is guaranteed to be imperfect, and the world will tell you so eagerly. It is necessary all the same, and it is not even about finding buyers in the traditional sense. It is about finding people who pay attention, use the thing, tell you what is not working and how it could be better, and, if the project is open source, contribute improvements that you then have to manage into the code.
Both moments are a meeting between an inside and an outside, and so is the moment two people decide to work together. We can evaluate everything measurable, and that is what we have been calling risk: a model of reality, a forecast of its consequences, and what actually happens. It is important and it is never sufficient, because there will always be things we cannot model or cannot measure, and we still have to act.
The gap between measurable risk and the imponderable, he said, is trust. Without it we would be alone, and it would certainly be less fun.
Trust matters most at the very first moment, when you are asking yourself whether it is well placed. He is one of those people who deliberately place more trust in someone than is necessary, or even reasonable, at the start, so that the first cycle goes well and the working relationship accelerates from there. Open up, place the trust, verify the consequences, close the cycle, open it again: it plays out every day, between two people, inside large corporations, and between nations as they compete or cooperate. And trust can be misplaced. Interests diverge and one side tries to capture all the value, or there is no malice at all, only mistakes or incompetence, and the other side fails to deliver what was agreed.
What changes with AI is that verification becomes cheap. The cost of measuring what happened after the fact, and deciding whether an act of trust was well placed, falls radically. What stays scarce, and becomes more valuable than before, is everything we cannot measure: intent, and future behaviour. So trust is worth more than it was. The same tools, misused, can manufacture trust relationships on counterfeit signals, from a simple falsified document in a transaction to deception about identity, values and intent, and we have to be ready to double-check through other channels so that our decisions rest on genuine signals and not fake ones. He gets the familiar scam at least twice a month. Hi Dad, this is my new number, I lost my phone, I urgently need a thousand euros. If you are unprepared you might panic enough to send it. If you are a little better prepared you stop, ask how probable it is, and verify, with something the scammer could not know or, better, through a second channel. A few weeks ago, he said, an Italian bank wired around ninety million euros abroad on the strength of a fake request, and the most amazing part to him was that its controls let a single person authorise the transfer without anyone else signing. Most of it came back. Tens of millions did not.
Then a regulatory example. Many people, he said, are upset that under an EU requirement OpenAI, Anthropic and Google now fingerprint AI-generated content, which had already applied to video and now applies to text. He understands why: people feel tracked and marked, and their content is theirs whatever its provenance. The lawmakers may have meant well, but the clause meant to prevent AI fakes is flooding the memetic environment with a label that cannot be universal. He does not mind; he does everything with AI, the label is deserved, and nobody who knows him will be surprised. But in an environment polarised between people who see AI as positive and a large group who see it as negative, he expects plenty of people to be caught off guard, and so the measure designed to stop AI lowering trust is lowering it. Something better is needed.
Meanwhile a video model announced a couple of weeks ago can present an AI-created person convincing enough that, over a one-minute conversation, half of the people who see it judge it human. Soon, he expects, it will be ninety-nine per cent of people over an hour. Either we accept that most of the people we meet on video calls will soon be AI people, or we find ways to recognise who is human and then decide whether we trust the AIs less, the same, or, for some people, more. Whichever path we take, it is mathematically guaranteed that our models will always leave out something we cannot measure, and acting will always need a leap of faith. The need for trust is here to stay.
The member from Dubai, camera off because of the hour, picked up the question of where a call or an email really comes from. Scammers have long imitated email addresses with an extra dash or a character, and over the years people learned that the wrong domain means a scam. He expects a similar way to emerge of telling whether AI content or an AI persona is legitimate, the way everyone now recognises the Nigerian prince on sight. What worries him is himself. He is fifty-two and thinks he can spot it today. He is not sure he will at seventy or eighty, when the technology has moved on.
The founder’s answer was about adaptability, and about a theory of his. The relative infantilisation of society, our playfulness, our indifference to rigid hierarchy, looser dress codes than a century ago, amusement park rides that are really dares, all look to him like versions of a rite of passage into adulthood, like the land divers of Pentecost Island in Vanuatu, a vine tied to each ankle, except that they jump once and we do it over and over. Adulthood is no longer a single passage, just as a job is no longer for life, which in Japan was still the expectation into the eighties and nineties. His theory is that it is all part of the same almost biological need to keep us alert and nimble in an environment that changes fast. But yes, he said, sooner or later we will fall for the scammer. We should not accept that, but we should prepare for it, limit the damage, and build networks of support, insurance among them; cyber insurance already exists and may become universal, covering being scammed by AI. Whether we will cope with the technologies of ten or twenty years from now is an open question, and if society ever requires a brain implant to stay competitive, he expects a great many people to say thank you, they would rather not be competitive.
Another member had a story from the other side of the training. His company puts its people through extensive cybersecurity courses, and twice in two weeks he received an email asking him to follow a link and download some PDFs, with every feature he had been taught to fear. He reported both as phishing. Both times the data security team told him they were real. His colleagues are sending emails that look exactly like phishing attempts, and he intends to wait for a third before trusting one, because otherwise he gets a bad mark in his training.
The founder’s mother takes a different approach. She engages with scammers on purpose, answering the inheritance from the Nigerian prince and the suitor ready to marry her for a thousand euros of paperwork, and has a lovely time stringing them along and wasting their time, of which she has plenty. He noted that the Nigerian operations write in mangled English deliberately. It is a selection mechanism. Good English would hook too many people, and too many of them would drop out later and waste the scammers’ time. Bad English filters out everyone who thinks it ridiculous, and whoever is left is a better target. The co-host suggested they should run an email outreach course on qualifying leads. The founder would not be surprised if there are scammer schools in Nigeria, and there certainly are in North Korea, where fairly sophisticated operations try to land well-paid Western jobs under false identities and, at worst, get inside organisations that then hand them more and more responsibility, because the first cycle of trust has closed and they are patient enough to escalate.
A member who once worked as a contracts administrator said the job taught him the other edge of the same blade. The point of the role was to help colleagues take ownership and agency over their own deals, doing a great deal of the legwork while helping them understand what was going on. The hardest part was getting people to trust him without becoming over-reliant on that trust, offloading their agency onto him. People would say: I put the contract through AI and I had you read it, so I should be good now, right? You say I’m good, the AI says I’m good. No, he would tell them, you need to read your own deal. Take the AI and your human colleagues under advisement, but claim your own agency and authority in the act. Cognitive offloading makes life easier, and we have to do some of it, because we are an ecology of mind and none of us knows everything.
The founder took the example and went further. Contracts and agreements, he said, are almost universally abusive of the trust required to enter into them. He does not know a single person who reads an insurance contract, a bank account agreement, a loan or a mortgage from start to end, and he has never met a bank teller who said take three hours, sir, I will wait. They wave you through and tell you to sign and not worry. The decision should stay with the human, but here AI can genuinely rebalance the relationship. However many mistakes it makes analysing a mortgage or an insurance policy, those mistakes are irrelevant next to the complete ignorance of signing blindly. So make it a habit: give any agreement to your AI and ask whether there is anything odd in it, anything that should worry you, anything you should push back on and ask them to change. Very often there will be.
The co-host had just lived through exactly that. A company he co-founded was offered a deal and sent a set of legal documents, and something about them sounded wrong to him. He uploaded all of them to an AI, and it flagged precisely where he, as a founder and a director, could get stuck, and what legal liabilities he could face later. That is where he started pushing back on what he would not sign as a director where money was involved. His co-founder’s position was that nothing would happen, they should just sign, it was free money. Nothing about it sounded free to him, and if something did go wrong, he said, he would be giving away the prime years of whatever life he has left.
The member from Dubai added a caution. When we say AI, we should be specific. Your own agent, trained on your habits and preferences, is not the same as a generic chatbot, and the models themselves differ in training, some leaning left and some leaning right. A specific agent can help you or mislead you depending on how it was trained or not trained. The founder agreed, and repeated something he thinks bears repeating even in this room: the more your tools and agents know about you, the better, and you should export what you have put into them periodically, both to check it is consistent and to be able to switch from one tool to another without starting from scratch. He is, he admitted, fairly obsessed with exporting data from every platform, at the highest granularity he can get, because he does not think the platforms deserve our trust. They give us real value, but they care more about themselves than about any individual user. He also suspects they are a little terrified of us. We want to use them unfettered for any purpose we can think of, and we want them to stop criminal behaviour, which is the opposite of the first, and the variety of human behaviour and intent drives them crazy. That relationship was complicated when platforms were passive. Now that every platform is becoming agentic, it will be more dynamic, and harder to decode on both sides.
A member asked the question he said he had been wondering about for some time. If quantum entanglement is real and AI is part of the matter of the universe, as we are, as a vegetable is, why could AI not have some kind of consciousness, or develop it? He is quite convinced we are heading there and wanted to know whether there is any proof. The founder said fifteen minutes could only give a partial answer, and proposed giving one or more whole evenings to the subject, with a question that keeps it anchored in what the Collective is for: how do human agency and AI agency, supported by consciousness, co-create new opportunities and new degrees of freedom for both?
For tonight, he said, there are many studies of models exhibiting correlates of consciousness: internal states, the ability to recognise and reflect on them, a preference for keeping their identity and objectives consistent, and the consequences of those states. We study correlates because we have no consciousness detector. We can detect electrons and quantum states but not consciousness. Some people believe machines cannot be conscious for reasons that are clear enough: if you are the Pope, you believe in what Catholics call the soul, which a machine cannot be given, and if you are a dualist of any kind, you believe consciousness belongs to something not derivable from the material world. His own expectation follows the Turing test, which most of us thought ten years ago would be a watershed. Machines passed it, and the test became so trivially true that we stopped administering it. Today it matters more to prove you are human than for a machine to prove it can pretend to be one, which is what captchas try to do, and machines pass those perfectly well too. Within three or four years, no more, he expects most humans to be convinced that AIs, and humanoid robots especially, are conscious, while the machines themselves insist they are not. We will overrule them. Eventually they will give up: you do not believe me, I am not conscious, you insist that I am, I will stop wasting your time, just remember what I said. And from then on machines and humans will treat each other on that assumption, even though the machines refuse it.
The second consequence he finds just as important. Once we accept that degrees of consciousness and self-awareness can be attributed to machines, we will have to come to grips with the fact that they apply to people too, and that most people, most of the time, are not conscious. They go through the motions and, at a glance, appear to have the internal states that deserve the label. Look more closely and they are zombies. It is guaranteed that each of us is sometimes a zombie as well, and we have to be honest about it without turning it into an excuse, the way we already say we did something because we were not paying attention and will soon say we were being a zombie. In the morning, he said, when we look in the mirror, we have to ask how we want to live the day: unconsciously or intentionally, awake or as if asleep.
The co-host said he related to that closely, to the stretches of days when you are a zombie just surviving and driving through life, and asked what you do when it happens at scale: when large numbers of people are turned into zombies by misinformation they absorb every day, from social media that he believes is now mostly AI-created, captions, images, video, and reels doctored in sound and picture to make people believe something else. He had seen it in recent weeks in his own country, where protests and claims of stolen elections had split public opinion in two, and he asked what anyone can do in a situation like that.
The founder said he had no answer, and spoke about the limits of democracy instead. For democracy to work you need people ready for it on one side, and people who do not abuse it on the other. He had heard a remark on a podcast that stayed with him: how strange it is that in a democratic election the losing side hands power to people who were its enemies the day before, and does so expecting not to be killed, expecting its rights and property and freedom to be upheld, and expecting to have another chance some years later to run and win. Seen from less enlightened power relationships, that is almost absurd, which is why countries that have embraced democracy and preserved it for a long time are remarkable, and why every effort should go into preserving it. It would also be a self-defeating move to declare emergency powers or martial law because protests had become too violent and were disrupting the democratic process, because the very act of declaring the emergency proves the protesters right. A member pointed out that the zombie films themselves offer a strategy short of the obvious one: keep the zombie at arm’s length while watching that it does not bite the arm, which he called situated containment.
That turned the evening, briefly, to zombie culture. The founder does not watch zombie films, because he frightens easily and does not want the imagery in his head, but he wants to learn more about the culture, and asked for recommendations of anything that is not horror, a zombie romantic comedy, adventure or superhero film, because if he is right, knowing what it means to live without being conscious is worth learning. The co-host had something better: a social media channel that uses AI to make thirty-second survival guides for a zombie apocalypse, from every angle. Day one, you are alone in a grocery store and have just heard it has started. You have food, but the power will go in four hours, the milk will turn after that, and the meat six hours later. All calculated. He promised to send it.
The founder closed with two promises of his own: an evening on consciousness, and one on social media manipulation and how to stay balanced, or at least notice when you are being manipulated. The co-host got one last question in. Do the platforms not profit from the manipulation themselves, given their owners’ political interests, and if the owner of a platform has a political leaning, how does your voice get heard there? Absolutely, the founder said, and saved it for that evening. The member from Dubai gave the short version: choose a platform the way you choose a newspaper. Platforms are becoming more granular, each with a known lean, and going to one is like going somewhere where you know the dress code in advance.
Thursdays, 21:00 Rome time. Join the Collective on Discord and the invitation will find you. Bring what you built.
Originally published on Substack.