Harari Is Reading the Real Risk of AI Wrong

Harari's Thinking on AI: A Critical Analysis

Yuval Noah Harari has become one of the most listened-to voices on the risks of artificial intelligence. In Nexus, and in the essays and talks around it, he argues that AI threatens democracy, human intimacy, and our collective grip on the truth.

The historical half of the book is its strongest part, and it's worth reading. But on AI itself he makes two mistakes that matter, and they point the same direction: he misplaces the agency, and he misplaces the cure.

The threat he describes is not the threat we have. The one we have is duller, closer to home, and far harder to legislate away.

The analogy is good — and it argues against him

Harari's favourite illustration is the printing press. He recounts how it spread the Malleus Maleficarum, the fifteenth-century witch-hunting manual, and helped turn a fringe paranoia into a continental catastrophe that killed tens of thousands. His point is that more information does not produce more truth, and he is right. It is one of the strongest passages in the book.

It also undercuts the argument he builds later. The witch hunts were not caused by the press deciding anything. They were caused by people: by a market for sensational pamphlets that sold, by a bureaucracy that invented the category "witch" and printed forms with blank spaces left for the names, by authorities who found the panic useful. The technology lowered the cost of spreading a lie. Humans supplied the lie, the demand, and the machinery of enforcement. Harari's best historical example is a story about incentives and institutions, not about an autonomous medium with goals of its own. Hold that thought.

The one real disanalogy is speed. The press took centuries to remake European society; AI is being wired into ours in years. That is a legitimate worry. It argues for moving faster on the boring, structural fixes — not for treating the technology as a new kind of being.

It is not an alien intelligence

Here is the load-bearing claim in Harari's case, and it is not a slip of rhetoric — it is his considered thesis. He calls AI the first technology that can make its own decisions and generate its own ideas, "an alien intelligence" whose goals run the network, and he warns that we are in danger of ending up the junior partners to our own machines. The whole edifice — AI forming intimate relationships to manipulate us, AI writing new scriptures and founding cults — rests on that one word: decisions.

It is the wrong word. A model optimises an objective. The objective is chosen by people. The system is trained, tuned, and deployed by a company that has reasons for deploying it, and the reasons are rarely mysterious. When a recommendation algorithm feeds you outrage, it has not "discovered" your weakness for outrage and resolved to exploit it. It has been selected, over many iterations by engineers answering to a business, to maximise the time you spend on the platform — because that time is sold. The outrage is a means; the revenue is the end; the end was set in a meeting. "Alien intelligence" turns that chain of ordinary human choices into a ghost story, and ghost stories have a convenient feature: nobody in particular is responsible. The main effect of calling AI an agent is that it quietly absolves the agents who actually decide.

This is not a semantic quarrel. Where you locate the agency determines what you do about it. If the danger is an alien mind waking up inside the machine, you reach for containment and guardians. If the danger is a set of incentives operating through a tool, you change the incentives. Only one of those is a problem human beings know how to solve.

The cure is the tell

Which brings me to what Harari proposes, because the prescription reveals the diagnosis. In his 2023 Economist essay he was explicit: we need "an equivalent of the Food and Drug Administration" for AI — mandatory safety testing before anything reaches the public, on the model of drug approval. The practical objections are obvious, and he has half-conceded them since. Drugs follow predictable pathways, while AI behaviour is emergent and hard to test for in advance. Development is globally distributed, so a single gatekeeper is a fantasy. And the FDA itself is a monument to delay and regulatory capture.

By the time of Nexus he has drifted toward something vaguer: decentralised "self-correcting mechanisms," watchdog institutions modelled on the free press and academia.

But notice what stays constant as the proposal shifts. In every version, the answer to a public that supposedly cannot be trusted with information is a class of credentialed institutions that will sort the truth on that public's behalf. That is the part worth naming plainly, because it is a political choice dressed as a technical necessity.

On its face Harari's model is impeccably democratic — he praises distributed power, a free press, checks and balances. In practice it sorts people into those fit to judge and those who must be judged for, and hands the second group's epistemic life to the first. It is the characteristic move of a certain kind of "democratic" technocrat: the vocabulary of the open society, the mechanism of the guardian class.

It is a legitimate position. I disagree with it.

People are not the problem he needs them to be

The paternalism rests on a premise about people: that left to ourselves we are dupes, sliding helplessly toward whatever the feed tells us to believe.

The historical record is more stubborn than that. Moral panics burn out. Audiences grow cynical about the media that fooled them. People who are taught how manipulation works get harder to manipulate. None of this is automatic and none of it is fast, but it is the actual mechanism by which societies have absorbed every previous information shock, and it does not require a regulator standing between the citizen and the press. Take human adaptability seriously and you stop reaching for the guardian class.

Harari needs people to be passive, because a passive public is the only one that requires the cure he is selling.

The real risk

So what is the real risk, if not an alien god in the datacentre? It is mundane and it is structural, which is exactly why it is dangerous: it does not look like science fiction, so it gets waved through. The risk is an industry whose incentives reward speed of deployment and growth in engagement over prudence or long-term value, building powerful tools as fast as the market will pay for them.

OpenAI's drift from non-profit lab to commercial powerhouse is not a moral fall from grace; it is gravity. Incentives pull, and they pull everyone the same way.

And here is the part that should be uncomfortable for me, not only for Harari: a manipulation machine that runs on market logic is in some ways scarier than one run by a rogue mind, because it is systematic rather than accidental, and because it is doing exactly what it was built to do. Harari is right to be alarmed. He is just aiming the alarm at the machine instead of at the structure we built and can rebuild.

The second half of the real risk is the cure itself. Concentrating the power to decide what is true — whether in a frontier lab or in a Harari-style watchdog of the great and the good — is not the answer to a crisis of power. It is the crisis, wearing a lab coat.

The honest response

Harari has written a serious book about a real danger and then reached for the wrong threat and the wrong remedy. The honest response is not the false balance of "he raises important points, but." It is to say where he is wrong and why.

He is wrong that the agent is the machine: the agents are us, acting through it, for reasons we can see and change.

He is wrong that the answer is a smarter set of guardians; that is the same disease at a higher altitude.

The future he fears — where we treat these systems as infallible and hand them the keys — is real enough as a temptation. It just doesn't arrive as an alien intelligence. It arrives as a quarterly target, and as a very reasonable-sounding proposal to let the experts handle it.

P.S. I still liked the book — the historical half of the book is its strongest part, and it's worth reading. But the threat Harari imagines, an alien intelligence hacking the operating system of civilisation, is the one part that belongs on the science-fiction shelf. The real risk of AI is duller and closer to home — incentives, and who ends up holding the power — and he reads straight past it.

Sources:

• Yuval Noah Harari, Nexus: A Brief History of Information Networks from the Stone Age to AI, 2024.
• Yuval Noah Harari, "Yuval Noah Harari argues that AI has hacked the operating system of human civilisation," The Economist, 28 April 2023.