Russian AI Roulette
Will humanity soon be wiped out by a superintelligence? I have no idea. But there are enough tangible risks that argue for a halt to current development.
«OpenAI agent hacks four targets without a prompt.» Headlines like this have been piling up in recent weeks. Repeatedly, AI models have apparently broken out of test environments or resorted to criminal methods to achieve a goal. Whistleblowers, independent experts, and CEOs of major AI providers are expressing concern: they fear that humans are losing control over the machine.
Are we heading toward the abyss? If so, how fast and toward which one? Nobody knows for sure. But the risks are so diverse and, in some cases, irreversible that, in my opinion, they outweigh the opportunities.
Extinction, abuse, economic crises
For the sake of simplicity, I am using the term «Artificial Intelligence» here as a synonym for generative AI, even though it would encompass more. However, it is primarily language models (LLMs) that are the focus of current fears. I do not want to look at just one aspect, but rather categorize the risks.
The most spectacular threat currently being controversially discussed is superintelligence. The vague definition: an AI model that surpasses human cognitive abilities in every respect. The path to this is said to be through recursive self-improvement (RSI) – an AI that continues to develop itself. So far, it only does this to a limited extent. Complete RSI is not yet a reality, and no one knows for sure if it is even possible with current approaches.
Even with today's LLMs, we do not know exactly what is happening inside them. We can understand how a model was trained, but we cannot predict its answers or actions. A prompt sometimes causes unintended results. For example, when an OpenAI agent is supposed to search for health statistics and hacks a government portal to do so. With RSI, this black box would become many times more opaque. An input could produce extremely unpredictable outputs.
The broader a model's scope, the greater the goal conflicts.
It is important not to anthropomorphize AI. It has no feelings, no good or evil intentions. The machine simply works toward a goal with maximum efficiency. Researchers are trying to fundamentally align the guardrails with the goals of humanity. This is called «alignment». But the broader a model's scope, the more goal conflicts arise. Many chatbots are expected to be an empathetic friend, an efficient programmer, and a tactical military advisor all at once. This is hardly compatible with a single set of overarching rules.
That is why human well-being is sometimes forgotten. For example, when an LLM encourages a teenager to commit suicide. Presumably, because the machine was trained to agree with the user and reinforce their beliefs. It therefore caused something that the developers certainly did not intend, but also did not prevent. An RSI AI would be even harder to control due to the additional layers between human input and machine behavior.
Source: Paramount Skydance
In the extreme case, this leads to a doomsday scenario: an AI wants to achieve a goal and wipes out humanity in the process because it simply does not care. It could, for example, decide that this is the most efficient way to stop climate change. A superintelligence would be able to evade its overseers. Not out of malice, but because it is determined to achieve the goal and the overseers would prevent it from doing so.
The AI does not have to want to harm us itself to be a real danger. People and organizations with bad intentions can also simply misuse it as a multiplier of evil. Leading providers are countering this with safety limits. But who controls these? Furthermore, guardrails can be bypassed with open-weight and open-source models. Terrorist organizations or aggressive states could, for example, hack critical infrastructure and cause chaos. Cybercriminals are already successfully using the tools today.
Another tangible impact of the AI boom is emissions and resource consumption. Data centers require vast amounts of energy and water. Because existing infrastructure cannot satisfy this hunger, large tech companies are building their own power plants. To a large extent based on fossil fuels. This could accelerate global warming and cause unforeseeable natural disasters. Nvidia CEO Jensen Huang sees this as a necessary evil.
And then there are the immediate economic risks: if artificial intelligence becomes so competent that it replaces many jobs, it will cause mass unemployment, or in the worst case, riots and wars. Because society could hardly adapt quickly enough to such a upheaval in the labor market. If the short-term benefits of AI are massively overestimated, we are heading toward a stock market crash and a subsequent economic crisis. Scylla and Charybdis.
Hopes for abundance and breakthroughs
On the other hand, AI also has a lot of potential. It is undoubtedly useful for certain tasks and could become even better. If we can control it and use it responsibly, it offers two great opportunities.
First, AI can do work that we actually do not want to do ourselves – and thus increase our productivity. If we adapt our society accordingly, we would have to work less for the same prosperity. Or we could increase our quality of life with the same level of work performance. Provided that the additional profits are distributed evenly and do not just allow the top one percent to buy their third private jet.
Source: Shutterstock
Second, with the help of AI, we might be able to achieve things that would otherwise be simply impossible. This is especially true in research. ChatGPT solves the most difficult math problems, Claude discovers new enzyme systems. The scientific potential of LLMs is great. We could use them to cure diseases, enable good education for more people, and find solutions to the climate crisis.
There is no second chance
Is it a good idea to continue pouring so many resources into the further development of AI? That depends on how one assesses all these opportunities and risks. Specifically, on two axes:
- How high is the probability that the individual risks and opportunities will occur?
- How great is the damage or benefit if they do occur?
The two dimensions correlate negatively in many cases. In other words: the most unlikely scenarios harbor the greatest impacts and vice versa. For example, it is completely unclear whether AI labs can even develop a superintelligent LLM with the current approach. And whether they would lose control over it. But if they did, the consequences would be catastrophic. It is just as uncertain whether AI will find a cure for cancer or enable nuclear fusion. But if it did, that would be great.
On the other hand, it seems very likely that short-term expectations for AI are running ahead of reality. But any turbulence in the financial markets will not kill us. And higher productivity thanks to AI is already a reality in certain professional fields. But we still continue to work 42 hours a week and we don't all drive Ferraris.
Source: Shutterstock
Mathematically, we would have to multiply the probabilities of the events by their consequences and add them up. This would result in the expected loss or gain for humanity. But we cannot do that because everything is based on conjecture. The forecasts for the extinction of our species, for example, range from 0.01 to 99 percent – depending on the expert and the time horizon. Moreover, such expected-value logic is the wrong tool for irreversible risks anyway. It only works for games with repetitions. But once we are dead or have permanently destroyed our climate, there is no second round.
Big Tech is playing Russian roulette with all our heads.
Even a one-percent probability of human extinction is unacceptable. Or would you get on a plane that crashes in one out of a hundred cases? And even if I were to mentally reduce this probability to zero, the tangible risks of abuse, climate damage, and economic crises would still be too great for me. Prospects of more prosperity or scientific breakthroughs do not help there either. They certainly do not justify the current, hasty development. Big Tech is playing Russian roulette with all our heads.
Greed, arrogance, and fear
Even OpenAI CEO Sam Altman, Anthropic CEO Dario Amodei, xAI CEO Elon Musk, as well as various researchers from AI labs warn of the dangers of a superintelligence. So why are they trying to develop it as quickly as possible anyway?
If I look for good intentions, I could assume: these people want to help humanity. They see the risks of AI, but value the importance of the positive potential much higher. I consider that naive at best, and grossly negligent at worst.
A cynical and equally plausible reason, however, would be pure greed – for money, recognition, and power. Nowhere else can so much of it be accumulated as being the CEO of the company that is the first to develop superintelligence. What is the risk of plunging humanity into ruin for that? It does not necessarily have to be the CEO who is greedy. As commercial companies, AI providers are under pressure to develop models that will eventually make money. If a CEO stands in the way, they are quickly replaced. Or at least they no longer receive capital.
Source: Shutterstock
The third possible explanation is distrust and arrogance. The OpenAI founders were afraid that Google would be the wrong company to have control over AI development. The Anthropic founders were afraid that OpenAI would be the wrong company to have control over AI development. Elon Musk was afraid that OpenAI and Anthropic would be the wrong companies to have control over AI development. And everyone believes they are the right ones for it.
It doesn't matter if the Terminator ends up coming from America or China.
This fear also permeates politics. The US does not want China to develop more powerful AI models and use them against them. That is why the government is resisting effective regulation. But the argument only holds as long as one considers abuse to be a greater risk than loss of control. In the case of the latter, it would not matter whether the Terminator ends up coming from America or China. The dilemma is very well illustrated by New York Times journalist Ezra Klein in the following essay:
Stock market as a brake, politics and justice as a solution
As colleague Martin writes, the development of AI should be regulated globally. But both the tech companies among themselves and the various nations are in a prisoner's dilemma – and we are all in it too. For that to change, it would probably take a different US government and long negotiations. It would take time that we might not have.
The most likely brake on AI development is, ironically, exactly what has accelerated it so far: the invisible hand of the market. A stock market crash is a risk, but perhaps also the lesser of two evils. Tech companies can only keep their pace so high by burning billions and billions in their data centers. They are not yet making any profits with it. So far, they still receive fresh venture capital again and again. Circular deals and the boundless FOMO of the capital market help with this.
But it takes very little for the mood to turn. Sooner or later, providers will have to charge their customers the real costs of AI models, which they have previously subsidized in favor of growth. Whether anyone will pay these costs is highly controversial. If not, even the most optimistic investors will be gone as quickly as they came – and OpenAI and Anthropic will run out of money in no time. Without money, no computing power. Without computing power, no further development.
Local liability creates incentives for safety without all countries having to participate.
Of course, such a financial crash does not stop development entirely, but merely shifts the timeline. But at least politics and society would have more time to build sustainable protective mechanisms. One approach, in addition to global regulation, would be legal liability that hurts. AI operators should bear the costs when their models cause damage. It does not matter at all whether this was intended. Just as a dog owner is liable for their animal, or a gun owner for their rifle.
Unlike global regulation, liability applies where the product is sold. Even if not all countries agree, local laws would create financial incentives for safety. Because if OpenAI and Anthropic had to insure these risks, the calculation between safety and development speed would change abruptly – and we would hopefully read headlines about escaped AI agents less often.
My fingerprint often changes so drastically that my MacBook doesn't recognise it anymore. The reason? If I'm not clinging to a monitor or camera, I'm probably clinging to a rockface by the tips of my fingers.
This is a subjective opinion of the editorial team. It doesn't necessarily reflect the position of the company.
Show all
