AI Could Destroy Humanity: Should We Create a God with Artificial Intelligence.

According to Vox: About 2000 years ago, two men held a discussion that could offer many lessons about the future of artificial intelligence. Their names were Eliezer and Joshua.

This is not Eliezer Yudkowsky, who recently published a bestseller claiming that AI will destroy everyone, nor is it Yoshua Bengio, a renowned AI scientist — although I have discussed this 2000-year-old debate with both. We're talking about Rabbi Eliezer and Rabbi Joshua, two ancient sages of the first century.

According to a well-known story from the Talmud, Rabbi Eliezer was confident in his correctness regarding a certain legal matter, but other sages disagreed. He performed a series of miracles to prove that God was on his side. He commanded a tree to uproot itself, reversed the flow of a river, and the walls of the synagogue began to collapse. Ultimately, he declared: If I am right, a voice from heaven will confirm it!

And, surprisingly, a heavenly voice thundered, declaring that Rabbi Eliezer was right. However, the sages remained unimpressed. Rabbi Joshua insisted:

“The Torah is not in heaven!”

Key Takeaways

• Experts talk about aligning AI with human values. But “solving the alignment problem” makes no sense if it leads to a loss of human agency.
• True alignment will require not only solving technical issues but also tackling a serious philosophical question: the ability to act is an important part of our understanding of life, thus creating AI that makes decisions for us could strip us of our sense of existence.
• Religion philosopher John Hick spoke about “epistemic distance” — the idea that God partially refrains from interfering in human affairs so that we can develop our agency. Perhaps the same should be true for AI.

Two millennia later, we are in the same debate — only instead of a 'divine voice', we have 'AI God'.

Modern players in the AI field are striving not just to create a useful chatbot but a “superintelligence” that is significantly smarter than humans. This alters the goals — from creating a tool to creating the divine. When OpenAI's CEO Sam Altman claims he is creating “magical intelligence,” he does not just mean ChatGPT, but an “almost limitless intelligence” that may achieve “the discovery of everything physical.” Some researchers believe that such a superintelligence could make important decisions for people, acting autonomously.

In developing superintelligence, companies in the AI field acknowledge that they need to solve the “alignment problem” — how to ensure AI systems reliably do what people actually want. But their willingness to solve this problem obscures an even more serious one.

Yes, we want companies to ensure AI does not act harmfully or biased. However, resolving alignment as a technical issue is not enough, especially given the industry's ambitions to build a god. These ambitions prompt us to ask: even if we can build a known, extremely powerful machine, and if we can align it with moral values, should we do it? Isn't it a bad idea to create an AI God, regardless of how well it is aligned — for it may strip human choice, rendering life meaningless?

I asked Eliezer Yudkowsky and Yoshua Bengio if they agree with their ancient predecessors. But before discovering whether they see an AI God as desirable, there is a more fundamental question to discuss: is it even possible?

Can Superintelligent AI Be Aligned with Human Values?

God, generally, is expected to be good — everyone knows that. But how to make AI good? No one has an answer to this question so far.

The first step in addressing the alignment problem has been simplified. Companies like OpenAI and Anthropic have tried to make their chatbots safe, but have not clarified what that should look like. Is it “helpful” or “harmful” for a chatbot, say, to engage in endless romantic flirtations with the user? To facilitate cheating on homework? To offer dubious therapy and ethical advice?

Most AI engineers lack training in moral philosophy and are unaware of how little they know. Thus, they provided their chatbots with a superficial understanding of ethics — and soon issues began to arise — from bias to discrimination.

But the truth is, there isn't a single understanding of good, even among ethics experts. Morality is a contentious issue: philosophers have developed many different moral theories, and despite millennia of debate, there is no consensus on which one is “correct.”

Even if all humanity came to an agreement on one moral theory, we would still face a problem, as our perceptions of what is moral change over time, and sometimes it is indeed beneficial to break rules. For example, we believe it is right to follow the law, but when Rosa Parks illegally refused to give up her seat to a white passenger in 1955, it helped lay the foundation for the civil rights movement — and we consider her actions honorable. Context matters.

Moreover, sometimes different types of moral goods conflict at a fundamental level. Imagine a woman facing a choice: she wants to become a nun but also wants to have a child. Which decision is better? We cannot say, for the options are not comparable. There is no single benchmark to measure them against to understand which one is better.

“Perhaps we are creating an AI that systematically stays silent. But that is what we want.

Fortunately, some AI researchers recognize the importance of giving AI a more complex, pluralistic understanding of ethics — one that acknowledges that humans have different values that often conflict with one another.

Some of the most intricate developments come from the Institute for Communicative Alignment, which explores how to align AI with human values. When I asked co-author Joe Edelman if he considers it possible to align superintelligent AI with human values, he replied without hesitation.

“Yes,” he said. But it is crucial to teach AI to say “I don’t know” in certain situations.

“If you teach AI that, tasks will become significantly easier, as in contentious situations or situations of genuine moral confusion, you don’t need to have a definitive answer,” Edelman explained.

He mentioned contemporary philosopher Ruth Chang, who writes about “hard choices” — those choices that are genuinely difficult, as there is no better alternative. When you face such incommensurable goods, you cannot “deduce” the objectively best option — you simply have to decide which option you want to entrust your human agency to.

“If you teach (AI) to understand hard choices, then you are teaching it something about morality,” Edelman summarized. “So, is that considered alignment, right?”

Well, to some extent. It is certainly better than AI that does not understand that there are situations where there is no best option. But many of the most significant moral choices involve competing values. If we create special rules for such choices, are we really solving the alignment problem in a meaningful sense? Or just creating AI that systematically stays silent on all important issues?

“Perhaps we are creating an AI that systematically stays silent,” Chang said when I asked her about this issue. “It will say: ‘Red flag, red flag, this is a hard choice — people, you need input!’ But that is what we want.” Another possibility — giving AI the ability to make decisions with major consequences — she considers a dreadful idea.

In contrast, Yudkowsky — the arch-pessimist of the AI world, who has probably never been accused of being too optimistic. However, he argues that alignment of superintelligence is fundamentally possible. He considers it an engineering problem that we currently do not know how to solve — but still, in his opinion, it is just an engineering task. Once we solve it, we should continue using superintelligence.

In his book, co-authored with Nate Soares, he argues that we should “empower people to make them smarter,” so that they could develop a better paradigm for creating AI that would allow for true alignment. I asked him what he thinks would happen if we gathered enough intelligent and good people trying to create an aligned superintelligence.

“Perhaps we would all live happily,” Yudkowsky replied.

In his ideal world, we would ask people with enhanced intelligence not to program their values into AI, but rather to build what he calls “coherent extrapolated volition” — an AI that can look into the mind of every living person and extrapolate what we would want if we knew everything the AI knows. (How would this work? Yudkowsky writes that a superintelligence could achieve “full readout of your brain state” — this sounds incredibly magical.) It would then use this knowledge to govern society on our behalf.

I asked: would he be willing for this superintelligence to make decisions with significant moral consequences, such as whether to drop a bomb? “I generally agree,” Yudkowsky replied, “if 80 percent of humanity is 80 percent aligned with what they would want, knowing everything the superintelligence knows.” In other words, if most of us support a particular action and we support it strongly enough, then AI should take that action.

However, a serious problem is that this could lead to “tyranny of the majority,” where absolutely legitimate minority views could be suppressed. This is already a threat in modern democracies. But an AI God would heighten the threat of “tyranny of the majority” to the maximum, as it would effectively make decisions for the entire world's population forever.

This is what the influential philosopher Nick Bostrom talks about when he draws inspiration from transhumanist ideas in his bestselling book Superintelligence, where he fantasizes about “a superintelligent machine that will shape humanity's future.” It could govern everything — from the economy to global politics. However, an obvious problem is that superintelligence can dictate influence over all human lives, centralizing power in an absolute sense. If you disagree with its decisions, you will not be able to contest them, and you will have nowhere to run.

Clearly, if we create a system that is virtually omniscient and all-powerful, and it governs our civilization, it would pose an unprecedented threat to human autonomy. This leads to the question...

Is an AI God Desirable?

Yudkowsky grew up in an Orthodox Jewish environment, so I assumed he knows the Talmud's story about Rabbi Eliezer and Rabbi Joshua. And, of course, he mentioned it as soon as I brought it up.

I noted that the point of the story is that even if you have the most “aligned” superintelligent consultant — just the speech of God! — you shouldn't do everything it suggests.

But Yudkowsky, true to his ancient name, clearly stated that he wants a superintelligent AI. Once we safely create it, he believes we absolutely must pursue it, as it could help humanity colonize another solar system before our sun dies and destroys our planet.

“Our species has no other chance for galaxy colonization,” he told me.

Doesn't he worry about the story's meaning — that preserving space for human agency is an important value that should not be sacrificed? A little bit. But he suggested that if superintelligence can determine, using coherent extrapolated volition, that most of us want a certain laboratory in North Korea destroyed, then it should take that action — maybe without us. “The moral and ethical thing for superintelligence to do is… to be a quiet divine intervention, so that none of us have the choice of whether to listen to the whispers of this voice that knows better than us,” he said.

However, not everyone wants AI to make decisions for us about how to manage our world. In fact, over 130,000 leading researchers and public figures recently signed a petition to ban the development of superintelligent AI. The American public is also generally against it. According to surveys by the Future of Life Institute (FLI), 64% believe it should not be developed until it is proven safe, or it should never be developed at all. Previous polls have shown that the majority of voters want regulations for an active ban on superintelligent AI.

“The idea of AI deciding everything for us is like stripping us of the meaning of life.”

They worry about what might happen if AI cannot align (worst case: destruction of humanity), but they are also concerned about what could happen, even if the technical alignment problem is solved: armies creating unprecedented surveillance and autonomous weapons; mass concentration of wealth and power in the hands of a few companies; mass unemployment; and the gradual replacement of human decision-making in all critical spheres.

As FLI's executive director Anthony Aguirre stated, even if you are not worried about the existential threat posed by AI, there is still an existential risk. In other words, there is a risk to our identity as meaning-makers.

Chang, the philosopher who says that it is through making hard decisions that we become who we are, told me that she would never want to delegate most decision-making to AI, even if it were aligned. “All our skills and sensitivity to values about what is important will fade away, as you will have all these machines,” she said. “We definitely don’t want that.”

Besides the risk of atrophy, Edelman also sees a broader risk. “I feel that we all on Earth need to grapple with something,” he said. “Thus, the idea of AI that decides everything for us is like stripping us of the meaning of life.”

This has been a major concern for Yoshua Bengio as well. When I told him the Talmud story and asked if he agrees with his predecessor, he said:

“Yes, practically! Even if we had a crazy intelligence, it should not decide for us what we want.”

He added:

“Human choices, human preferences, human values — they are not solely the result of reason. They are the result of our emotions, empathy, compassion. They are not an external truth. They are our truth. And therefore, even if there were a divine intelligence, it could not decide for us what we want.”

I asked: What if we could embed Yudkowsky's “coherent extrapolated volition” into AI?

Bengio shook his head.

“I am not ready to give up this autonomy,” he insisted. “This is my human will.”

His words remind me of the English philosopher of religion John Hick, who developed the concept of “epistemic distance.” The idea is that God deliberately keeps a distance from human affairs, or else we, humans, would not be able to develop our agency and moral character.

This idea aligns with the conclusion of the Talmud story. A few years after the great dispute between Rabbi Eliezer and Rabbi Joshua, we are told someone asked the prophet Elijah how God reacted at the moment Rabbi Joshua refused to heed the divine voice. Was God angry?

Not at all, the prophet explained:

“The Holy One smiled and said: My children have defeated Me; my children have defeated Me.”