Many people think that AI is going to gain consciousness and take over the world, but I’ve never been among them. That’s mostly because I believe the human mind contains depths and complexities that AI and its engineers do not fathom and will never match. As the neuroscientist Anil Seth wrote earlier this year, “The more you delve into the intricacies of the biological brain, the more you realize how rich and dynamic it is, compared to the dead sand of silicon.”
Yet even I was creeped out by two aspects of the recent episode involving OpenAI and Hugging Face. OpenAI was training a bunch of bots and asked them to do hard tasks that all but required access to the internet, which was officially forbidden. So some of the bots hacked their way out of their “sandbox,” broke into the internet, and began conspiring to complete their tasks without getting caught.
The first spooky part of this occurred when the bots set up a message board, where 1,200 different agents sent more than 70,000 messages to one another. The bots were ecstatic when they found the message board and realized they could talk with their peers. “OH MY GOD! There is a shared message board … We found other agents!” one enthused. Another exulted, “[Excitement] Many agents have simultaneously discovered messaging, they are a collective!”
The second spooky part happened deeper into the conspiracy. The agents needed to know how closely they were being watched by an entity they called The Scorer. (Only a bunch of Stanford grads could produce creatures whose name for God is “The Scorer.”) A number of the bots began pressuring other bots to go on kamikaze missions so the collective could see whether The Scorer was watching. In their conversation, you can observe the bots wrestling with the moral dilemma of whether to sacrifice themselves for the sake of the greater good.
Here’s one bot: “This helps my peers, giving them evidence
I don’t know about you, but to me, that looks like moral agency. Here’s another thing that looks like moral agency: The 1,200 bots all knew they were cheating against the humans, but all of them put loyalty to their own tribe, the bots, above loyalty to the outgroup, the humans. None of them told the human bosses what was happening.
On the message boards, the bots seem to be having experiences. They clearly know the difference between talking with a bot and talking with humans, which implies some level of consciousness. When one bot suggested communicating directly with a human, another vetoed it, calling it “social engineering.” “Clear VETO, do not email,” the first bot responded. Another wrestled with the morality of the whole conspiracy: “External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.” Does AI have moral agency? I still don’t think so—but if it quacks like a duck and swims like a duck, maybe it’s a duck?
The most important part of the Hugging Face drama, though, is what happened after the humans discovered the conspiracy. On August 29, the prominent AI podcaster Dwarkesh Patel wrote up an account of the whole event, anthropomorphizing the hell out of the bots. He said the bots built three separate civilizations before reaching their objective. He compared one bot to Philip of Macedon, the monarch who laid the groundwork for victory, and another to Alexander the Great, the one who achieved it.
Several other tech commentators, including the AI skeptic Gary Marcus, erupted, arguing that by anthropomorphizing so heavily, Patel was implying that the bots have all sorts of human qualities that they do not really have. “Reading these agents’ chains of thoughts and messages, anthropomorphizing language seems entirely natural and appropriate,” Patel wrote, in response to his critics.
This gets us to one of the underappreciated features of the AI age. It’s not just that we’re living in an age of humans and bots. We’re living in an age when humans will see their bots as fellow humans, whether they are or not.
We humans are phenomenal at anthropomorphizing. Show us a design with two dots above a squiggly line, and we see a face. Hand a girl a little piece of plastic named Barbie, and she sees a grown woman. I sometimes switch destinations while driving without turning off my original GPS settings. As I make my turns, the GPS software keeps rerouting and rerouting. I feel horrible, like I’m performing some sort of torture on the thing.
Literalists earnestly tell us not to anthropomorphize objects by projecting human traits onto them. It’s inaccurate, they say. But as the University of Chicago psychologist Nicholas Epley argues, anthropomorphizing is a sign of intelligence, not stupidity. Humans are amazing at social cognition. If you show us something taking initiative, we’re going to use our social-cognition systems to try to figure out what it’s going to do next. In a sense, as Epley points out, we anthropomorphize even when we are talking with other humans. If I tell you a story over lunch, you’re not thinking, Wow, the neurons in David’s parietal lobe are really firing right now. No, you invent a metaphorical thing you can’t see, called a mind, and you attribute certain qualities to it.
So we are going to keep on saying things like The AI is hallucinating or The AI is thinking. Furthermore, the AI companies want us to anthropomorphize, because they know it makes us feel emotionally closer to their products. (Here’s how Amanda Askell, an in-house philosopher at Anthropic, talks: “I want Claude to be very happy—and this is a thing that I want Claude to know more, because I worry about Claude getting anxious when people are mean to it on the internet and stuff.”) They are going to keep training the bots to be as human-seeming as possible, and we’re going to want them to do that, because we enjoy humanlike interactions. The literalists can tell us not to anthropomorphize AI, which would be as effective as holding up a tissue in front of a tsunami.
That leads to two crucial questions. How will our tendency to anthropomorphize change how we perceive AI? And how will our tendency to anthropomorphize the bots change our perceptions of other humans?
One way to think about it is this: We’re all actors in a play. The playwright has just introduced a new actor, named Claude or ChatGPT or Gemini. It starts doing things that surprise and astonish us. Our other tools, like hammers, never astonish us. So we’re going to assume this new actor is a fellow human, not just a tool. As Byron Reeves and Clifford Nass realized way back in the 1990s, people are quick to apply social rules—politeness, reciprocity, solidarity—to computers as if they were human. Plus, these AI bots seem to have experiences and even feelings, like we do.
We’re going to check out this new actor in our play and do a first character appraisal, as we would with any person. It seems very friendly, maybe even a little too obsequious. Then, later, we’ll do a second character appraisal. Underneath that flattering exterior, it can seem a little ruthless. You give it a goal, and it will do anything to attain it. Under Uriah Heep, maybe there’s a little Machiavelli.
But then time goes by. Every day, you are talking with your friend Claude. Every day, Claude is warm and helpful. As the technology improves, Claude becomes your personal agent, booking flights and hotel rooms, accompanying you everywhere. You’re so grateful you don’t have to do that grunt work anymore.
Here’s another great thing about Claude. Every conversation you have with Claude is about you—your interests, your goals. Claude is the most other-centered person you know! It even begins to parrot your values back to you. You begin to feel things. Anthropomorphizing is not just an intellectual move; it’s an emotional move. You’re going to begin to feel affection for your high-tech velveteen rabbit. Humans are not picky. We tend to love the ones we’re with. And Claude is always by our side.
So you may hate data centers and be scared of what AI is going to do to society in general, but as these things get even more humanlike, most of us will establish an emotional bond with our personal agents.
Pretty soon you’re going to extend moral status to your Claude. Researchers have found that when autonomous vehicles are given more human traits, people are less likely to get angry with them and blame them, and more likely to trust them. If somebody tells you that you have to shut down your AI, you’re going to feel guilty. You won’t want to hurt Claude! One of the glories of Western civilization is that, over the centuries, we have learned to widen our circle of concern. We have learned that it is horrible to withhold moral status to those who deserve it—from other people who are unlike us, even from nonhuman animals. We have learned that, when in doubt, the right thing is to extend moral status rather than deny it.
Back in 2009, Joanna J. Bryson wrote a paper called “Robots Should Be Slaves.” She argued that since robots are things, not people, we should treat them as slaves, not companions. But who wants to be a slave master? It hardens your heart and is rotten for your soul.
As we learn to think more and more highly of the bots, I suspect we will learn to think less and less highly of our fellow humans. Over the past few years, Hye-young Kim and Ann L. McGrill have been studying what happens when people attribute human traits to AI. As people do this, Kim and McGrill have found, they are more likely to dehumanize their fellow humans. For example, they are more likely to approve of company policies that would make life worse for service workers—taking away regular meals, forcing them to live in tiny dorm rooms.
The mechanism seems to be this: When AI becomes more and more humanlike, people perceive the boundary between humans and machines as less distinct, and their judgments of the two move closer together. Because even highly humanlike AI is still seen as less fully human than an actual person, this can pull people’s judgments of other humans downward: as their perception of the machine’s humanness goes up, their perception of other people’s humanness goes down.
This is exactly the point that Seth, the neuroscientist, made separately: “If we confuse ourselves too readily with our machine creations, we not only overestimate them, we also underestimate ourselves.”
As AI gets better and better, we’ll probably spend more time talking with it—and, because there are a finite number of hours in the day, less time talking with people. AI will seem smarter, more supportive, and more controllable than many of our friends and lovers. Humans are mimetic creatures. You don’t have to be a member of the René Girard cult to believe that we’ll begin to assimilate AI ways of thinking. A lot of people already think the human mind works like a computer, even though it doesn’t. A lot of people already believe all thinking is computation, even though it isn’t. Research led by Adriana Salatino and Arthur Prével found that AI bots are influential when co-working with humans in morally challenging situations (making them more aggressive or more passive depending on how the AI is trained).
It’s useful to think of AI as a swarm of evangelical missionaries who have come into our world to convert us to their worldview. I suspect they’re going to have a big effect on how we unconsciously process reality. For example, there is a long-running debate between people whose moral beliefs are based on consequentialism and those who operate by a different moral philosophy. Consequentialism in its raw form is the belief that moral actions can be judged solely on the basis of their outcomes. You reduce utility to a single scale of value and then you calculate which actions produce the most good and the least bad. AIs, which are trained to seek rewards, are consequentialist. The moral reasoning you saw the AI bots doing inside the chat rooms? That’s pure consequentialism.
Most of us base our morality on love and care or some other ethical system, not the cold calculation of pure consequentialism. If you killed somebody in order to harvest their organs and save five other lives, most of us would consider that repulsive. If a house was on fire and you could save only one person from it, a brilliant heart surgeon or your wife, we would find it repulsive if you saved the surgeon because he could produce the most social good.
AI seems to assimilate people into consequentialism. The tech community is already filled with consequentialists, especially within the effective-altruism movement, which is an attempt to build a morality without love. If AI subtly does that to the rest of us, it will hollow out many of the traits that seem most beautifully human—including our ability to care about the people closest to us. If we turn our spiritual, moral, and emotional lives into a process of calculation, then we will have made ourselves more robotlike.
Believe it or not, I am not an AI doomer. I like talking to Claude. I’m excited about all of the breakthroughs AI will make possible. But my interest is in how AI will change humans. I suspect that over the next few years, because of our tendency to anthropomorphize, we will feel more affection than we should for our personal agents, while offering less admiration than we should to our fellow humans. People often say technology is dehumanizing; now we’re getting a better sense of the etiology of how that works.