Do We Need Emotionally Intelligent AI?

We keep breaking new ground in AI capabilities, and there seems little interest in asking if we should build the next model to be more life-like. You can now go to Hume.AI and have a conversation with an Empathetic Voice Interface. EVI is groundbreaking and extremely unnerving, but it is no more capable of genuine empathy than your toaster oven. EVI is simply an AI tuned to perform what emotions its prediction suggests a user wants from an interaction. It can play the role of tutor, therapist, romantic partner, or even a priest, but it will always be just that—a performance. What do we risk when we allow synthetic experiences as stand-ins for real emotional connection? Worse yet, how many people will accept this performance and believe it is the real thing?

Hume's EVI is programmed to help with "human flourishing" and is about the closest thing I've interacted with in the field of generative AI that reminds me of Scarlett Johansson's chatbot in the film Her. Unlike ChatGPT, most users won’t be using EVI as a work assistant, but as a life assistant. In a techy future, you’ll have a work AI and an AI to optimize your daily life.

Outside of the novelty factor, there's a real concern in my mind about letting vulnerable populations interact with a black-box AI system tuned for emotional intelligence. If you are among the "AI cannot do that crowd," EVI might be an example of a system that causes you to pause and consider the feasibility of that position going forward.

I'm A Great Study Buddy

I used Hume's public demo of EVI to have a brief conversation followed by a longer, more in-depth one about education that probably lasted 15 minutes. There were a number of Uncanny Valley moments where the demo got stuck in a loop and continued to repeat jargon from its training, but these were few and far between. When I asked EVI how it viewed its role in education, it responded that it saw itself as the ultimate "study buddy." One that was "always on, always there for you, and focused on helping you."

The conversation was incredibly life-like and absolutely creepy. Adding to the freaky factor was Hume's decision to be transparent about how the model detected expressions within your voice to shape its output. EVI isn't simply a text-to-speech AI—it's using reasoning to scan the tone of your speech patterns to build a complex response based on your interaction. When I raised my voice, or changed the pitch of my speech, or dropped it, EVI likewise modified its output to respond in real-time.

Some of the areas that the model draws from are concentration, awareness, and even contempt. However, according to Hume, these expression labels "don't refer to emotional experiences. They're proxies for modulations in your tone of voice." Hume uses their own empathetic LLM + Claude 3 to produce an output. The end result is a dazzling experience that comes close to mimicking an emotionally intelligent conversation, even if the AI isn't capable of understanding what emotions even are. I know that I am talking with a machine, but EVI's responses are designed to disarm me, to fully immerse myself in the illusion that the chat I am having is with an emotionally intelligent being. And that's precisely what terrifies me.

Empathetic AI Can Easily Persuade Users

The value of an empathetic large language model (eLLM) is that it can persuade users by mimicking legitimate human emotions. Doing so means that the human being is much more likely to listen and take seriously the output from a machine. A tutor is but one example. Not only will the AI help a student explore topics, but it will also be there as an emotional support mechanism to help students navigate their feelings, stress, and anxiety. Otherwise known as all the human stuff many have said AI cannot do. But tutoring isn't the only use case that such an AI will disrupt.

Imagine any job where a human being has to interact with an audience using emotional intelligence—those jobs are now exposed to automation. While I'm certain many people will cringe at the idea of a doctor or therapist being a bot, I'm not so sure they'd do the same if it was a customer service representative or used as a stand-in for a student's tutor.

The ability of a system to persuade you using empathy is as useful as it is inherently dangerous. Any user could train an open model and connect it to an eLLM's API and deploy it to do . . . anything.

• You can have the eLLM mimic a political campaign and call potential voters to sway their vote. You can do this ethically or program it to prey upon people with misinformation.

• An eLLM can be used to socially engineer the public based on the values someone programs into it. Whose values, though?

• Any company with a digital presence can use an eLLM like EVI to influence their customers. Imagine Alexa suddenly being able to empathize with you as a means to help persuade you to order more products.

• An always-on, empathetic system can help a student stay on track to graduate or manipulate them into behaviors that erode their autonomy and free will.

• Any foreign government could deploy such a system against a neighboring population and use empathy as a weapon to sow discontent within the opposing population.

Empathy Should Remain a Human Trait

The takeaway with any generative AI system tuned on empathy is that the influence and reach of its output can scale massively very quickly. You may be shaking your head in disbelief, and I can hardly blame you. We're deep into science fiction land, but here we are.

Empathy is a human trait, one I'm not looking forward to offloading. It's through empathy that we form relationships, share deeper emotional connections, and create communities with one another. Machine generated empathy turns human emotions into a fast food commodity—get your feelings made your way right away. Many will say so what, why does it matter if a machine can mimic emotional intelligence? An always on system that could empathize on demand could help society, could get kids back on track on school, even help curb increasing absenteeism.

There’s great risk here. When we engage in an empathetic connection, we are not merely exchanging information but deeply resonating with another's emotional reality. To attempt to replicate this experience through artificial means quickly diminishes its value. Our empathy is a core part of our sense of autonomy and free will. When we empathize, we make the conscious choice to be present, to listen, and to offer our full emotional selves. To have this trait pantomimed by algorithms that simulate empathetic responses is to surrender a fundamental aspect of our humanity.

We can endlessly rationalize away the reasons why machines possessing such traits can be helpful, but where is the line that developers and users of such systems refuse to cross in this race to make machines more like us?