Making a Voice Agent Sound Human Takes More Than a Good Voice

You can have the most realistic voice on the platform and still sound like ChatGPT with a speaker.

That's the line I keep coming back to. Most voice agents don't have a voice problem. They have a words problem. A realistic voice reading robotic words is still a robot, and your callers can hear it.

To test this, I built a counseling agent and called it myself as someone whose dog had just died. That's about the hardest kind of call there is to get right, and it's exactly where a flat, cheerful agent falls apart.

What Expressive Mode does

Retell's Expressive Mode gives your agent audio tags: sigh, pause, clear throat, excited, gentle. 18 voices support it. You switch it on in the voice settings, and you can override Retell's default instructions for how the tags get used.

Two things to know up front. The tags are a fixed list, so the model won't reliably sneeze or cough no matter what you write. And the default instructions are a starting point. They aren't enough.

The prompt is what makes it land

Two things in the agent's own prompt do the real work.

First, an expressive delivery section with rules for your specific use case. For the counseling agent, I told it to lean on empathy, use pauses, slow down, let a feeling land, and stress the one word that matters most. A dental office or a plumbing company would get different rules.

Second, a full example conversation at the bottom of the prompt with the tags already in it. Agent, caller, agent, caller, the kind of exchange it'll actually run into. Show the model, don't just tell it.

Tag density matters too. Put a tag in every sentence and you get weird swings, excited about one thing and sighing about something sad two seconds later. Use no tags and it's flat. There's a middle, and finding it takes testing.

The settings behind it

Under the agent handbook there are three settings we lean on. The colloquial model is a tiny model trained on thousands of real phone calls that rewrites the agent's words in real time so they sound like something a person would say. It costs about 900 tokens and 50 milliseconds of latency. Then natural filler words, and high empathy.

Then voice speed. I keep mine at 1.06, because competent speakers talk 5 to 10% faster. It varies voice to voice, and you can set it to adjust to the caller.

Here's the distinction people miss. Expressive Mode controls the speech. The colloquial model controls the words. You need both.

The honest limitations

Consistency. The agent always clears its throat when told to, but the intensity varies from one run to the next. Getting a good output once doesn't mean the agent is done, so we test the same lines several times before anything ships.

Voice variance. What works on one voice won't necessarily work on another, and accents add more complexity. Switching voices means testing again.

If you're evaluating a voice agent for your business, listen past the voice itself. Ask whether it sounds like a person choosing words, or a script being read out loud. That's usually the difference.