What Actually Goes Into a Good Voice Agent

From the outside a voice agent looks simple. Someone calls, a friendly voice picks up, the caller gets help. Underneath that are dozens of settings, and most of what separates a good agent from one that embarrasses your business lives in details your callers never see.

I scaled my agency to $11K in a month built entirely on Retell AI, so this is what I've learned delivering for real clients. In the video I build a demo agent for a surf shop, Kai, who gives the surf report, books lessons, and helps with gear. Here's what goes into one.

It starts with the prompt

99% of the issues that come up with a voice agent come down to the prompt. It's by far the most important part. We usually build what's called a single prompt agent, and since the models keep getting smarter, it's one of the best ways to build an all around production ready agent unless you have a very specific use case.

Every prompt we write has a role, rules, steps, and skills. Agents perform best when you give them a role. And because these models are pattern matchers, we show rather than tell: we give the agent examples of exactly the output we want.

Even when a tool helps draft the prompt, we read it line by line, as tedious as that is. Otherwise something in there causes mistakes, you test and test and test, and you never knew what was in the prompt to begin with. We also keep the model's temperature at the lowest setting so the agent stays predictable, and turn on structured output when it has to call functions with precision.

Tools that let it do real work

A good agent doesn't just talk. On Retell it can end the call naturally when the caller says goodbye, do warm and cold transfers to your team (a must for production agents), hand off to another agent for a different department on the same phone system, and check availability and book appointments through a calendar integration. Custom functions connect it to your CRM or your own systems, and it can send texts.

One small detail callers notice: when the agent runs one of these tools, we have it say something like "let me check that for you." Without that setting it can literally sit silent while it's checking, and it may even get the answer back without telling the caller.

Then there's the knowledge base, your FAQs, documents, or website, which can refresh every 24 hours. We tell the agent that if the info isn't there, say it doesn't know. That prevents hallucination, so it doesn't make something up. Knowledge base formatting is honestly a skill of its own, and a well formatted one performs better.

The settings that make it sound human

This part is so important and people overlook it. A soft coffee shop or call center background sound, at around 0.80 to 0.85 volume, sounds the most natural. Interruption sensitivity usually sits in the 70s to 80s. Set it too low and a caller can be shouting while the agent keeps talking. Set it to 100 and one small word cuts it off. Backchanneling adds little "yeah"s while the caller talks. We always have background noise and background speech removal on.

There are safety settings too. A reminder like "are you still there?" about every 15 seconds, two times max. Ending the call on silence so the agent isn't sitting there burning minutes. A max call duration about 30 to 40% above the expected length, so if something goes wrong it doesn't run for two hours. And a fallback voice in case the main one fails mid call.

Testing, and getting the data out

These agents are probabilistic, not deterministic. So we regenerate an answer 10 times to see how often the agent gives the one we want, and keep fixing until it's around 9 or 10 out of 10. That way you can be sure it's consistent. We also run simulated callers, like an angry customer, against success criteria, and we can run 50 of those at the same time. Testing is probably the most important skill in this whole business.

After the call, the agent can pull out the details you care about, like the caller's name and reason for calling, and send them anywhere: an email, a Google Sheet, your CRM. And if you own the Retell account, you simply add us to your workspace.

Most of this is invisible to your callers, which is the point. If you're comparing builders, ask how they test. That answer tells you a lot.