
- An AI voice agent is software that holds a spoken phone call: it listens, decides, speaks and acts, using four parts that hand over in turn.
- It differs from an IVR (menus) and a chatbot (text): it takes free speech on a real phone line, and a pause or a talk-over is part of the design.
- It handles calls with clear answers. A person still takes judgment, distress, disputes and anything the AI cannot check.
What an AI voice agent is
An AI voice agent is software that holds a spoken conversation on a phone call. It listens to the caller, works out what they want, speaks a reply, and takes actions such as booking a slot or updating a record. It can answer inbound calls or place outbound ones. It is built from four parts that hand over in turn.
Disclosure: VoiceGenie publishes this guide and is one of the vendors in this market. The explanation holds whichever product you look at, and the last section says when ours is the wrong choice.
Think of a front desk that never goes home. The caller says, "Can I move my 4 o'clock to Thursday?" The agent checks the calendar, offers two times, books one, and sends a confirmation text through your SMS provider. If the caller instead says "my father died and I need to cancel his appointments," the agent should stop and bring in a person. Most of what follows is about telling those two calls apart.
The four parts of a call, and how they hand over
Every spoken turn goes through the same loop. Four parts do the work.
| Part | What it does | In one line |
|---|---|---|
| Telephony | Connects the call and carries audio both ways | The phone line |
| Speech recognition (speech to text) | Turns the caller's voice into written words | The ears |
| Language model | Reads the words, the instructions and the knowledge base, then chooses a reply or an action | The brain |
| Voice (text to speech) | Turns the reply into spoken audio | The mouth |
Two details matter to a buyer. First, the parts overlap. Good systems do not wait for the caller to finish before they start work: recognition runs while the caller speaks, and the voice starts as soon as the first words of the reply exist. That overlap is how the whole loop fits into a gap a human would accept.
Second, telephony is yours to choose. On VoiceGenie you bring your own carrier and numbers, such as Twilio, Plivo, Telnyx, Vonage, or any SIP trunk or PBX. We do not sell the phone line. That also means call quality depends partly on the carrier you pick.
Where the language model fits
The language model is what separates this from older bots. It does not match keywords against a script. It reads the whole conversation so far and the instructions you wrote, then produces the next reply. That is why a caller can say "Thursday afternoon, but not too late" and be understood. It is also why it can be wrong: a model can produce a fluent answer that is not true, so the agent should answer from a knowledge base you control and read back names, dates and amounts for the caller to confirm.
Latency and interruptions: why the pause matters
Latency is the silence between the caller finishing a sentence and the agent starting to reply. People are sensitive to it. In a study of 10 languages, speakers in all of them minimized silence between turns and avoided talking over each other [2]. A long gap reads as "is anyone there?" and callers start repeating themselves or hang up.
The telephone industry has a yardstick for delay. The ITU says that when one-way delay stays under 150 ms, most applications, speech included, feel essentially transparent, and it recommends not exceeding 400 ms for general network planning [1]. Those figures are for the line alone. A voice agent adds recognition, a model and speech on top, so it spends more of the budget than a plain phone call does.
We have not found a sourced industry figure for the total reply time of voice agents, so we do not quote one. The reliable test is to call it yourself, on a mobile phone, at the busiest hour, and listen.
Interruptions, or barge-in
People interrupt. They say "yes, yes, I know" before the agent finishes. A good agent hears speech while it is talking, stops at once, and listens. The harder cases are the cough, a barking dog, a TV in the background and the caller who says "uh-huh" mid-sentence, which is not a request to stop. When you test an agent, interrupt it on purpose, talk over it with noise behind you, and pause for three seconds to see whether it jumps in too early.
How it differs from an IVR and from a chatbot
| IVR menu | Chatbot | AI voice agent | |
|---|---|---|---|
| Channel | Phone | Text on web or app | Phone |
| Input | Keypad or set phrases | Typed text | Ordinary speech |
| Follows | A fixed tree | A flow or a model | Instructions and a knowledge base |
| Handles a change of mind | Poorly | Reasonably | Reasonably |
| Takes actions | Limited | Often | Yes, through integrations |
| Main risk | Callers give up in the menu | Answers that loop | A confident wrong answer, or a slow reply |
An IVR is a map drawn in advance, and a caller whose problem is not on the map presses 0. Gartner reported in 2019 that only 9% of customers said they fully resolved an issue in self-service [4], which is the usual complaint about menus. A chatbot can understand free text but sits on a website, and many callers will not open one. In a 2026 Gartner survey of 3,566 customers, only 27% said they would be willing to try a chatbot again after a bad experience [6]. For the longer comparisons, see our guides on conversational voice AI versus IVR and on AI voice agents versus chatbots.
What it can do, what it cannot, and what a person still takes
The useful sorting question is: what does a wrong answer cost? If the cost is a repeat call, the agent can finish the call. If it is money, safety or a regulator's deadline, the agent collects facts and a person decides.
| Call | Who owns it | Why |
|---|---|---|
| Opening hours, directions, "is my order shipped?" | AI | One right answer, read from your data |
| Booking, moving or confirming an appointment | AI | Calendar lookup, then a confirmation |
| Payment or appointment reminders | AI | One message, one action |
| New lead from a form at 9:12 pm | AI | Speed matters, questions are fixed |
| Price negotiation, custom quotes | Person | Judgment and authority |
| Complaints, refunds that need discretion | Person, AI gathers facts | The caller needs to be heard |
| Distress, bereavement, emergencies | Person, straight away | Empathy, and in an emergency, safety |
| Anything the agent cannot verify | Person | Do not guess |
What it cannot do: it cannot be trusted to invent policy, it does not know what is missing from your knowledge base, and it can mishear names, accents and numbers over a poor line. It is not a lawyer, a doctor or an adviser.
When the call goes to a person on VoiceGenie, the person sees a summary, the transcript, the caller's mood and what the AI already did, so the caller does not repeat themselves. Here is a sample of an AI agent making a reminder call.
Hello Daniel, I'm calling from City Care Clinic. You have an appointment with Dr Mehta tomorrow at eleven in the morning. Will you be able to make it, or would you like me to move it to another time?
In English: Hello Daniel, I'm calling from City Care Clinic. You have an appointment with Dr Mehta tomorrow at eleven in the morning. Will you be able to make it, or would you like me to move it to another time?
Listen for three things: whether it names who is calling, whether it asks one question at a time, and how it handles a caller who changes the subject.
Calls are not only about talk. ContactBabel reports that average speed to answer in US contact centers rose from 31 seconds in 2012 to 99 seconds in 2024, and that 8.9% of calls were abandoned in 2024 [5]. That gap, calls that wait or are dropped, is the capacity problem an agent can fill. Your team then takes the calls that need judgment.
| Call type | Who takes it | Why |
|---|---|---|
| Status: order, claim, ticket, booking | AI handles it | A lookup, not a judgment. Orders resolve first time 72% and general inquiries 74% (SQM), so the ceiling is high. |
| Hours, policy, coverage and how-to | AI handles it | Answers live in the knowledge base. Scoring every call finds a wrong or stale answer in days, not at the next audit. |
| Callback booking when the queue is long | AI handles it | Turns an abandon into a scheduled contact without adding a seat. Skip it if your callbacks already miss their window. |
| Payments and balance calls | AI, then a person | Card digits spoken to the AI or an agent can put recordings in PCI scope. Keep them out of the recording path. |
| Identity checks and account changes | AI, then a person | Manual checks cost time on every call (ContactBabel puts US spend at $12.3bn a year). A failed check goes to a person. |
| Billing questions and credits | AI, then a person | Explaining is a lookup. Deciding a credit needs authority and tone. Billing resolves first time 71% (SQM). |
| Technical troubleshooting | AI, then a person | Technical calls resolve first time 63% (SQM). Never make the caller repeat the steps. |
| Claims, disputes and complaints | Straight to a person | Claims resolve first time 59% and complaints 47% (SQM). Judgment, authority and tone decide the outcome. |
| Cancel-or-keep, distress, safety, repeat callers | Straight to a person | Trust is spent or the stakes are high. One owner with authority beats another round of questions. |
A glossary of the terms you will meet
| Term | Plain meaning | Why a buyer cares |
|---|---|---|
| Speech to text (STT, ASR) | Software that turns spoken audio into written words | Accuracy on your callers' accents and your product names |
| Text to speech (TTS) | Software that speaks written text aloud | How human the voice sounds, and in which languages |
| Large language model (LLM) | The model that writes the next reply | Drives understanding, and also wrong answers |
| Natural language understanding (NLU) | Working out the meaning and intent behind the words | Why "move it to Thursday" works without keywords |
| Intent | What the caller wants to do | Used to route and to start actions |
| Knowledge base | The facts the agent may answer from | Keeps answers to what you approved |
| Latency | The silence before the agent answers | Long gaps make callers repeat or hang up |
| Barge-in | The caller speaking over the agent | The agent should stop and listen |
| Endpointing | Deciding the caller has finished speaking | Too early cuts people off, too late feels slow |
| IVR | Press-a-key phone menu | What most callers compare an agent to |
| SIP trunk | Internet-based phone line from a carrier | Where your calls come from |
| Hand-off | Passing the call to a person with context | Whether callers repeat themselves |
| Webhook | A message sent to your systems after a call | How records reach your CRM |
| Hallucination | A fluent answer that is not true | Why you test and read transcripts |
| Containment | Share of calls that never reach a person | A caller who gives up also counts, so do not use it alone |
Two older terms you may see: a voice cloning system copies a real person's voice, and speaker diarization labels who spoke when. The FCC ruled in 2024 that AI-generated human-sounding voices, including cloned ones, are artificial voices under the US TCPA [3], so consent rules apply to calls that use them.
How to try one, and when it is the wrong choice
Start with one call type that has a clear answer, such as reminders, confirmations or after-hours questions. Run it on a small list or on overflow, read every transcript for a week, and widen only when the wrong answers are near zero. For a build walk-through, read our beginner's guide to building an AI voice bot. Cost on VoiceGenie is $0.14 per minute for AI, billed per second, plus your carrier. Human seats are per person per month, such as $29 for the desk. See pricing and the AI receptionist use case.
Turn the AI announcement on for each agent and put the recording notice in its opening line. Neither is automatic.
VoiceGenie is the wrong choice in a few cases:
- You only need a simple press-a-key menu, or a human-only desk.
- You need to choose or supply your own language model. Customers cannot bring their own.
- Your systems cannot be read in real time, so the agent would have to guess.
- Most of your calls are disputes or distress, where a person adds the most.
On security: VoiceGenie's security program is built to the SOC 2 Type 2, ISO 27001 and ISO 42001 standards, and it handles personal data in line with GDPR. Security documentation is shared on request. The rules in this guide are general information gathered from public sources, and they differ by country. Check the regulator's own site for the current rule before you launch.
Common questions
What is an AI voice agent in simple terms?
An AI voice agent is software that answers or places phone calls and talks like a person. It turns speech into text, decides what to say or do, and speaks the reply, usually while updating a calendar or CRM. It handles routine calls end to end and passes the rest to a human with the context.
How does an AI voice agent work?
Four parts work in turn. The phone network carries the audio, speech recognition turns it into text, a language model picks the reply or action, and a synthetic voice speaks it. That loop repeats for every turn of the conversation, and the agent stops speaking if the caller talks over it.
What is the difference between an AI voice agent and an IVR?
An IVR plays a fixed menu and asks you to press keys or say set phrases. An AI voice agent accepts ordinary speech, understands what you mean, answers from a knowledge base and can take actions. The IVR follows a tree someone drew in advance. The agent follows instructions and goes where the conversation goes.
Does an AI voice agent have to say it is an AI?
The rule depends on the country and the call. In the US, the FCC treats AI-generated voices as artificial voices under the TCPA, so consent and identification rules apply. On VoiceGenie, announcing the AI is a setting you switch on per agent, and we advise turning it on. Check current rules with the official source.
Can an AI voice agent replace a human agent?
No, and that is the wrong goal. It answers every call and handles the ones with a clear answer, such as hours, bookings and reminders. A person takes judgment calls, upset callers and disputes. The aim is capacity: your staff spend their time on the calls that need them.
Sources
- ITU-T, Recommendation G.114, One-way transmission time, 2003
- Stivers et al., Universals and cultural variation in turn-taking in conversation, PNAS, 2009 (summary)
- FCC, Declaratory Ruling FCC 24-17 on AI-generated voices under the TCPA, 2024
- Gartner, Only 9% of customers report solving their issue in self-service, 2019
- ContactBabel, US Contact Center Decision-Makers' Guide, 17th edition, 2025
- Gartner, Survey Finds Only 27% of Customers Would Try a Chatbot Again After a Negative Experience, press release 2 Sep 2026
About this post. The rules, prices and figures here were gathered from public sources and were correct to the best of our knowledge on the date shown. They change, and they differ by country and state. This is general information, not legal or financial advice. Before you act, check the latest position with the regulator or company concerned, using the links above. If you spot an error, write to support@voicegenie.ai.
Founder and CEO of VoiceGenie. Writes about how AI agents and human teams share the phone work, and what it takes to run them on real calls.
More posts by Anurag Jain

