Inside PolyAI: The Technology Behind AI Conversations

Voice AI is easy to demonstrate and surprisingly difficult to engineer.

A model can speak a sentence with an impressive synthetic voice in seconds. That does not mean it can hold a real conversation. Human conversation is full of incomplete sentences, pauses, interruptions, background noise, changes in intention, and moments where neither person knows exactly when the other has finished speaking.

That creates a different engineering problem from text-based AI.

When someone types a message into a chatbot, the system already knows where the user’s input ends. In a phone conversation, there is no equivalent boundary. The AI has to work out whether a silence means the caller has finished, is thinking, is searching for information, or is about to continue.

PolyAI‘s recent work provides an interesting look inside this problem. Its Dialog-RSN-1 system is designed to process conversational audio directly and make decisions about turn-taking, speech recognition, responses, and tool calls within the same model. The important story is not the product itself. It is what this architecture reveals about the difficult parts of making AI participate in a live conversation.

The first problem happens before the AI answers

Consider a simple exchange:

Caller: “I need to change my appointment for…”

Then the caller pauses.

What should the AI do?

If it assumes the person has finished, it may interrupt with a question.

If it waits too long, the caller may think the system has stopped responding.

Humans solve this almost automatically. We use timing, intonation, context and the structure of the sentence to predict whether another person is finished.

A conventional voice AI system usually approaches the problem as a sequence.

Audio is captured.

A speech-recognition system converts it into text.

The text is sent to a language model.

The model generates a response.

The response is converted back into speech.

This is often called a cascaded architecture.

There is nothing inherently wrong with it. In fact, separating these components gives developers considerable control. But every boundary between components can introduce another decision, another source of latency, and another place where information can be lost.

That becomes particularly important when the original information is audio.

A transcript is not the conversation

Speech recognition is fundamentally a compression step.

A caller’s voice contains much more than the words themselves. There is timing, emphasis, hesitation, background sound, overlapping speech and other acoustic information.

Once that audio becomes text, much of that information is no longer directly available to the language model.

Imagine someone says:

“Yeah… I think that’s the number.”

The words alone do not tell the complete story.

The hesitation could indicate uncertainty. The speaker could be searching for the number. Their tone could indicate frustration. There could be another person talking in the background.

A transcript can represent the linguistic content, but the original audio contains additional evidence.

This is one of the ideas behind PolyAI’s Dialog-RSN-1. Rather than requiring every interaction to be reduced to a transcript before the main model can reason about it, PolyAI describes the system as an audio-native dialog model that directly perceives the caller’s audio.

That architectural decision changes where some of the intelligence sits.

The model is no longer receiving only the result of a speech-recognition system. It can use the audio itself as part of the conversational context.

The surprisingly difficult problem: knowing when to speak

One of the most interesting details in PolyAI’s architecture is that turn-taking is treated as a model-level problem.

A voice agent needs to answer a basic question continuously:

Is the caller still talking?

Dialog-RSN-1 predicts a turn-taking state as part of its output. PolyAI describes three states: EMPTY, ONGOING and COMPLETE.

EMPTY means there is no user speech to respond to.

ONGOING means the caller has started speaking but has not finished.

COMPLETE means the model considers the user’s turn finished and can respond.

This sounds simple until real conversations are introduced.

People pause in the middle of sentences.

They say “um.”

They breathe.

They search for a number.

They become distracted.

They speak while another person is talking.

They interrupt the AI.

A system that treats every silence as the end of a turn will constantly talk over people. A system that waits for absolute certainty will feel slow.

The problem is therefore not simply speech detection.

It is conversation-state prediction.

That distinction is important because it moves the problem from a small piece of audio infrastructure into the reasoning layer of the agent.

Why interruption is a serious engineering problem

Humans interrupt one another constantly.

In voice AI, this is called barge-in.

Suppose an AI says:

“Your appointment is currently scheduled for Tuesday at…”

The caller immediately says:

“No, Thursday.”

A useful system needs to recognize that interruption, stop speaking, understand the new input and continue from the new conversational state.

A simplistic system might finish its sentence anyway.

Another system might stop whenever it hears noise, even if the caller did not actually interrupt.

Both behaviors feel wrong.

PolyAI has discussed barge-in as a major engineering problem because telephony audio is rarely clean. Background television, acoustic echo, other speakers and network noise can all produce signals that resemble speech.

The system therefore has to distinguish meaningful interruption from irrelevant sound.

That requires more than a microphone and a speech-to-text API.

The architecture has to remember what just happened

Interruption creates another problem.

Imagine the AI was halfway through explaining something when the caller interrupted.

After stopping, the AI cannot simply treat the interruption as an isolated new question. It needs to understand what it was saying, what the caller interrupted, and what the new statement changes.

This is where conversational state becomes important.

A useful voice agent has to maintain something resembling a running model of the interaction:

• What the caller originally wanted
• What information has already been collected
• What the AI was about to say
• What action is currently being performed
• Whether the caller has corrected something
• Whether an external system has returned information
• Whether the conversation should continue or be transferred

The difficult part is that this state changes continuously.

A conversation is not a collection of independent prompts.

It is a sequence of decisions where every new piece of information can change what should happen next.

Why latency is not just an LLM problem

Voice AI has another constraint that text AI can largely ignore.

Silence feels different on a phone call.

If a chatbot takes two seconds to generate an answer, the user can watch the interface while waiting.

On a phone call, those two seconds are simply silence.

PolyAI has argued that voice latency has to be considered across the entire system rather than measured only by model inference speed.

That includes:

Turn detection → speech processing → reasoning → tool calls → response generation → speech generation → telephony

A model can be extremely fast and still produce a slow conversation if an external database takes too long to return an answer.

For example, imagine a caller asks:

“Where is my order?”

The AI may understand that request almost immediately.

But if it has to call an order-management API, wait for the response, process the result and then speak, the external system becomes part of the conversational latency.

This is why voice-agent architecture cannot be evaluated by looking at the language model alone.

The model is only one component in the critical path.

Why PolyAI’s audio-native approach changes the critical path

The traditional pipeline contains multiple stages.

The caller speaks.

The system determines that the caller has stopped.

Speech recognition finishes processing the audio.

The language model receives the transcript.

The language model generates a response.

The response is passed to speech synthesis.

The caller hears it.

Each stage introduces its own delay.

Dialog-RSN-1 attempts to reduce some of those boundaries by allowing the model to reason directly over audio and incorporate turn-taking into the same model.

PolyAI says the model can begin preparing its response while the caller is still speaking, with the final decision about whether the turn is complete being made as part of the model’s processing.

This does not magically remove every source of latency.

It changes the architecture of the waiting process.

That distinction is more important than simply saying that the model is “faster.”

There is an interesting compromise in the design

One of the less obvious parts of PolyAI’s approach is that audio-native reasoning does not mean everything has to become speech-to-speech.

Dialog-RSN-1 reasons over audio but produces text as its response representation before the final speech layer.

That may appear like an unnecessary middle step.

Technically, it provides an important control point.

Text can be inspected.

Text can be logged.

Text can be tested.

Text can be checked against policies.

Text can be blocked before it reaches the caller.

For an enterprise system, these properties matter.

A completely opaque speech-to-speech system may sound natural, but an organization still needs to understand what the system decided, which tools it attempted to use, and what response it generated.

Keeping a textual representation in the system can therefore provide an engineering and governance boundary even when the model itself is processing raw audio.

The voice is only one part of the system

Another misconception about voice AI is that the voice is the difficult part.

Speech synthesis has improved enormously.

The harder question is what happens between the caller speaking and the synthetic voice responding.

A production system needs several layers working together.

There is audio processing.

There is turn detection.

There is speech recognition.

There is reasoning.

There is conversational memory.

There are tools and external APIs.

There are permissions.

There are business rules.

There is response generation.

There is speech synthesis.

There is telephony infrastructure.

And there is the logic for deciding when the AI should stop and involve a human.

This is why building a reliable voice agent is closer to building a distributed real-time software system than simply connecting an LLM to a text-to-speech service.

Tool calls turn conversation into action

The moment an AI needs to do something outside the conversation, another layer appears.

Suppose a caller says:

“Move my appointment to Friday.”

The AI needs more than language understanding.

It needs to identify the intended action, determine which appointment belongs to the caller, authenticate the request if required, communicate with the scheduling system, receive the result, and explain that result.

The language model becomes part of a larger control loop.

Understand → decide → call a tool → receive information → update context → respond

That architecture creates another important engineering question:

What is the AI allowed to do?

A model might understand a request perfectly but still not have permission to execute it.

This is why tool calling in an enterprise voice system cannot be treated as a simple API feature. It becomes part of the safety boundary of the entire agent.

The system has to know when not to continue

A capable voice agent also needs an exit strategy.

There are situations where continuing autonomously is worse than transferring the conversation.

The caller may be too difficult to authenticate.

The request may fall outside the agent’s authority.

An external system may be unavailable.

The caller may explicitly ask for a human.

The model may not have enough information to proceed safely.

This makes escalation part of the architecture rather than a failure condition.

A production voice agent is therefore not simply optimized to maximize the number of conversations it completes.

It has to know which conversations it should handle and which ones should leave the automated system.

Training a voice model is different from training a text model

PolyAI says Dialog-RSN-1 was developed through supervised fine-tuning and reinforcement fine-tuning of open-weight multimodal models.

The training data includes millions of conversations derived from real deployments and synthetic conversations, with expert-generated labels.

The interesting part is what the model is being trained to produce.

According to PolyAI, the output format includes information such as turn-taking decisions, reasoning, textual responses, tool calls, citations and speech transcriptions.

In other words, the training target is not simply:

“Generate the best answer.”

It is closer to:

“Understand the state of this conversation and determine what the system should do next.”

That is a fundamentally different optimization problem.

The model is learning behavior across an interaction rather than only predicting the next useful piece of text.

Why context matters more in voice

A text prompt usually provides a clear boundary around the user’s request.

Voice interaction does not.

The meaning of a sentence can depend heavily on what happened thirty seconds earlier.

Consider:

“Yeah, that one.”

There is almost no useful information in those words by themselves.

But if the previous exchange was:

“Do you want the Tuesday appointment or the Thursday appointment?”

then the answer becomes meaningful.

This means voice AI requires strong conversational context.

The model must retain enough information from earlier turns to interpret short, ambiguous responses.

Audio-native processing adds another layer because some of that context can exist outside the transcript itself.

The goal is not merely to remember more text.

It is to preserve the information necessary to understand what the caller is doing in the conversation.

What PolyAI reveals about the next generation of voice AI

The larger significance of PolyAI’s work is not that one company has found a final architecture for voice agents.

The more interesting development is that voice is increasingly being treated as its own AI engineering problem.

Early voice systems were often built as chains of specialized components.

That approach made sense because each problem could be solved independently.

But as models become capable enough to handle multiple parts of the interaction, the boundaries between those components become potential bottlenecks.

Turn-taking can become part of reasoning.

Speech recognition can use conversational context.

Tool calls can become part of the model’s decision process.

Audio can become an input to reasoning rather than merely a source from which text is extracted.

The architecture starts moving from:

hear → transcribe → think → speak

toward something closer to:

listen → understand the conversational state → decide → act → respond

That is a meaningful architectural shift.

The unresolved problem

None of this means voice AI has solved conversation.

Real-world calls remain unpredictable.

Audio quality varies.

People speak differently.

Languages and accents introduce additional complexity.

External APIs fail.

Users change their minds.

Business rules conflict with what the caller wants.

And a model can still misunderstand something even when it has access to the original audio.

The difficult problem is therefore not making an AI that can produce a convincing voice.

It is making a system that can remain reliable while everything around the conversation is imperfect.

That requires models, but it also requires infrastructure, evaluation, permissions, observability and careful system design.

PolyAI’s work is interesting precisely because it exposes this broader problem.

The future of voice AI may depend less on making language models simply speak and more on teaching AI systems how to participate in the structure of human conversation itself.

That means knowing when to listen.

Knowing when to speak.

Knowing when an interruption matters.

Knowing what information is relevant.

Knowing when an action is permitted.

And, just as importantly, knowing when it should stop and let a human take over.

That is the engineering problem hiding underneath the apparently simple experience of talking to an AI.