Pipeline Options

For the complete documentation index, see llms.txt.

VoxEngine supports a wide variety of Voice AI architectures and components. The right pipeline depends on your priorities — low latency, interactivity needs, voice choice, and low-level control are some factors to consider.

High-level comparison

Voximplant Connectors & Pipeline Support

Voximplant always manages the telephony side of the call: routing, answering, bridging media, call control, and any orchestration logic you add in VoxEngine. Our Voice AI API clients - also referred to as connectors - provide a direct connection from VoxEngine to the provider’s environment with a low-latency WebSocket connection. Voximplant’s various connector options differ in how they handle transcription, reasoning, and speech synthesis internally.

LLMs like OpenAI, Gemini, and Grok provide direct access to the speech-enabled LLM. Speech input and output parameters are managed as part of the connector.

Ultravox provides a speech-native LLM, but integrates TTS within their environment, only exposing a Speech-to-Speech interface to Voximplant. Similarly, ElevenLabs, Cartesia, and Deepgram provide a speech-to-speech interface to Voximplant, but handle LLM, STT, and TTS elements internally. Configuration of these internal elements vary by vendor. We refer to the integrations as “indirect speech-to-speech”.

All vendors provide some level of end-of-turn detection, with some vendors providing more explicit control over turn-taking and barge-in.

The table below summarizes support by vendor from the Voximplant connector perspective.

Speech-to-speech

In this pattern, caller audio is bridged to the provider in realtime, and the provider returns speech audio directly back into the call.

How it works

• Caller audio is streamed from VoxEngine to a realtime voice agent over WebSockets.
• The provider handles speech recognition, reasoning, and speech generation in one session.
• VoxEngine manages telephony, routing, media bridge, and call control around that session.

Tradeoffs

Best for

• You want the fastest possible conversational loop.
• You are happy with the provider’s built-in voices.
• You want the simplest operational model.

Related guides

Direct speech-to-speech connectors

These connectors expose the speech-enabled LLM directly to VoxEngine.

Indirect speech-to-speech connectors

These connectors present a speech-to-speech interface to VoxEngine while managing more of the internal speech and agent stack inside the provider environment.

Hybrid-cascade

Hybrid-cascade - sometimes called half-cascade - keeps the realtime LLM in the loop for user speech input and reasoning, but moves speech output to a separate synthesis provider. This is often the best compromise between latency and voice flexibility.

How it works

• Caller audio is streamed to a realtime LLM.
• The realtime LLM returns text rather than audio (or you transcribe the audio).
• VoxEngine sends the text into a TTS engine and streams the generated speech back to the caller.

Tradeoffs

Best for

• You want a specific voice provider or voice quality that your LLM provider does not offer.
• You want more control over output speech without moving to a full cascade.
• You need a practical middle ground between speed and flexibility.

Related guides

Full cascade

Full cascade separates speech-to-text, reasoning, and text-to-speech into independent stages. This gives you the most control over the pipeline and the most freedom to mix providers.

How it works

• VoxEngine transcribes caller audio with an STT integration.
• Your scenario sends text to an LLM or OpenAI-compatible endpoint.
• The text response is synthesized through a TTS provider and streamed back into the call.
• Voice activity detection, turn detection, and barge-in control are handled explicitly in the orchestration layer.

Tradeoffs

Best for

• You need full control over transcription, reasoning, and speech generation.
• You want to mix providers based on quality, language support, or cost.
• You need explicit turn-taking logic or custom orchestration around the LLM.

Related guides

For turn-taking, interruption, and end-of-turn behavior, see the Speech Flow Control guides.

Choosing a pipeline

If you are new to Voice AI starting from scratch, we recommend the following progression:

1. Start with speech-to-speech to validate the call flow and prompt design quickly.
2. Move to hybrid-cascade if you need a different voice or output speech behavior than your speech-LLM provider offers.
3. Move to full cascade when you need full provider selection, explicit turn-taking control, and/or custom orchestration between stages.

That progression keeps the first version simple while leaving room to add flexibility only when the product requirements justify the extra moving parts.