For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Platform docsVideosCommunitySign up
CapabilitiesGetting startedVoice AI OrchestrationVoxEngine PlatformAPI ReferenceFAQ
CapabilitiesGetting startedVoice AI OrchestrationVoxEngine PlatformAPI ReferenceFAQ
      • Overview
        • Cartesia
        • Deepgram
        • ElevenLabs
        • Gemini
        • Grok
        • Inworld
        • OpenAI
        • Pipecat
        • Ultravox
        • Yandex
  • Management API
    • Reference
    • Authorization
    • Errors
  • Web SDK
    • Overview
  • Android SDK
    • Overview
  • Android SDK v3
    • Overview
  • iOS SDK
    • Overview
  • React Native SDK
    • Overview
  • Flutter SDK
    • Overview
LogoLogo
Platform docsVideosCommunitySign up
On this page
  • Related guides
  • Contents
  • Usage
  • Factory functions
  • createRealtimeAPIClient
  • RealtimeAPIClient
  • Methods
  • addEventListener
  • clearMediaBuffer
  • close
  • conversationItemCreate
  • conversationItemDelete
  • conversationItemRetrieve
  • conversationItemTruncate
  • id
  • removeEventListener
  • responseCancel
  • responseCreate
  • sendMediaTo
  • sessionUpdate
  • stopMediaTo
  • webSocketId
  • Events
  • WebSocketMediaStarted
  • WebSocketMediaEnded
  • RealtimeAPIEvents
VoxEngineVoice AI

Yandex

Realtime API client for Yandex AI Studio speech scenarios.
||View as Markdown|
Was this page helpful?
Edit this page
Previous

Ultravox

Next

MCP

Built with

Yandex provides a VoxEngine client for connecting a call or media unit to the Yandex Realtime API over WebSocket.

Use Yandex.createRealtimeAPIClient(...) to create a RealtimeAPIClient for the current scenario.

Related guides

Yandex connector overview

Learn how Yandex Realtime fits into a VoxEngine call flow.

Contents

  • Usage: required module import and basic flow.
  • Factory functions: create the Yandex Realtime API client.
  • Methods: session, items, and response control methods.
  • Events: WebSocket media bridge events.
  • RealtimeAPIEvents: Yandex Realtime API event names and payload fields.

Usage

Add the module before using the namespace:

1require(Modules.Yandex);

Create the client, bridge media, and listen for Yandex Realtime API events.

Factory functions

createRealtimeAPIClient

Creates a Yandex.RealtimeAPIClient instance.

1createRealtimeAPIClient(parameters: RealtimeAPIClientParameters): Promise<Yandex.RealtimeAPIClient>

Parameters

ParameterTypeReq.Description
parametersRealtimeAPIClientParameters✓

Returns

TypeDescription
Promise<Yandex.RealtimeAPIClient>Resolves to the Yandex.RealtimeAPIClient instance.

RealtimeAPIClient

Methods

addEventListener

Adds a handler for the specified Yandex.RealtimeAPIEvents or Yandex.Events event. Use only functions as handlers; anything except a function leads to the error and scenario termination when a handler is called.

1addEventListener(event: Yandex.Events | Yandex.RealtimeAPIEvents | string, callback: (event: object) => any): void

Parameters

ParameterTypeReq.Description
eventYandex.Events | Yandex.RealtimeAPIEvents | string✓Event constant or event name to subscribe to.
callback(event: object) => any✓Function called when the event is emitted.

Returns

TypeDescription
voidDoes not return a value.

clearMediaBuffer

Clears the Yandex WebSocket media buffer.

1clearMediaBuffer(parameters?: ClearMediaBufferParameters): void

Parameters

ParameterTypeReq.Description
parametersClearMediaBufferParameters✗

Returns

TypeDescription
voidDoes not return a value.

close

Closes the Yandex connection (over WebSocket) or connection attempt.

1close(): void

Parameters

This method does not accept parameters.

Returns

TypeDescription
voidDoes not return a value.

conversationItemCreate

Add a new Item to the Conversation’s context.

1conversationItemCreate(parameters: Object): void

Parameters

ParameterTypeReq.Description
parametersObject✓

Returns

TypeDescription
voidDoes not return a value.

conversationItemDelete

Send this event when you want to remove any item from the conversation history.

1conversationItemDelete(parameters: Object): void

Parameters

ParameterTypeReq.Description
parametersObject✓

Returns

TypeDescription
voidDoes not return a value.

conversationItemRetrieve

Send this event when you want to retrieve the server’s representation of a specific item in the conversation history.

1conversationItemRetrieve(parameters: Object): void

Parameters

ParameterTypeReq.Description
parametersObject✓

Returns

TypeDescription
voidDoes not return a value.

conversationItemTruncate

Send this event to truncate a previous assistant message’s audio.

1conversationItemTruncate(parameters: Object): void

Parameters

ParameterTypeReq.Description
parametersObject✓

Returns

TypeDescription
voidDoes not return a value.

id

Returns the RealtimeAPIClient id.

1id(): string

Parameters

This method does not accept parameters.

Returns

TypeDescription
stringThe requested string value.

removeEventListener

Removes a handler for the specified Yandex.RealtimeAPIEvents or Yandex.Events event.

1removeEventListener(event: Yandex.Events | Yandex.RealtimeAPIEvents | string, callback?: (event: object) => any): void

Parameters

ParameterTypeReq.Description
eventYandex.Events | Yandex.RealtimeAPIEvents | string✓Event constant or event name to subscribe to.
callback(event: object) => any✗Function called when the event is emitted.

Returns

TypeDescription
voidDoes not return a value.

responseCancel

Send this event to cancel an in-progress response.

1responseCancel(parameters: Object): void

Parameters

ParameterTypeReq.Description
parametersObject✓

Returns

TypeDescription
voidDoes not return a value.

responseCreate

This event instructs the server to create a Response, which means triggering model inference.

1responseCreate(parameters: Object): void

Parameters

ParameterTypeReq.Description
parametersObject✓

Returns

TypeDescription
voidDoes not return a value.

sendMediaTo

Starts sending media from the Yandex (via WebSocket) to the media unit. Yandex works in real time.

1sendMediaTo(mediaUnit: VoxMediaUnit, parameters?: SendMediaParameters): void

Parameters

ParameterTypeReq.Description
mediaUnitVoxMediaUnit✓
parametersSendMediaParameters✗

Returns

TypeDescription
voidDoes not return a value.

sessionUpdate

Send this event to update the session’s configuration.

1sessionUpdate(parameters: Object): void

Parameters

ParameterTypeReq.Description
parametersObject✓

Returns

TypeDescription
voidDoes not return a value.

stopMediaTo

Stops sending media from the Yandex (via WebSocket) to the media unit.

1stopMediaTo(mediaUnit: VoxMediaUnit): void

Parameters

ParameterTypeReq.Description
mediaUnitVoxMediaUnit✓

Returns

TypeDescription
voidDoes not return a value.

webSocketId

Returns the Yandex WebSocket id.

1webSocketId(): string

Parameters

This method does not accept parameters.

Returns

TypeDescription
stringThe requested string value.

Events

These events describe audio received through the Yandex WebSocket media bridge.

WebSocketMediaStarted

Triggered when the audio stream sent by a third party through an Yandex WebSocket is started playing.

Event constant: Events.WebSocketMediaStarted

Payload

FieldTypeReq.Description
clientRealtimeAPIClient✓The Yandex.RealtimeAPIClient instance.
tagstring✗Special tag to name audio streams sent over one WebSocket connection. With it, one can send 2 audios to 2 different media units at the same time.
encodingstring✗Audio encoding formats.
customParameters{ [key: string]: string }✗Custom parameters.

WebSocketMediaEnded

Triggers after the end of the audio stream sent by a third party through an Yandex WebSocket (1 second of silence).

Event constant: Events.WebSocketMediaEnded

Payload

FieldTypeReq.Description
clientRealtimeAPIClient✓The Yandex.RealtimeAPIClient instance.
tagstring✗Special tag to name audio streams sent over one WebSocket connection. With it, one can send 2 audios to 2 different media units at the same time.
mediaInfoWebSocketMediaInfo✗Information about the audio stream that can be obtained after the stream stops or pauses (1 second of silence).

RealtimeAPIEvents

These events mirror server messages from the Yandex Realtime API. The data field contains the provider event payload.

Unknown

The unknown event.

Event constant: RealtimeAPIEvents.Unknown

Payload

HTTPResponse

The HTTP response event.

Event constant: RealtimeAPIEvents.HTTPResponse

Payload

WebSocketError

The WebSocket error response event.

Event constant: RealtimeAPIEvents.WebSocketError

Payload

ConnectorInformation

Contains information about connector.

Event constant: RealtimeAPIEvents.ConnectorInformation

Payload

Error

Returned when an error occurs, which could be a client problem or a server problem.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerError

Event constant: RealtimeAPIEvents.Error

Payload

SessionCreated

Returned when a Session is created.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerSessionCreated

Event constant: RealtimeAPIEvents.SessionCreated

Payload

SessionUpdated

Returned when a session is updated with a session.update event, unless there is an error.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerSessionUpdated

Event constant: RealtimeAPIEvents.SessionUpdated

Payload

ConversationItemCreated

Returned when a new Item is created in the Conversation.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerConversationItemCreated

Event constant: RealtimeAPIEvents.ConversationItemCreated

Payload

ConversationItemRetrieved

Returned when a conversation item is retrieved with conversation.item.retrieve.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerConversationItemRetrieved

Event constant: RealtimeAPIEvents.ConversationItemRetrieved

Payload

ConversationItemInputAudioTranscriptionCompleted

This event is the output of audio transcription for user audio written to the user audio buffer.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerConversationItemInputAudioTranscriptionCompleted

Event constant: RealtimeAPIEvents.ConversationItemInputAudioTranscriptionCompleted

Payload

ConversationItemInputAudioTranscriptionDelta

Returned when the text value of an input audio transcription content part is updated with incremental transcription results.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerConversationItemInputAudioTranscriptionDelta

Event constant: RealtimeAPIEvents.ConversationItemInputAudioTranscriptionDelta

Payload

ConversationItemInputAudioTranscriptionSegment

Returned when an input audio transcription segment is identified for an item.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerConversationItemInputAudioTranscriptionSegment

Event constant: RealtimeAPIEvents.ConversationItemInputAudioTranscriptionSegment

Payload

ConversationItemInputAudioTranscriptionFailed

Returned when input audio transcription is configured, and a transcription request for a user message failed.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerConversationItemInputAudioTranscriptionFailed

Event constant: RealtimeAPIEvents.ConversationItemInputAudioTranscriptionFailed

Payload

ConversationItemTruncated

Returned when an earlier assistant audio message item is truncated by the client with a conversation.item.truncate event.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerConversationItemTruncated

Event constant: RealtimeAPIEvents.ConversationItemTruncated

Payload

ConversationItemDeleted

Returned when an item in the conversation is deleted by the client with a conversation.item.delete event.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerConversationItemDeleted

Event constant: RealtimeAPIEvents.ConversationItemDeleted

Payload

InputAudioBufferCommitted

Returned when an input audio buffer is committed.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerInputAudioBufferCommitted

Event constant: RealtimeAPIEvents.InputAudioBufferCommitted

Payload

InputAudioBufferCleared

Returned when the input audio buffer is cleared by the client with an input_audio_buffer.clear event.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerInputAudioBufferCleared

Event constant: RealtimeAPIEvents.InputAudioBufferCleared

Payload

InputAudioBufferDTMFEventReceived

Returned when a DTMF event is received (telephony integrations). https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerInputAudioBufferDtmfEventReceived

Event constant: RealtimeAPIEvents.InputAudioBufferDTMFEventReceived

Payload

InputAudioBufferSpeechStarted

Sent by the server when in server_vad mode to indicate that speech has been detected in the audio buffer.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerInputAudioBufferSpeechStarted

Event constant: RealtimeAPIEvents.InputAudioBufferSpeechStarted

Payload

InputAudioBufferSpeechStopped

Returned in server_vad mode when the server detects the end of speech in the audio buffer.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerInputAudioBufferSpeechStopped

Event constant: RealtimeAPIEvents.InputAudioBufferSpeechStopped

Payload

InputAudioBufferTimeoutTriggered

Returned when the Server VAD timeout is triggered for the input audio buffer.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerInputAudioBufferTimeoutTriggered

Event constant: RealtimeAPIEvents.InputAudioBufferTimeoutTriggered

Payload

OutputAudioBufferCleared

Returned when the output audio buffer is cleared.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerOutputAudioBufferCleared

Event constant: RealtimeAPIEvents.OutputAudioBufferCleared

Payload

OutputAudioBufferStarted

Returned when the output audio buffer starts playing/streaming.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerOutputAudioBufferStarted

Event constant: RealtimeAPIEvents.OutputAudioBufferStarted

Payload

OutputAudioBufferStopped

Returned when the output audio buffer stops.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerOutputAudioBufferStopped

Event constant: RealtimeAPIEvents.OutputAudioBufferStopped

Payload

ResponseCreated

Returned when a new Response is created.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseCreated

Event constant: RealtimeAPIEvents.ResponseCreated

Payload

ResponseDone

Returned when a Response is done streaming.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseDone

Event constant: RealtimeAPIEvents.ResponseDone

Payload

ResponseOutputItemAdded

Returned when a new Item is created during Response generation.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseOutputItemAdded

Event constant: RealtimeAPIEvents.ResponseOutputItemAdded

Payload

ResponseOutputItemDone

Returned when an Item is done streaming.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseOutputItemDone

Event constant: RealtimeAPIEvents.ResponseOutputItemDone

Payload

ResponseContentPartAdded

Returned when a new content part is added to an assistant message item during response generation.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseContentPartAdded

Event constant: RealtimeAPIEvents.ResponseContentPartAdded

Payload

ResponseContentPartDone

Returned when a content part is done streaming in an assistant message item.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseContentPartDone

Event constant: RealtimeAPIEvents.ResponseContentPartDone

Payload

ResponseOutputTextDelta

Returned when the text value of an “output_text” content part is updated.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseOutputTextDelta

Event constant: RealtimeAPIEvents.ResponseOutputTextDelta

Payload

ResponseOutputTextDone

Returned when the text value of an “output_text” content part is done streaming.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseOutputTextDone

Event constant: RealtimeAPIEvents.ResponseOutputTextDone

Payload

ResponseOutputAudioTranscriptDelta

Returned when the model-generated transcription of audio output is updated.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseOutputAudioTranscriptDelta

Event constant: RealtimeAPIEvents.ResponseOutputAudioTranscriptDelta

Payload

ResponseOutputAudioTranscriptDone

Returned when the model-generated transcription of audio output is done streaming.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseOutputAudioTranscriptDone

Event constant: RealtimeAPIEvents.ResponseOutputAudioTranscriptDone

Payload

ResponseOutputAudioDone

Returned when the model-generated audio is done.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseMcpCallArgumentsDone

Event constant: RealtimeAPIEvents.ResponseOutputAudioDone

Payload

ResponseFunctionCallArgumentsDelta

Returned when the model-generated function call arguments are updated.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseFunctionCallArgumentsDelta

Event constant: RealtimeAPIEvents.ResponseFunctionCallArgumentsDelta

Payload

ResponseFunctionCallArgumentsDone

Returned when the model-generated function call arguments are done streaming.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseFunctionCallArgumentsDone

Event constant: RealtimeAPIEvents.ResponseFunctionCallArgumentsDone

Payload

ResponseMCPCallArgumentsDelta

Returned when MCP tool call arguments are updated during response generation.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseMcpCallArgumentsDelta

Event constant: RealtimeAPIEvents.ResponseMCPCallArgumentsDelta

Payload

ResponseMCPCallArgumentsDone

Returned when MCP tool call arguments are finalized during response generation.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseMcpCallArgumentsDone

Event constant: RealtimeAPIEvents.ResponseMCPCallArgumentsDone

Payload

ResponseMCPCallInProgress

Returned when an MCP tool call has started and is in progress.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseMcpCallInProgress

Event constant: RealtimeAPIEvents.ResponseMCPCallInProgress

Payload

ResponseMCPCallCompleted

Returned when an MCP tool call has completed successfully.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseMcpCallCompleted

Event constant: RealtimeAPIEvents.ResponseMCPCallCompleted

Payload

ResponseMCPCallFailed

Returned when an MCP tool call has failed.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerResponseMcpCallFailed

Event constant: RealtimeAPIEvents.ResponseMCPCallFailed

Payload

MCPListToolsInProgress

Returned when listing MCP tools is in progress for an item.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerMcpListToolsInProgress

Event constant: RealtimeAPIEvents.MCPListToolsInProgress

Payload

MCPListToolsCompleted

Returned when listing MCP tools has completed for an item.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerMcpListToolsCompleted

Event constant: RealtimeAPIEvents.MCPListToolsCompleted

Payload

MCPListToolsFailed

Returned when listing MCP tools has failed for an item.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerMcpListToolsFailed

Event constant: RealtimeAPIEvents.MCPListToolsFailed

Payload

RateLimitsUpdated

Emitted at the beginning of a Response to indicate the updated rate limits.https://yandex.cloud/ru/docs/ai-studio/serverEvents/realtimeServerRateLimitsUpdated

Event constant: RealtimeAPIEvents.RateLimitsUpdated

Payload