Voice AI ClientsOpenAI

Example: Function calling

View as Markdown

For the complete documentation index, see llms.txt.

This example answers an inbound Voximplant call, connects it to OpenAI Realtime, and handles function calls in VoxEngine. Select a model version to keep the session configuration and complete scenario synchronized. gpt-realtime-2.1 is the default for new applications.

It includes three tools:

  • get_weather
  • hang_up
  • warm_transfer

Jump to the Full VoxEngine scenarios.

Prerequisites

Session setup

The scenario uses sessionUpdate to define:

  • Realtime model instructions
  • low reasoning effort
  • nested semantic_vad turn detection
  • tool schemas (tools)
  • tool_choice: "auto"

Tool definitions are declared in SESSION_CONFIG.session.tools and sent right after SessionCreated.

GPT Realtime 2.1 session configuration
1const SESSION_CONFIG = {
2  session: {
3    type: "realtime",
4    model: "gpt-realtime-2.1",
5    instructions: SYSTEM_PROMPT,
6    reasoning: { effort: "low" },
7    output_modalities: ["audio"],
8    audio: {
9      input: {
10        turn_detection: { type: "semantic_vad", interrupt_response: true },
11      },
12      output: {
13        voice: "alloy",
14      },
15    },
16    tools: [
17      {
18        type: "function",
19        name: WEATHER_TOOL,
20        description: "Get the weather for a given location",
21        parameters: {
22          type: "object",
23          properties: {
24            location: { type: "string" },
25          },
26          required: ["location"],
27        },
28      },
29      {
30        type: "function",
31        name: HANGUP_TOOL,
32        description: "Hang up the call",
33        parameters: {
34          type: "object",
35          properties: {},
36          required: [],
37        },
38      },
39      {
40        type: "function",
41        name: WARM_TRANSFER_TOOL,
42        description: "Warm transfer the caller to a phone number",
43        parameters: {
44          type: "object",
45          properties: {
46            destination_number: { type: "string" },
47            message: { type: "string" },
48            delay_ms: { type: "integer" },
49          },
50          required: [],
51        },
52      },
53    ],
54    tool_choice: "auto",
55  },
56};
57
58voiceAIClient.sessionUpdate(SESSION_CONFIG);

Connect call audio

After SessionUpdated, the example bridges call audio to OpenAI and prompts the agent to greet the caller. See the full scenarios below for the version-specific greeting.

Connect call audio
1VoxEngine.sendMediaBetween(call, voiceAIClient);

Function calling flow

The example listens for OpenAI.RealtimeAPIEvents.ResponseFunctionCallArgumentsDone, parses arguments, and returns tool output with conversationItemCreate (type: "function_call_output"), then calls responseCreate so the assistant continues.

get_weather

Returns a stub weather payload for the requested location.

hang_up

Sets pendingHangup = true, returns hangup_scheduled, and hangs up after assistant audio completes (ResponseOutputAudioDone or WebSocketMediaEnded).

warm_transfer

Places a PSTN leg, plays a brief intro to the callee, then bridges the original caller to the transfer leg after a delay.

Barge-in

The scenario clears buffered model audio whenever OpenAI detects caller speech:

Barge-in
1voiceAIClient.addEventListener(OpenAI.RealtimeAPIEvents.InputAudioBufferSpeechStarted, () => {
2  voiceAIClient.clearMediaBuffer();
3});

Full VoxEngine scenarios

Notes

  • Tool implementations are demo stubs. Replace them with production APIs and transfer logic.
  • The warm transfer demo uses a default destination if the model does not provide one.
voxeengine-openai-function-calling.js
1/**
2 * Voximplant + OpenAI Realtime API connector demo
3 * Scenario: answer an incoming call and handle OpenAI Realtime 2.1 function calls.
4 */
5
6require(Modules.OpenAI);
7
8const SYSTEM_PROMPT = `
9You are Voxi, a helpful phone assistant.
10Keep responses short and telephony-friendly.
11If the caller asks for weather, call get_weather.
12If the caller wants to end the call, say a brief goodbye and call hang_up.
13If the caller asks for a warm transfer, call warm_transfer with destination_number.
14`;
15
16const WEATHER_TOOL = "get_weather";
17const HANGUP_TOOL = "hang_up";
18const WARM_TRANSFER_TOOL = "warm_transfer";
19
20const DEFAULT_TRANSFER_NUMBER = "+18889277255";
21const DEFAULT_TRANSFER_DELAY_MS = 3000;
22const DEFAULT_TRANSFER_GREETING =
23    "Hi, this is Voxi. I'm warm transferring a caller. Please hold for a brief note, then I'll connect you.";
24
25const SESSION_CONFIG = {
26    session: {
27        type: "realtime",
28        model: "gpt-realtime-2.1",
29        instructions: SYSTEM_PROMPT,
30        reasoning: {effort: "low"},
31        output_modalities: ["audio"],
32        audio: {
33            input: {
34                turn_detection: {type: "semantic_vad", interrupt_response: true},
35            },
36            output: {
37                voice: "alloy",
38            },
39        },
40        tools: [
41            {
42                type: "function",
43                name: WEATHER_TOOL,
44                description: "Get the weather for a given location",
45                parameters: {
46                    type: "object",
47                    properties: {
48                        location: {type: "string"},
49                    },
50                    required: ["location"],
51                },
52            },
53            {
54                type: "function",
55                name: HANGUP_TOOL,
56                description: "Hang up the call",
57                parameters: {
58                    type: "object",
59                    properties: {},
60                    required: [],
61                },
62            },
63            {
64                type: "function",
65                name: WARM_TRANSFER_TOOL,
66                description: "Warm transfer the caller to a phone number",
67                parameters: {
68                    type: "object",
69                    properties: {
70                        destination_number: {type: "string"},
71                        message: {type: "string"},
72                        delay_ms: {type: "integer"},
73                    },
74                    required: [],
75                },
76            },
77        ],
78        tool_choice: "auto",
79    },
80};
81
82VoxEngine.addEventListener(AppEvents.CallAlerting, async ({call}) => {
83    let voiceAIClient;
84    let transferCall;
85    let transferInProgress = false;
86    let pendingHangup = false;
87
88    call.addEventListener(CallEvents.Disconnected, () => VoxEngine.terminate());
89    call.addEventListener(CallEvents.Failed, () => VoxEngine.terminate());
90
91    try {
92        call.answer();
93        // call.record({hd_audio: true, stereo: true}); // Optional: record the call
94
95        voiceAIClient = await OpenAI.createRealtimeAPIClient({
96            apiKey: VoxEngine.getSecretValue("OPENAI_API_KEY"),
97            model: "gpt-realtime-2.1",
98            onWebSocketClose: (event) => {
99                Logger.write("===OpenAI.WebSocket.Close===");
100                if (event) Logger.write(JSON.stringify(event));
101                VoxEngine.terminate();
102            },
103        });
104
105        voiceAIClient.addEventListener(OpenAI.RealtimeAPIEvents.SessionCreated, () => {
106            voiceAIClient.sessionUpdate(SESSION_CONFIG);
107        });
108
109        voiceAIClient.addEventListener(OpenAI.RealtimeAPIEvents.SessionUpdated, () => {
110            VoxEngine.sendMediaBetween(call, voiceAIClient);    // media connected
111            voiceAIClient.conversationItemCreate({              // request a greeting
112                item: {
113                    type: "message",
114                    role: "user",
115                    content: [
116                        {
117                            type: "input_text",
118                            text: "The phone call just connected. "
119                                + 'Say exactly: "Hello! How can I help today?"',
120                        },
121                    ],
122                },
123            });
124            voiceAIClient.responseCreate({});       // ask for the response now
125        });
126
127        voiceAIClient.addEventListener(
128            OpenAI.RealtimeAPIEvents.InputAudioBufferSpeechStarted,
129            () => {
130                Logger.write("===BARGE-IN: OpenAI.InputAudioBufferSpeechStarted===");
131                voiceAIClient.clearMediaBuffer();
132            },
133        );
134
135        // Handle function calls
136        voiceAIClient.addEventListener(
137            OpenAI.RealtimeAPIEvents.ResponseFunctionCallArgumentsDone,
138            async (event) => {
139                const payload = event?.data?.payload || event?.data || {};
140                const toolName = payload.name || payload.tool_name;
141                const toolCallId = payload.call_id || payload.callId;
142                const rawArgs = payload.arguments;
143
144                if (!toolName || !toolCallId) {
145                    Logger.write("===TOOL_CALL_MISSING_FIELDS===");
146                    Logger.write(JSON.stringify(payload));
147                    return;
148                }
149
150                let args = {};
151                if (typeof rawArgs === "string") {
152                    try {
153                        args = JSON.parse(rawArgs);
154                    } catch (error) {
155                        Logger.write("===TOOL_ARGS_PARSE_ERROR===");
156                        Logger.write(rawArgs);
157                        Logger.write(error);
158                    }
159                } else if (rawArgs && typeof rawArgs === "object") {
160                    args = rawArgs;
161                }
162
163                Logger.write("===TOOL_CALL_RECEIVED===");
164                Logger.write(JSON.stringify({toolName, args}));
165
166                if (toolName === WEATHER_TOOL) {
167                    const location = args.location || "Unknown";
168                    const result = {
169                        location,
170                        temperature_f: 72,
171                        condition: "sunny",
172                    };
173                    voiceAIClient.conversationItemCreate({
174                        item: {
175                            type: "function_call_output",
176                            call_id: toolCallId,
177                            output: JSON.stringify(result),
178                        },
179                    });
180                    voiceAIClient.responseCreate({});
181                    return;
182                }
183
184                if (toolName === HANGUP_TOOL) {
185                    pendingHangup = true;
186                    voiceAIClient.conversationItemCreate({
187                        item: {
188                            type: "function_call_output",
189                            call_id: toolCallId,
190                            output: JSON.stringify({status: "hangup_scheduled"}),
191                        },
192                    });
193                    voiceAIClient.responseCreate({});
194                    return;
195                }
196
197                if (toolName === WARM_TRANSFER_TOOL) {
198                    if (transferInProgress) {
199                        voiceAIClient.conversationItemCreate({
200                            item: {
201                                type: "function_call_output",
202                                call_id: toolCallId,
203                                output: JSON.stringify({status: "transfer_already_in_progress"}),
204                            },
205                        });
206                        voiceAIClient.responseCreate({});
207                        return;
208                    }
209
210                    transferInProgress = true;
211
212                    const destination =
213                        args.destination_number ||
214                        args.destination ||
215                        args.phone_number ||
216                        args.number ||
217                        DEFAULT_TRANSFER_NUMBER;
218                    const delayMs = Number.isFinite(args.delay_ms)
219                        ? args.delay_ms
220                        : DEFAULT_TRANSFER_DELAY_MS;
221                    const message = args.message || DEFAULT_TRANSFER_GREETING;
222
223                    try {
224                        transferCall = VoxEngine.callPSTN(destination, call.callerid());
225                        // transferCall = VoxEngine.callUser({username: destination, callerid: call.callerid()});
226                        // transferCall = VoxEngine.callSIP(`sip:${destination}@your-sip-domain`, call.callerid());
227                        // transferCall = VoxEngine.callWhatsappUser({number: destination, callerid: call.callerid()});
228
229                        transferCall.addEventListener(CallEvents.Connected, () => {
230                            Logger.write(`===WARM_TRANSFER_CONNECTED=== ${destination}`);
231                            transferCall.say(message);
232
233                            setTimeout(() => {
234                                try {
235                                    voiceAIClient.clearMediaBuffer();
236                                    call.stopMediaTo(voiceAIClient);
237                                    voiceAIClient.stopMediaTo(call);
238                                    VoxEngine.sendMediaBetween(call, transferCall);
239                                    voiceAIClient.close();
240                                    Logger.write("===WARM_TRANSFER_BRIDGED===");
241                                } catch (bridgeError) {
242                                    Logger.write("===WARM_TRANSFER_BRIDGE_ERROR===");
243                                    Logger.write(bridgeError);
244                                }
245                            }, delayMs);
246                        });
247
248                        transferCall.addEventListener(CallEvents.Failed, (event) => {
249                            Logger.write("===WARM_TRANSFER_FAILED===");
250                            Logger.write(JSON.stringify(event));
251                            transferInProgress = false;
252                        });
253
254                        voiceAIClient.conversationItemCreate({
255                            item: {
256                                type: "function_call_output",
257                                call_id: toolCallId,
258                                output: JSON.stringify({
259                                    status: "transfer_started",
260                                    destination,
261                                    delay_ms: delayMs,
262                                }),
263                            },
264                        });
265                        voiceAIClient.responseCreate({});
266                    } catch (transferError) {
267                        Logger.write("===WARM_TRANSFER_ERROR===");
268                        Logger.write(transferError);
269                        transferInProgress = false;
270                        voiceAIClient.conversationItemCreate({
271                            item: {
272                                type: "function_call_output",
273                                call_id: toolCallId,
274                                output: JSON.stringify({error: "warm_transfer_failed"}),
275                            },
276                        });
277                        voiceAIClient.responseCreate({});
278                    }
279                    return;
280                }
281
282                voiceAIClient.conversationItemCreate({
283                    item: {
284                        type: "function_call_output",
285                        call_id: toolCallId,
286                        output: JSON.stringify({error: `Unhandled tool: ${toolName}`}),
287                    },
288                });
289                voiceAIClient.responseCreate({});
290            },
291        );
292
293        voiceAIClient.addEventListener(OpenAI.RealtimeAPIEvents.ResponseOutputAudioDone, () => {
294            if (!pendingHangup) return;
295            Logger.write("===HANGUP_AFTER_AGENT_AUDIO===");
296            pendingHangup = false;
297            call.hangup();
298        });
299
300        voiceAIClient.addEventListener(OpenAI.Events.WebSocketMediaEnded, () => {
301            if (!pendingHangup) return;
302            Logger.write("===HANGUP_AFTER_MEDIA_ENDED===");
303            pendingHangup = false;
304            call.hangup();
305        });
306
307        // Consolidated "log-only" handlers
308        [
309            OpenAI.RealtimeAPIEvents.ResponseCreated,
310            OpenAI.RealtimeAPIEvents.ResponseDone,
311            OpenAI.RealtimeAPIEvents.ResponseOutputAudioTranscriptDone,
312            OpenAI.RealtimeAPIEvents.ResponseFunctionCallArgumentsDelta,
313            OpenAI.RealtimeAPIEvents.ConnectorInformation,
314            OpenAI.RealtimeAPIEvents.HTTPResponse,
315            OpenAI.RealtimeAPIEvents.WebSocketError,
316            OpenAI.RealtimeAPIEvents.Unknown,
317            OpenAI.Events.WebSocketMediaStarted,
318            OpenAI.Events.WebSocketMediaEnded,
319        ].forEach((eventName) => {
320            voiceAIClient.addEventListener(eventName, (event) => {
321                Logger.write(`===${event.name}===`);
322                if (event?.data) Logger.write(JSON.stringify(event.data));
323            });
324        });
325    } catch (error) {
326        Logger.write("===UNHANDLED_ERROR===");
327        Logger.write(error);
328        voiceAIClient?.close();
329        VoxEngine.terminate();
330    }
331});

More information