Example: Using Vertex AI | Voximplant Voice AI

For the complete documentation index, see llms.txt.

This example answers an inbound Voximplant call and connects Gemini Live API through Vertex AI. It reads Vertex service account credentials from Voximplant Secrets through a shared credentials scenario.

⬇️ Jump to the Full VoxEngine scenario.

Prerequisites

Set up an inbound entrypoint for the caller:
Create a routing rule that points the destination (phone number / WhatsApp / SIP username / app user alias) to this scenario.
Store your Vertex parameters in Voximplant ApplicationStorage:
- GCP_PROJECT_ID
- GCP_REGION
Create a GOOGLE_CREDENTIALS Secret containing the complete service account JSON as a string.

For full Vertex setup details, use the rest of this guide. To create a service account key (JSON), see: https://docs.cloud.google.com/iam/docs/keys-create-delete

Credentials scenario (separate route entry)

Include the credentials scenario before the main scenario in your routing rule. It reads the JSON string from GOOGLE_CREDENTIALS and stops immediately if the Secret is missing.

Routing rule scenario order:

gemini-vertex-credentials → gemini-using-vertex-ai

Why a separate scenario?

VoxEngine shares global scope across scenarios in the same routing rule, so the credentials scenario sets GEMINI_VERTEX_CREDENTIALS for the main scenario. Keep non-secret project and region values in ApplicationStorage.

Example credentials scenario:

voxeengine-gemini-vertex-credentials.js

/**
 * Read Vertex AI service account JSON from Voximplant Secrets.
 * Include this scenario BEFORE the main Vertex AI scenario in the routing rule.
 */
// shared with subsequent scenarios in the routing rule
var GEMINI_VERTEX_CREDENTIALS = VoxEngine.getSecretValue("GOOGLE_CREDENTIALS");
if (!GEMINI_VERTEX_CREDENTIALS) throw new Error("Set GOOGLE_CREDENTIALS in Voximplant Secrets");

JSON string

VoxEngine.getSecretValue("GOOGLE_CREDENTIALS") returns the JSON string expected by the connector. Pass it directly as credentials.

Session setup

The Gemini Live API session is configured via connectConfig, passed into Gemini.createLiveAPIClient(...).

In the full scenario, see GEMINI_CONNECT_CONFIG:

responseModalities: ["AUDIO"] asks Gemini to speak back in real time.
apiVersion: "v1alpha" matches the Vertex Live API endpoint.
speechConfig selects a prebuilt voice.
systemInstruction defines the assistant behavior.
tools and toolConfig enable function calling.

The Vertex-specific parameters are passed directly to Gemini.createLiveAPIClient(...):

backend: Gemini.Backend.VERTEX_AI
project → GCP_PROJECT_ID
location → GCP_REGION
credentials → the JSON string set by the credentials scenario

Connect call audio

After Gemini.LiveAPIEvents.SetupComplete, bridge audio between the call and Gemini:

Connect call audio

VoxEngine.sendMediaBetween(call, geminiLiveAPIClient);

The same handler sends a starter message to trigger the greeting:

Trigger the greeting

geminiLiveAPIClient.sendClientContent({
  turns: [{ role: "user", parts: [{ text: "Say hello and ask how you can help." }] }],
  turnComplete: true,
});

Events

The scenario logs Gemini events for debugging:

Gemini.LiveAPIEvents: SetupComplete, ServerContent, ToolCall, ToolCallCancellation, ConnectorInformation, Unknown
Gemini.Events: WebSocketMediaStarted, WebSocketMediaEnded

Notes

This example uses the Vertex AI backend (Gemini.Backend.VERTEX_AI).
The credentials scenario only sets GEMINI_VERTEX_CREDENTIALS in the shared global scope and does not start any calls.

Gemini 3.8 Live on Vertex AI

gemini-3.8-live is not available on Vertex AI unless the Google Cloud project has Gemini Enterprise early access. This example keeps the Vertex-supported model gemini-live-2.5-flash-preview-native-audio-09-17. Developer API examples use gemini-3.8-live.

See the VoxEngine API Reference for more details.

Full VoxEngine scenario

voxeengine-gemini-using-vertex-ai.js

/**
 * Voximplant + Gemini Live API connector demo
 * Scenario: answer an incoming call using Gemini Live API via Vertex AI.
 */
require(Modules.Gemini);
require(Modules.ApplicationStorage);
const SYSTEM_PROMPT = `You are Voxi, a helpful voice assistant for phone callers. Keep responses short and telephony-friendly (usually 1-2 sentences).`;
// -------------------- Gemini Live API settings --------------------
const CONNECT_CONFIG = {
    responseModalities: ["AUDIO"],
    speechConfig: {
        voiceConfig: {
            prebuiltVoiceConfig: {voiceName: "Aoede"},
        },
    },
    systemInstruction: {
        parts: [{text: SYSTEM_PROMPT}],
    },
    inputAudioTranscription: {},
    outputAudioTranscription: {},
};
VoxEngine.addEventListener(AppEvents.CallAlerting, async ({call}) => {
    let voiceAIClient;
    // Termination functions - add cleanup and logging as needed
    call.addEventListener(CallEvents.Disconnected, ()=>VoxEngine.terminate());
    call.addEventListener(CallEvents.Failed, ()=>VoxEngine.terminate());
    try {
        if (typeof GEMINI_VERTEX_CREDENTIALS !== "string" || !GEMINI_VERTEX_CREDENTIALS) {
            throw new Error("Load the credentials scenario and set GOOGLE_CREDENTIALS in Voximplant Secrets");
        }
        call.answer();
        // call.record({hd_audio: true, stereo: true}); // optional: call recording
        voiceAIClient = await Gemini.createLiveAPIClient({
            credentials: GEMINI_VERTEX_CREDENTIALS,                           // stored in a different scenario
            model: "gemini-live-2.5-flash-preview-native-audio-09-17",
            backend: Gemini.Backend.VERTEX_AI,                                // Set the Vertex back-end
            project: (await ApplicationStorage.get("GCP_PROJECT_ID")).value,  // required for Vertex
            location: (await ApplicationStorage.get("GCP_REGION")).value,     // required for Vertex
            connectConfig: CONNECT_CONFIG,
            onWebSocketClose: (event) => {
                Logger.write("===Gemini.WebSocket.Close===");
                if (event) Logger.write(JSON.stringify(event));
                VoxEngine.terminate();
            },
        });
        // ---------------------- Event handlers -----------------------
        // Wait for Gemini setup, then bridge audio and trigger the greeting
        voiceAIClient.addEventListener(Gemini.LiveAPIEvents.SetupComplete, () => {
            VoxEngine.sendMediaBetween(call, voiceAIClient);
            voiceAIClient.sendClientContent({
                turns: [{role: "user", parts: [{text: "Say hello and ask how you can help."}]}],
                turnComplete: true,
            });
        });
        // Capture transcripts + handle barge-in
        voiceAIClient.addEventListener(Gemini.LiveAPIEvents.ServerContent, (event) => {
            const payload = event?.data?.payload || {};
            if (payload.inputTranscription?.text) {
                Logger.write(`===USER=== ${payload.inputTranscription.text}`);
            }
            if (payload.outputTranscription?.text) {
                Logger.write(`===AGENT=== ${payload.outputTranscription.text}`);
            }
            if (payload.interrupted) {
                Logger.write("===BARGE-IN=== Gemini.LiveAPIEvents.ServerContent");
                voiceAIClient.clearMediaBuffer();
            }
        });
        // Log all Gemini events for illustration/debugging
        [
            Gemini.LiveAPIEvents.SetupComplete,
            Gemini.LiveAPIEvents.ServerContent,
            Gemini.LiveAPIEvents.ToolCall,
            Gemini.LiveAPIEvents.ToolCallCancellation,
            Gemini.LiveAPIEvents.ConnectorInformation,
            Gemini.LiveAPIEvents.Unknown,
            Gemini.Events.WebSocketMediaStarted,
            Gemini.Events.WebSocketMediaEnded,
        ].forEach((eventName) => {
            voiceAIClient.addEventListener(eventName, (event) => {
                Logger.write(`===${event.name}===`);
                if (event?.data) Logger.write(JSON.stringify(event.data));
            });
        });
    } catch (error) {
        Logger.write("===SOMETHING_WENT_WRONG===");
        Logger.write(JSON.stringify(error));
        VoxEngine.terminate();
    }
});