Custom video sources and backgrounds

This article explains how to use a custom video source in calls and conferences.

Custom video source API allows you to send custom video to a call or a conference from video sources managed on the application side. It gives the possibility to implement such functionality as camera filters, background blur, video background replacement, and any kind of the video preprocessing, using 3rd party libraries. It is important to consider that the application is responsible for the management of all resources for video processing including their proper deallocation.

Configure the SDK

Before the SDK is initialized, it is required to create a WebRTC EglBase instance that holds EGL state and shared configuration.

Android SDK v2

Create an EglBase instance via the EglBase.create() API, and add it into ClientConfig before calling Voximplant.getClientInstance().

val eglBase = EglBase.create()
val clientConfig = ClientConfig()
clientConfig.eglBase = eglBase
val client = Voximplant.getClientInstance(
    /* executor = */ Executors.newSingleThreadExecutor(),
    /* context = */ applicationContext,
    /* clientConfig = */ clientConfig,
)

Android SDK v3

The EglBase instance should be created and set to the VICalls.eglBase property before the calls module is initialized using the VICalls.initialize API.

val eglBase = EglBase.create()
VICalls.eglBase = eglBase
VICalls.initialize()

Set up a custom video source

Custom video source takes the video frames from a SurfaceTexture into the application renders them. The WebRTC SurfaceTextureHelper class provides API that creates a SurfaceTexture and allows the SDK to convert a texture frame into a required format to send the frame to a call or a conference. SurfaceTextureHelper should be created via an EglBase instance that the SDK has been configured with. It is also required to configure the SurfaceTexture size according to the video resolution.

val surfaceTextureHelper: SurfaceTextureHelper? = SurfaceTextureHelper.create(
    "threadName",
    eglBase.eglBaseContext,
)
surfaceTextureHelper!!.setTextureSize(frameWidth, frameHeight)

Android SDK v2

ICustomVideoSource interface represents the custom video source. Create it via the Voximplant.getCustomVideoSource API and configure it with the SurfaceTextureHelper instance via the ICustomVideoSource.setSurfaceTextureHelper API.

Subscribe to custom video source events to handle when the SDK is ready to consume the video frames.

val customVideoSource = Voximplant.getCustomVideoSource()
customVideoSource.setSurfaceTextureHelper(surfaceTextureHelper)
customVideoSource.setCustomVideoSourceListener(object : ICustomVideoSourceListener {
    override fun onStarted() {
        // start rendering to the surfaceTextureHelper.surfaceTexture
    }
    override fun onStopped() {
        // stop rendering to the surfaceTextureHelper.surfaceTexture
    }
})

Android SDK v3

The CustomVideoSource class represents the custom video source. Create its instance and set it up with the SurfaceTextureHelper instance. Use the CustomVideoSource.addListener API to subscribe for the custom video source events.

val customVideoSource = CustomVideoSource()
customVideoSource.surfaceTextureHelper = surfaceTextureHelper
customVideoSource.addListener(object : VideoSource.EventsListener {
        override fun onStart() {
            // start rendering to the surfaceTextureHelper.surfaceTexture
        }
        override fun onStop(reason: VideoSource.StopReason) {
            // stop rendering to the surfaceTextureHelper.surfaceTexture
        }
        override fun onError(error: VideoSource.Error) {}
    }
)

Use the custom video source in a call

Android SDK v2

Use the Call.useCustomVideoSource API to make the SDK use the provided custom video source instead of camera device.

val callSettings = CallSettings()
callSettings.videoFlags = VideoFlags(true, true)
val call: ICall? = client.call(user, callSettings)
call!!.useCustomVideoSource(customVideoSource)
try {
    call!!.start()
} catch (callException: CallException) {
    // handle error
}

Android SDK v3

Create a LocalVideoStream instance with the custom video source and set it to the call or conference settings.

val localVideoStream = LocalVideoStream(customVideoSource)
val callSettings = CallSettings()
callSettings.localVideoStream = localVideoStream
callSettings.receiveVideo = true
val call: Call? = VICalls.createCall(
    number = "number_to_call",
    callSettings = callSettings,
)
try {
    call!!.start()
} catch (callException: CallException) {
    // handle error
}

See also

• Voximplant DeepAR video call GitHub demo