Screen sharing

Voximplant SDKs allow developers to share their device screen to a call or conference.

How to share the screen in Web SDK

Screen sharing works in all desktop web browsers, such as Chrome, Edge, Safari, and Firefox.

To share the screen during an active call started via call() method, use the shareScreen() method during an active call. This method replaces the user video with screen sharing. Please note that you cannot share both the user video and device screen simultaneously.

1
// during an active call
2
call
3
  .shareScreen()
4
  .then(() => {
5
    // screen sharing started successfully
6
  })
7
  .catch(() => {
8
    // screen sharing did not start, catch errors
9
    // probably the user declined the sharing prompt
10
  });

To stop screen sharing, use the stopSharingScreen() method. You can also stop screen sharing via the browser’s native button to stop screen sharing.

To share the screen during a conference, you can use the shareScreen() method as well to replace the user video with screen sharing. If you need to share both user video and device screen, you can use the joinAsSharing() method to add your screen sharing as a separate conference participant.

The joinAsSharing() method accepts three parameters:

• num: conference number
• sendAudio: optional boolean parameter that specifies whether to send audio track or not
• extraHeaders: optional X-headers to pass into the call’s INVITE message

1
const sharingCall = client.joinAsSharing('my_conf');

To stop screen sharing that is connected as a separate conference participant, use the SharingCall.hangup() method.

To track if a user changed the video to screen sharing or vice versa, subscribe to the MediaRendererUpdated event. The result of this event has the type parameter, which specifies the video source.

1
client.addEventListener(VoxImplant.Hardware.HardwareEvents.MediaRendererUpdated, (res) => {
2
  if(res.type === 'sharing') {
3
    // the video source is changed to screen sharing
4
  }
5
})

How to share the screen in Android SDK

Screen sharing for Android is based on Android’s MediaProjection API. The minimum required Android version is 21.

Before you start screen sharing, request the user permission to share the screen via the MediaProjectionManager.createScreenCaptureIntent method, then handle the result via the startActivityForResult method.

Screen sharing can start only if a user gives permission for this action.

1
// create an intent and request the user permission
2
MediaProjectionManager mediaProjectionManager = (MediaProjectionManager)
3
    getContext().getSystemService(MEDIA_PROJECTION_SERVICE);
4
Intent screenSharingIntent = mediaProjectionManager.createScreenCaptureIntent();
5
startActivityForResult(screenSharingIntent, 1);
6
7
// handle the result
8
@Override
9
public void onActivityResult(int requestCode, int resultCode, Intent data) {
10
    if (requestCode == 1 && resultCode == Activity.RESULT_OK) {
11
        // user granted the permission, ready to start the screen sharing
12
    }
13
}

To start screen sharing, use the IstartScreenSharing method and pass to it the Intent.

Use the ICallCompletionHandler event to check if the screen sharing started successfully. In case of success, the ICallCompletionHandler.onComplete event is triggered, otherwise, the ICallCompletionHandler.onFailure event is triggered with the error code. You can see the error code list in the CallError enum.

1
call.startScreenSharing(screenSharingIntent, new ICallCompletionHandler() {
2
    @Override
3
    public void onComplete() {
4
        // screen sharing is successfully started
5
    }
6
7
    @Override
8
    public void onFailure(CallException exception) {
9
        // failed to start screen sharing
10
    }
11
});

To stop screen sharing, change the sendVideo method’s first argument. Set it to true to stop screen sharing and start sending the user video. Set it to false to stop screen sharing and send nothing to the call.

1
boolean shouldSendVideo;
2
...
3
call.sendVideo(shouldSendVideo, new ICallCompletionHandler() {
4
    @Override
5
    public void onComplete() {
6
        // screen sharing is successfully stopped
7
        // if sendVideo is set to true, the SDK sends the video from the camera
8
        // if sendVideo is set to false, the SDK stops sending the video
9
    }
10
11
    @Override
12
    public void onFailure(CallException exception) {
13
        // an error occurred, check CallError from the exception for more details
14
    }
15
});

How to share the screen in iOS SDK

There are two ways to share your screen in iOS:

• Application screen sharing: Easy to implement, but you can share only the screen of your application. For example, if you build a banking application, and you need to share the client’s screen for support purposes.
• Broadcast screen sharing: You can share the screen of your device including home screen, panels, and any other 3rd-party applications, but you need to install the Broadcast Upload App/Setup UI App extensions.

The minimum required iOS version to share the screen is 11.

Application screen sharing

To share the application screen, use the VICall.startInAppScreenSharing method. After the user accepts the standard iOS security prompt, the screen sharing starts. The user’s decision is not saved between application launches.

1
let call: VICall
2
call.startInAppScreenSharing { [weak self] error in
3
    if let error = error {
4
        // failed to start screen sharing
5
        return
6
    }
7
    // screen sharing is successfully started
8
}

To stop sharing your application screen, use the VICall.setSendVideo method. Set it to true to stop screen sharing and start sending user video. Set it to false to stop screen sharing and send nothing to the call.

Broadcast screen sharing

iOS Broadcast screen sharing allows users to share the screen of your device including home screen, panels, and any other 3rd-party applications, but you need to install the Broadcast Upload App/Setup UI App extensions.

To implement Broadcast screen sharing you need to install a separate application extension to your Xcode project.

Open your Xcode project and go File → New → Target → Broadcast Upload Extension:

Click Next, enter a name for a new target, and unselect the Include UI Extension checkbox as you do not need it.

After Broadcast Upload Extension adds to your Xcode project, you can launch it via two ways:

• Control panel, available since iOS 11. Add a screen recording button in the Settings → Control center → Customize controls → Screen recording. Then open your control panel and choose the Screen recording feature, then select BroadcastUploadExtension.
• Button in your application, available since iOS 12. Add an RPSystemBroadcastPickerView button to your application to launch screen sharing.

After the user gives permission for broadcasting, the application extension process launches, and the control is switched to the RPBroadcastSampleHandler broadcastStartedWithSetupInfo method.

Next, your application needs to connect to the Voximplant cloud and authorize the user. The token-based authorizaion is preferred.

You can pass the credentials and the roomid from your app to Voximplant through the app extension, e.g., Keychain or UserDefaults. Bear in mind that this requires setting up the App Group.

1
// app extension code:
2
class SampleHandler: RPBroadcastSampleHandler {
3
    let viclient: VIClient
4
    ...
5
    override func broadcastStarted(withSetupInfo setupInfo: [String : NSObject]?) {
6
        viclient.connect()
7
        ...
8
    }
9
    ...
10
    // MARK: VIClientSessionDelegate
11
    func clientSessionDidConnect(_ viclient: VIClient) {
12
       viclient.login(withUser: username; token: accesstoken, success: {
13
           (displayName: String, tokens: VIAuthParams) in
14
           // successfully logged in
15
       }, failure: { (error: Error) in
16
           // error handling
17
       }
18
    }
19
    ...
20
}

Now the application should create a new call, connect it to the conference, and start sending frames to the conference. The screen frames are processed via the VICustomVideoSource class.

Due to RAM restrictions for app extensions (50 Mb RAM limit):

• Receiving of voice and video should be disabled for screen sharing call
• Audio sending should be disabled
• The codec should be H.264

1
// app extension code:
2
// create a custom video source instance
3
let screenVideoSource = VICustomVideoSource(screenCastFormat: ())
4
5
// create and start call
6
let roomid: String
7
let settings = VICallSettings()
8
// It is important to use only .H264 encoding in appex.
9
settings.preferredVideoCodec = .H264
10
settings.receiveAudio = false
11
settings.videoFlags = VIVideoFlags.videoFlags(receiveVideo: false, sendVideo: true)
12
if let screenSharingCall:VICall = viclient.callConference(roomid, settings: settings) {
13
     // change the custom camera instance
14
     screenSharingCall.videoSource = screenVideoSource
15
     screenSharingCall.sendAudio = false
16
     screenSharingCall.start()
17
}

To pass the screen capture to the call, implement a custom video source via the initScreenCastFormat constructor.

Broadcast extension sends the frames via the RPBroadcastSampleHandler.processSampleBuffer method.

There are three types of the frames: .video, .audioApp, .audioMic. The .video frames should be passed to a custom video source via the sendVideoFrame:rotation method. You can obtain the rotation from the resulting frame.

1
// app extension code
2
override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer,
3
                           with sampleBufferType: RPSampleBufferType) {
4
    if sampleBufferType != .video {
5
        return
6
    }
7
8
    if let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) {
9
        var rotation: VIRotation?
10
        if let uint32rotation = CMGetAttachment(sampleBuffer,
11
                                  key: RPVideoSampleOrientationKey as CFString,
12
                                  attachmentModeOut: nil)?.uint32Value
13
        {
14
            switch CGImagePropertyOrientation(rawValue: uint32rotation) {
15
               case .up?, .upMirrored?:
16
                rotation  = ._0
17
            case .right?, .rightMirrored?:
18
                rotation = ._270
19
            case .left?, .leftMirrored?:
20
                rotation = ._90
21
            case .down?, .downMirrored?:
22
                rotation  = ._180
23
            }
24
        }
25
26
        self.screenVideoSource.sendVideoFrame(pixelBuffer, rotation:rotation)
27
    }
28
}

As you have two separate participants in the conference (the call and the screen sharing), you need to synchronize the statuses of two calls. For example, you have to pass the information about the end of one call to the other, between the processes of the operating system. To do this, you can use the CFNotifyCenterGetDarwinNotifyCenter method.