The initial step in utilizing Multistream is to include enableMultistream within the joinOptions parameter for both the meeting.join() and meeting.joinWithMedia() methods.

Example

const joinOptions = {
  enableMultistream: true,
  ...
};

await meeting.join(joinOptions);

// or

await meeting.joinWithMedia({joinOptions, mediaOptions});

This article is divided into three sections:

1. Layouts: Determines the preferred visualization of the remote streams.
2. Events: Monitors all remote streams and additional events.
3. Methods and RemoteMediaManager: Facilitates layout updates and other operations.

Terminology

Throughout this document we'll refer to the following terms:

• Layout: An arrangement of remote streams in the user interface.
• Pane: An individual element within the layout grid.
• Remote Media: The media stream and associated details.
• Remote Media Group: A collection of remote media as a group, along with additional helper APIs.
• CSI: A unique identifier linked to a stream, connecting a member participant with that stream.

anchorLayout

A "layout" refers to the arrangement of remote streams in the user interface. Multistream allows you to customize this layout. To do so, call addMedia() with the desired layout details. The existing arguments for addMedia() are unaffected by this process. For more information, refer to this documentation.

The AddMediaOptions now includes a new property, remoteMediaManagerConfig, for multistream functionality. To implement a custom layout, you can use the same data format as shown in the predefined layouts.

The following are the default configuration values for remoteMediaManagerConfig, which are used if no overrides are provided:

const remoteMediaManagerConfig = {
  audio: {
    numOfActiveSpeakerStreams: 3,
    numOfScreenShareStreams: 1,
  },
  video: {
    preferLiveVideo: true,
    initialLayoutId: 'AllEqual',

    layouts: {
      AllEqual: AllEqualLayout,
      OnePlusFive: OnePlusFiveLayout,
      Single: SingleLayout,
      Stage: Stage2x2With6ThumbnailsLayout,
      ScreenShareView: RemoteScreenShareWithSmallThumbnailsLayout,
    }
  }
}

A detailed explanation of this object can be found here.

Here are the predefined layouts for the default configuration:

Layout Details

The following setions describe each available layout in detail.

AllEqual

An "all equal" grid, with size up to 3 x 3 = 9.

Example

const AllEqualLayout: VideoLayout = {
  activeSpeakerVideoPaneGroups: [
    {
      id: 'main',
      numPanes: 9,
      size: 'best',
      priority: 255,
    },
  ],
};

OnePlusFive

A layout with one big pane for the highest-priority active speaker and 5 small panes for other active speakers.

Example

const OnePlusFiveLayout: VideoLayout = {
  activeSpeakerVideoPaneGroups: [
    {
      id: 'mainBigOne',
      numPanes: 1,
      size: 'large',
      priority: 255,
    },
    {
      id: 'secondarySetOfSmallPanes',
      numPanes: 5,
      size: 'very small',
      priority: 254,
    },
  ],
};

Single

A layout with a single remote active speaker video pane.

Example

const SingleLayout: VideoLayout = {
  activeSpeakerVideoPaneGroups: [
    {
      id: 'main',
      numPanes: 1,
      size: 'best',
      priority: 255,
    },
  ],
};

Stage

A staged layout with four pre-selected meeting participants in the main 2x2 grid and 6 small panes for other active speakers at the top.

The predetermined participants can be included using the CSI value, which can be updated once the values are accessible after joining the meeting.

Example

const Stage2x2With6ThumbnailsLayout: VideoLayout = {
  activeSpeakerVideoPaneGroups: [
    {
      id: 'thumbnails',
      numPanes: 6,
      size: 'thumbnail',
      priority: 255,
    },
  ],
  memberVideoPanes: [
    {id: 'stage-1', size: 'medium', csi: undefined},
    {id: 'stage-2', size: 'medium', csi: undefined},
    {id: 'stage-3', size: 'medium', csi: undefined},
    {id: 'stage-4', size: 'medium', csi: undefined},
  ],
};

ScreenShareView

A strip of eight small video panes (thumbnails) displayed at the top of a remote screenshare.

Example

const RemoteScreenShareWithSmallThumbnailsLayout: VideoLayout = {
  screenShareVideo: {size: 'best'},
  activeSpeakerVideoPaneGroups: [
    {
      id: 'thumbnails',
      numPanes: 8,
      size: 'thumbnail',
      priority: 255,
    },
  ],
};

RemoteMediaManager Configuration Object

This object configures the RemoteMediaManager. Based on this configuration, you will receive the multistream remote media.

{
  audio: {
    numOfActiveSpeakerStreams: number; 
    numOfScreenShareStreams: number; 
  };
  video: {
    preferLiveVideo: boolean; 
    initialLayoutId: LayoutId;

    layouts: {[key: LayoutId]: VideoLayout}; 
  };
}

Properties of Audio Object

Properties of Video Object

Properties of VideoLayout Object

PaneSize can take the following values

Properties of ActiveSpeakerPaneObject

Properties of MemberPaneObject

Member video panes can only be accessed within the stage layout, which allows for the pinning of specific participants' positions in the layout. The stage layout is created when memberVideoPanes is included in the layout configuration. In this case, developers are responsible for managing and displaying participants as they join or leave. This is different from activeSpeakerVideoPaneGroups, where a consistent list of streams featuring active speakers/participants is always received. Additionally, the active speaker video pane can be pinned. Therefore, participants can be pinned either by using memberVideoPanes or by invoking pinActiveSpeakerVideoPane(remoteMedia, csi) for activeSpeakerVideoPaneGroups.

Each member or participant can have multiple streams. These streams can be of various types such as audio, video, screenshareVideo, or screenshareAudio. Each stream is uniquely identified by a CSI. A member can have more than one stream of the same type. For instance, a user can have two video streams (each with a unique CSI) if they join the meeting using multiple devices.

In most scenarios, you'll primarily utilize activeSpeakerVideoPaneGroups.

anchorEvents

Events can be listened to using the meeting.on() method.

Here's an example:

meeting.on('event name', (data) => {
  console.log(data);
});

meeting.on('media:remoteVideo:layoutChanged', ({
  layoutId, activeSpeakerVideoPanes, memberVideoPanes, screenShareVideo
}) => {
  console.log('layoutId: ', layoutId);
  console.log('activeSpeakerVideoPanes:', activeSpeakerVideoPanes);
  console.log('memberVideoPanes:', memberVideoPanes);
  console.log('screenShareVideo:', screenShareVideo);
});

Media Events

The following table and sections elaborate on the media events received during a meeting that pertain to multistream.

media:remoteScreenShareAudio:created

To obtain the list of remote media elements from the group, invoke getRemoteMedia().

const remoteMedia = screenShareAudioMediaGroup.getRemoteMedia()[0];
const remoteMediaStream = remoteMedia.stream;

Asynchronous: No

Parameters

Returns: Array<RemoteMedia>

media:remoteVideo:layoutChanged

media:remoteVideoSourceCountChanged

media:remoteAudioSourceCountChanged

Meeting Events

The following table and sections elaborate on the meeting events received during a meeting that pertain to multistream.

Remote Media Events

The remoteMedia object may include streams from various participants and can dynamically update as different participants begin speaking. The memberId within the remoteMedia object allows for the display of corresponding names or data alongside the stream. This memberId can be utilized to retrieve the name of the participant.

Events can be listened to using the remoteMedia.on() method.

Example

remoteMedia.on('Event name', (data) => {
  console.log(data);
});

With sourceUpdate, you can inspect the memberId and sourceState to obtain the most recent information about the remoteMedia.

Here's the possible information received during the sourceUpdate event:

anchorMethods

Multiple methods are available for handling various scenarios and updating the layout.

Get an Array of a Member's CSIs

const csiList = meeting.members.getCsisForMember(memberId, mediaType='video', mediaContent='main');

Asynchronous: No

Parameters

Returns: Array<number>

Find A Member by CSI

const member = meeting.members.findMemberByCsi(csi);

Asynchronous: No

Parameters:

Returns: Member

anchorRemote Media Manager

In multistream meetings, the meeting.addMedia() API establishes the media connection. It configures streams to receive remote media based on the remoteMediaManagerConfig provided as an option.

The Meeting.addMedia() method accepts an optional configuration object to set up the RemoteMediaManager:

meeting.addMedia({
  ...,
  remoteMediaManagerConfig?: RemoteMediaManagerConfiguration;
})

RemoteMediaManager configuration object

{
  audio: {
    numOfActiveSpeakerStreams: number; // number of audio streams we want to receive
    numOfScreenShareStreams: number; // 1 should be enough because in Webex only 1 person at a time can be presenting screen share
  };
  video: {
    preferLiveVideo: boolean; // applies to all pane groups with active speaker policy
    initialLayoutId: LayoutId;

    layouts: {[key: LayoutId]: VideoLayout}; // a map of all available layouts, a layout can be set via setLayout() method
  };
}

This object's details are outlined in the Explanation of the RemoteMediaManager Object section.

Set a Remote Video CSI

meeting.remoteMediaManager.setRemoteVideoCsi(remoteMedia, csi);

Asynchronous: No

Parameters:

Returns: void

Add a Member Video Pane

const remoteMedia = await meeting.remoteMediaManager.addMemberVideoPane(
  { id: PaneId; size: PaneSize; csi?: CSI; }
);

Asynchronous: Yes

Parameters: An object with following parameters

Returns: Promise<RemoteMedia>

Get the Currently Selected Layout ID

const layoutId = meeting.remoteMediaManager?.getLayoutId();

Asynchronous: No

Parameters: No

Returns: string

Change the Layout

Calling this method triggers the media:remoteVideo:layoutChanged event.

await meeting.remoteMediaManager.setLayout(layoutId);

Asynchronous: Yes

Parameters:

Returns: Promise<void>

Set the prefer live video

meeting.remoteMediaManager.setPreferLiveVideo(preferLiveVideo);

Asynchronous: No

Parameters:

Returns - void

Set Active Speaker CSIs

Set CSIs for multiple RemoteMedia instances belonging to a RemoteMediaGroup.

meeting.remoteMediaManager.setActiveSpeakerCsis(remoteMediaCsis);

Asynchronous: No

Parameters: An array of object with the following properties, for example, Array<{ remoteMedia, csi }>.

Returns: void

Set a Remote Video CSI

Set a new CSI on a given remote media object

meeting.remoteMediaManager.setRemoteVideoCsi(remoteMedia, csi);

Asynchronous: No

Parameters:

Returns - void

Add a Member Video Pane

Add a new member video pane to the currently selected layout.

const remoteMedia = await meeting.remoteMediaManager.addMemberVideoPane(newPane);

Asynchronous: Yes

Parameters: An object with the following properties

Returns - Promise<RemoteMedia>

Remove Member Video Pane

Remove a member video pane from the currently selected layout.

await meeting.remoteMediaManager.removeMemberVideoPane(paneId);

Asynchronous: Yes

Parameters: An object with the following properties:

Returns - Promise<void>

Pin an Active Speaker Video Pane

Pin an active speaker remote media object to the given CSI value.

meeting.remoteMediaManager.pinActiveSpeakerVideoPane(remoteMedia, csi);

This function pins an active speaker's remote media object to a specified CSI value. Consequently, the remote media will only play audio/video from that specific CSI. This remains in effect until either the unpinActiveSpeakerVideoPane() function is called or the current layout is altered.

Asynchronous: No

Parameters: An object with the following properties

Returns - void

Unpin Active Speaker Video Pane

Unpin a remote media object from the specific CSI value to which it was previously pinned.

meeting.remoteMediaManager.unpinActiveSpeakerVideoPane(remoteMedia);

Asynchronous: No

Parameters:

Returns - void

Pinned

Determine whether a given remote media object is part of an active speaker group and if it has been pinned.

const isPinned = meeting.remoteMediaManager.isPinned(remoteMedia);

Asynchronous: No

Parameters:

Returns - boolean

anchorSet Size Hint

Clients can specify a size hint if they are displaying remote video on screens of varying sizes and need to adjust the video resolution for optimal display or bandwidth conservation. When the remote receives this hint, it starts transmitting the video at the requested resolution.

setSizeHint() can be invoked on the remoteMedia object. Each video stream corresponds to a remoteMedia object.

remoteMedia.setSizeHint(width, height);

Asynchronous: No

Parameters:

Returns - void

anchorKitchen Sink App

To explore Multistream functionality, feel free to explore our Meeting Samples App.

Under Manage Meeting section, check the box with label Use a multistream connection before clicking on Join Meeting or Join with Media.