Realtime Media API

The media plane shares live WebRTC MediaStreams (camera, mic, screen, canvas.captureStream()) between peers, with the same addressing model as data — session-wide, a subset of peers, or by topic. A client shares a stream scoped to an audience; the SDK realizes that intent over the peer-to-peer mesh, tags inbound streams { peerId, topic }, and handles renegotiation and glare.

For a task-oriented walkthrough, see the Share Realtime Media cookbook recipe. Protocol details are in spec §18; design rationale in protocol/rfcs/0001-realtime-media.md.

Browser-only, mesh-only today

The media API needs MediaStream/WebRTC, so it is available in the TypeScript SDK (browser) only. Python, Swift, and other SDKs have no media API yet — they must first gain RTC. client.media throws if RTC is not enabled on the client.

The mesh has no server-side media fan-out: sharing one stream to K peers means K encoded uploads from the publisher. Subset/small-group sharing is the well-suited primary case; a media topic with many subscribers is a mesh foot-gun. The API is topology-neutral so an SFU can be enabled later (§18.6) with no application-code change — but no SFU exists yet.

Scopes

A publication's scope is declarative intent, never an imperative "open these connections." It mirrors the data plane one-to-one:

Scope	share call	Data-plane analogue	Reaches
session	media.share(stream)	broadcast(payload)	every peer in the session (default)
audience	media.share(stream, { to })	send(to, payload)	the listed subset of peers
topic	media.share(stream, { topic }) + media.subscribe(topic)	publish + subscribe	peers subscribed to the media topic

Because media is per-RTCPeerConnection, subset targeting is the underlying primitive. Resolution to concrete peers (from sdks/typescript/src/media-publisher.ts):

• session → every currently connected RTC peer. New peers pick up the stream as they connect.
• audience → the peers in to that are connected; not-yet-connected audience peers receive the track once their connection is established.
• topic → the subscribers the server reports via the topic/peers map (§17.1), filtered to those connected. The set is reconciled whenever the subscriber list changes. Sharing on a topic registers publisher interest (§17.2) so the server delivers the topic/peers map to the publisher without it subscribing — a publisher never has to join its own topic to learn who to send to, and so is never forced to receive peers' streams on it. The SDK registers on share(stream, { topic }) and withdraws on stop().

Media flows over the same peer connections opened for WebRTC data channels — establish them with connectRTC().

TypeScript API — Media class

Access the media plane through client.media (a Media instance). Defined in sdks/typescript/src/media.ts; types in sdks/typescript/src/media-types.ts.

Member	Signature	Description
share	share(stream: MediaStream, opts?: ShareOptions): Publication	Share a stream to the resolved scope. Returns a Publication synchronously; renegotiation happens in the background.
subscribe	subscribe(topic: string): Promise<void>	Declare interest in a media topic. Reuses the data-topic subscription authority (§18.3).
unsubscribe	unsubscribe(topic: string): Promise<void>	Withdraw interest in a media topic.
on	on(event: "stream" | "streamended", cb: (s: RemoteStream) => void): Unsubscribe	Subscribe to remote-stream lifecycle. "stream" fires when an inbound stream is first surfaced; "streamended" when a track ends or the peer disconnects.
remote$	Observable<RemoteStream[]>	Reactive list of all inbound remote streams; emits the full array on every add/remove/retag. Initial value [].

client.getPeerConnection(peerId) is the escape hatch to the raw connection (see below); it lives on StarfishClient, not Media.

Publication

share() returns a handle to the local stream you are sharing. Defined in sdks/typescript/src/media-types.ts; implemented in media-publisher.ts.

Member	Signature	Description
id	readonly string	Publication id (e.g. pub…).
setEnabled	setEnabled(enabled: boolean): void	Mute/unmute by toggling enabled on every track. No renegotiation.
replaceTrack	replaceTrack(track: MediaStreamTrack): Promise<void>	Swap the outgoing track in place (e.g. camera → screenshare) without re-sharing. Replaces the track on every per-peer sender.
stop	stop(): void	Stop sharing: remove the tracks from all peers (renegotiates) and stop them. Receivers observe streamended.

Sharing the same stream again updates its scope rather than duplicating it (§18.2).

getPeerConnection

client.getPeerConnection(peerId: string): RTCPeerConnection | null

Escape hatch returning the live RTCPeerConnection for a peer, or null if none exists. For power users needing transceiver-level control, stats, or bandwidth tuning. Defined on StarfishClient (sdks/typescript/src/client.ts), delegating to the RTC layer.

Types

/** Options for `media.share`. Omit both `to` and `topic` to share to the session. */
interface ShareOptions {
  to?: string[];       // audience scope — share to this subset of peers
  topic?: string;      // topic scope — share to whoever subscribed to this topic
  simulcast?: unknown; // reserved for the future SFU; a no-op in the mesh phase
}

/** A handle to a local stream the client is sharing. */
interface Publication {
  readonly id: string;
  setEnabled(enabled: boolean): void;
  replaceTrack(track: MediaStreamTrack): Promise<void>;
  stop(): void;
}

/** An inbound remote media stream, tagged with its origin peer and topic. */
interface RemoteStream {
  peerId: string;        // who is sending it
  topic: string | null;  // the media topic, or null for session/audience scope
  stream: MediaStream;
}

Scope precedence in share: topic > to > session. If topic is set it wins; else if to is set it is an audience share; otherwise the stream is shared to the whole session. simulcast is accepted and ignored in mesh.

The SDK's source types use browser-neutral …Like interfaces (MediaStreamLike, MediaStreamTrackLike) so the plane is testable without a real WebRTC stack. At runtime these are the browser's MediaStream / MediaStreamTrack.

Protocol Message Types

The media plane adds no new frame envelope and requires no server changes in the mesh phase. Its only wire-level addition is the peer-to-peer media.map control frame; stream / streamended are SDK-level events derived from the WebRTC ontrack/track-end callbacks, not frames the server sends.

Resource	Method	Kind	Direction	Description
media	map	event	peer → peer	Track↔topic mapping over the starfish.control data channel (§18.4)
media	stream	event	SDK-local	Inbound media stream started (from ontrack)
media	streamended	event	SDK-local	Inbound media stream ended (track end or peer disconnect)

media.map — track↔topic mapping (§18.4)

SDP m-lines carry no application-level topic name, so when a publisher adds a track for a scope it sends a small media.map over the starfish.control DataChannel tagging each track with its topic (or null for session/audience scope). The receiver uses it to label the inbound ontrack as a RemoteStream { peerId, topic }. This is the only media-specific protocol addition in the mesh phase.

{
  "header": {
    "id": "media_001",
    "resource": "media",
    "method": "map",
    "kind": "event",
    "session": "show-abc",
    "from": "client_a7f3"
  },
  "payload": {
    "tracks": [
      { "mid": "0", "streamId": "stage-cam-stream", "topic": "stage-cam" },
      { "mid": "1", "streamId": "cam-stream", "topic": null }
    ]
  }
}

Field	Type	Description
tracks[].mid	string	SDP media-line id. Receivers SHOULD prefer mid when present.
tracks[].streamId	string	The MediaStream.id, used to correlate the track.
tracks[].topic	string | null	Media topic tag, or null/omitted for session/audience scope.

media.map is (re)sent whenever the publisher's track set changes, after the corresponding renegotiation. It is advisory: a receiver that has not yet received the map for a track MAY hold it untagged until the map arrives. Receivers MUST validate inbound topic tags against their own subscription set and silently drop unauthorized media (§18.3, §18.8).

welcome.media — server media capability (§18.5)

The server advertises media support with an optional, additive media block in the server.welcome payload. A server that omits it is assumed to support the mesh phase (the signaling relay is unchanged) or, by policy, no media at all.

{
  "payload": {
    "status": "ok",
    "version": 1,
    "clientId": "client_a7f3",
    "media": {
      "modes": ["mesh"],
      "sfu": null
    }
  }
}

Field	Type	Description
modes	string[]	Topologies offered, in preference order. "mesh" is the floor; a future server adds "sfu", e.g. ["mesh", "sfu"].
sfu	object | null	Reserved SFU endpoint descriptor; null in the mesh phase. Shape left open for the SFU phase; clients that only speak mesh MUST ignore it.

The client selects a mode from what the server offers intersected with session policy. Mesh is always the fallback.

SFU Forward-Compatibility (not yet available)

The mesh API is deliberately shaped so an SFU can be introduced later with no application-code change and no envelope change (§18.6). These are seams, not current features:

• Declarative scope, not imperative connections. share({ session | to | topic }) is audience intent. In mesh the SDK opens peer connections and enforces membership client-side; under an SFU the same call opens one upstream connection and the SFU enforces the ACL and fans out. App code is byte-for-byte identical.
• Reserved quality/selection fields. share(stream, { simulcast }) and (future) subscribe(topic, { quality }) are accepted and ignored in mesh, so introducing them later is not an API break.
• Endpoint-neutral signaling. The RTC offer/answer/ICE flow already works against any endpoint that speaks it — a peer or an SFU node — so the signaling target is treated as an abstract endpoint id/role.
• Capability negotiation. welcome.media.modes advertises topologies; adding "sfu" and a non-null sfu descriptor is purely additive.

Mesh audience/topic membership is enforced client-side by the publishing client — only as strong as that client. Applications requiring authoritative media access control must use SFU mode once it is available (§18.8).

Other SDKs

The media plane is TypeScript-first. Python, Swift, Go, and JVM SDKs will gain a same-shaped API (native MediaStream/track types, language-native reactive receive) once each acquires WebRTC support. Until then, only the TypeScript SDK exposes client.media.