VNOX-Client/docs/04-clients/screen-capture.md
2026-10-24 16:51:15 +02:00

4.9 KiB
Raw Permalink Blame History

Local screen capture and separate audio

The voice view's Screen preview opens a native window. Choose an optional playback application or output device, then Choose screen or window. The operating system's ScreenCast portal asks which source may be recorded. Stop, closing the preview, and cancelling selection revoke capture. Preview pixels are held only in memory, with one latest RGBA frame; they do not enter chat history, preferences, files, or logs. Microphone capture remains entirely in the existing voice pipeline and is never enabled by preview.

Current capability

Linux Wayland capture uses xdg-desktop-portal + PipeWire + GStreamer. Video is limited to 1920×1080/30fps, copied out of PipeWire buffers, and displayed natively. The application selector lists active Stream/Output/Audio playback streams; applications with several playback streams may have several entries. It does not list microphones or VNOX's own streams. The system-output choice is explicit and may include every application and notification playing on that output.

Selected audio uses a separate local PipeWire connection, never the portal's restricted video FD. Its stable object.serial is checked against a fresh graph before capture. target-object, node.dont-fallback, node.dont-reconnect, node.dont-move and on-disconnect=error prevent silently selecting another stream, microphone or default output. The UI displays actual captured audio level. Session teardown stops both pipelines before closing the portal FD.

Broadcast and watch

After the local preview is visible, Start broadcast explicitly announces it in the authenticated voice channel. The server checks current voice/guild membership and voice-speaking permissions. Other members choose Watch broadcast; receiving never starts automatically. A separate native viewer renders actual decoded frames and can mute only the broadcast. Closing the viewer revokes its subscription and clears queued samples. Broadcast audio uses the same selected CPAL output device and deafen/volume controls as voice; it does not silently open the default output. Stereo application audio stays stereo.

GStreamer webrtcbin negotiates ICE + DTLS-SRTP, VP8 video and independent Opus tracks. Authenticated LNEx protobuf carries only bounded SDP/ICE signalling. One publisher has at most four viewers; packets never use custom crypto or video fragments. Existing voice remains an independent microphone lane by default, respecting mute/PTT/VAD; broadcasting does not duplicate it. The transport also supports an explicitly enabled separate microphone track through the gated, bounded existing-capture tap for integrations that need it.

The default uses host ICE candidates, suitable for reachable LAN/public-address peers. Administrators may explicitly set VNOX_STUN_SERVER=stun://host:port and VNOX_TURN_SERVER=turns://user:password@host:port in the client environment. Some NATs require TURN; no public STUN server, third-party relay or system-audio fallback is silently selected. ICE failure times out visibly. TURN credentials are configuration secrets and must not be committed. Windows process-loopback and macOS capture adapters remain unsupported; those builds report capability rather than recording a different source.

Dependencies and verification

screen-capture is enabled by default. A build without it remains available with cargo run --no-default-features. Linux needs GStreamer core/base development packages the GStreamer WebRTC, libnice, VP8/Opus plugins and the PipeWire GStreamer plugin (gst-plugin-pipewire on Arch; gstreamer1.0-pipewire on Debian/Ubuntu), a ScreenCast-capable portal backend, PipeWire and pw-dump. Installation must not rewrite user audio configuration.

cargo test --lib media::                      # Includes actual two-peer VP8 + separate Opus delivery
VNOX_GATEWAY_BIN=/path/to/vnox-serverd cargo test --lib media::gateway_tests -- --ignored
python3 scripts/check-selected-audio.py       # Temporary null sink + synthetic app, checks disconnect
cargo run --bin screen_probe -- --list-audio  # Metadata only
cargo run --bin screen_probe -- --choose      # Explicit portal consent, 15s local capture, no files

The Python integration check requires pactl, gst-launch-1.0, the GStreamer level element and a running session manager. It creates and removes only its own temporary null sink; it does not change the default device. Manual consent and real compositor preview still require an interactive desktop check. The two-peer transport fixture uses only synthetic video and distinct 440/880 Hz audio, decodes both independently, and never opens microphone/output hardware.

Primary contracts: ScreenCast portal, PipeWire properties, GStreamer AppSink.