VNOX-Client/docs/04-clients/screen-capture.md

3.6 KiB
Raw Blame History

Local screen capture and separate audio

The voice view's Screen preview opens a native window. Choose an optional playback application or output device, then Choose screen or window. The operating system's ScreenCast portal asks which source may be recorded. Stop, closing the preview, and cancelling selection revoke capture. Preview pixels are held only in memory, with one latest RGBA frame; they do not enter chat history, preferences, files, or logs. Microphone capture remains entirely in the existing voice pipeline and is never enabled by preview.

Current capability

Linux Wayland capture uses xdg-desktop-portal + PipeWire + GStreamer. Video is limited to 1920×1080/30fps, copied out of PipeWire buffers, and displayed natively. The application selector lists active Stream/Output/Audio playback streams; applications with several playback streams may have several entries. It does not list microphones or VNOX's own streams. The system-output choice is explicit and may include every application and notification playing on that output.

Selected audio uses a separate local PipeWire connection, never the portal's restricted video FD. Its stable object.serial is checked against a fresh graph before capture. target-object, node.dont-fallback, node.dont-reconnect, node.dont-move and on-disconnect=error prevent silently selecting another stream, microphone or default output. The UI displays actual captured audio level. Session teardown stops both pipelines before closing the portal FD.

This is local preview, not broadcasting. The existing client/server media protocol does not yet negotiate separate video/application-audio tracks, codec capabilities or recipient subscriptions. No frames are sent to other members. The next integration must use established WebRTC transport with authenticated signalling, independent tracks, backpressure and participant authorization in the combined server daemon. Do not send video through custom UDP fragments or reuse the microphone encryption/packet format without a documented protocol change. Windows process-loopback and macOS adapters are not implemented; unsupported builds report that capability rather than recording a different source.

Dependencies and verification

screen-capture is enabled by default. A build without it remains available with cargo run --no-default-features. Linux needs GStreamer core/base development packages and the PipeWire GStreamer plugin (gst-plugin-pipewire on Arch; gstreamer1.0-pipewire on Debian/Ubuntu), a ScreenCast-capable portal backend, PipeWire and pw-dump. Installation must not rewrite user audio configuration.

cargo test --lib media::                      # Synthetic video/audio; no desktop recording
python3 scripts/check-selected-audio.py       # Temporary null sink + synthetic app, checks disconnect
cargo run --bin screen_probe -- --list-audio  # Metadata only
cargo run --bin screen_probe -- --choose      # Explicit portal consent, 15s local capture, no files

The Python integration check requires pactl, gst-launch-1.0, the GStreamer level element and a running session manager. It creates and removes only its own temporary null sink; it does not change the default device. Manual consent and real compositor preview still require an interactive desktop check.

Primary contracts: ScreenCast portal, PipeWire properties, GStreamer AppSink.