3.6 KiB
Local screen capture and separate audio
The voice view's Screen preview opens a native window. Choose an optional playback application or output device, then Choose screen or window. The operating system's ScreenCast portal asks which source may be recorded. Stop, closing the preview, and cancelling selection revoke capture. Preview pixels are held only in memory, with one latest RGBA frame; they do not enter chat history, preferences, files, or logs. Microphone capture remains entirely in the existing voice pipeline and is never enabled by preview.
Current capability
Linux Wayland capture uses xdg-desktop-portal + PipeWire + GStreamer. Video is
limited to 1920×1080/30fps, copied out of PipeWire buffers, and displayed natively.
The application selector lists active Stream/Output/Audio playback streams;
applications with several playback streams may have several entries. It does not
list microphones or VNOX's own streams. The system-output choice is explicit and
may include every application and notification playing on that output.
Selected audio uses a separate local PipeWire connection, never the portal's
restricted video FD. Its stable object.serial is checked against a fresh graph
before capture. target-object, node.dont-fallback, node.dont-reconnect,
node.dont-move and on-disconnect=error prevent silently selecting another
stream, microphone or default output. The UI displays actual captured audio
level. Session teardown stops both pipelines before closing the portal FD.
This is local preview, not broadcasting. The existing client/server media protocol does not yet negotiate separate video/application-audio tracks, codec capabilities or recipient subscriptions. No frames are sent to other members. The next integration must use established WebRTC transport with authenticated signalling, independent tracks, backpressure and participant authorization in the combined server daemon. Do not send video through custom UDP fragments or reuse the microphone encryption/packet format without a documented protocol change. Windows process-loopback and macOS adapters are not implemented; unsupported builds report that capability rather than recording a different source.
Dependencies and verification
screen-capture is enabled by default. A build without it remains available with
cargo run --no-default-features. Linux needs GStreamer core/base development
packages and the PipeWire GStreamer plugin (gst-plugin-pipewire on Arch;
gstreamer1.0-pipewire on Debian/Ubuntu), a ScreenCast-capable portal backend,
PipeWire and pw-dump. Installation must not rewrite user audio configuration.
cargo test --lib media:: # Synthetic video/audio; no desktop recording
python3 scripts/check-selected-audio.py # Temporary null sink + synthetic app, checks disconnect
cargo run --bin screen_probe -- --list-audio # Metadata only
cargo run --bin screen_probe -- --choose # Explicit portal consent, 15s local capture, no files
The Python integration check requires pactl, gst-launch-1.0, the GStreamer
level element and a running session manager. It creates and removes only its
own temporary null sink; it does not change the default device. Manual consent
and real compositor preview still require an interactive desktop check.
Primary contracts: ScreenCast portal, PipeWire properties, GStreamer AppSink.