VNOX-Client/docs/04-clients/screen-capture.md
2026-10-24 16:51:15 +02:00

80 lines
4.9 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Local screen capture and separate audio
The voice view's **Screen preview** opens a native window. Choose an optional
playback application or output device, then **Choose screen or window**. The
operating system's ScreenCast portal asks which source may be recorded. **Stop**,
closing the preview, and cancelling selection revoke capture. Preview pixels are
held only in memory, with one latest RGBA frame; they do not enter chat history,
preferences, files, or logs. Microphone capture remains entirely in the existing
voice pipeline and is never enabled by preview.
## Current capability
Linux Wayland capture uses xdg-desktop-portal + PipeWire + GStreamer. Video is
limited to 1920×1080/30fps, copied out of PipeWire buffers, and displayed natively.
The application selector lists active `Stream/Output/Audio` playback streams;
applications with several playback streams may have several entries. It does not
list microphones or VNOX's own streams. The system-output choice is explicit and
may include every application and notification playing on that output.
Selected audio uses a **separate local PipeWire connection**, never the portal's
restricted video FD. Its stable `object.serial` is checked against a fresh graph
before capture. `target-object`, `node.dont-fallback`, `node.dont-reconnect`,
`node.dont-move` and `on-disconnect=error` prevent silently selecting another
stream, microphone or default output. The UI displays actual captured audio
level. Session teardown stops both pipelines before closing the portal FD.
## Broadcast and watch
After the local preview is visible, **Start broadcast** explicitly announces it
in the authenticated voice channel. The server checks current voice/guild
membership and voice-speaking permissions. Other members choose **Watch
broadcast**; receiving never starts automatically. A separate native viewer
renders actual decoded frames and can mute only the broadcast. Closing the
viewer revokes its subscription and clears queued samples. Broadcast audio uses
the same selected CPAL output device and deafen/volume controls as voice; it does
not silently open the default output. Stereo application audio stays stereo.
GStreamer `webrtcbin` negotiates ICE + DTLS-SRTP, VP8 video and independent Opus
tracks. Authenticated LNEx protobuf carries only bounded SDP/ICE signalling.
One publisher has at most four viewers; packets never use custom crypto or video
fragments. Existing voice remains an independent microphone lane by default,
respecting mute/PTT/VAD; broadcasting does not duplicate it. The transport also
supports an explicitly enabled separate microphone track through the gated,
bounded existing-capture tap for integrations that need it.
The default uses host ICE candidates, suitable for reachable LAN/public-address
peers. Administrators may explicitly set `VNOX_STUN_SERVER=stun://host:port` and
`VNOX_TURN_SERVER=turns://user:password@host:port` in the client environment.
Some NATs require TURN; no public STUN server, third-party relay or system-audio
fallback is silently selected. ICE failure times out visibly. TURN credentials
are configuration secrets and must not be committed. Windows process-loopback
and macOS capture adapters remain unsupported; those builds report capability
rather than recording a different source.
## Dependencies and verification
`screen-capture` is enabled by default. A build without it remains available with
`cargo run --no-default-features`. Linux needs GStreamer core/base development
packages the GStreamer WebRTC, libnice, VP8/Opus plugins and the PipeWire GStreamer plugin (`gst-plugin-pipewire` on Arch;
`gstreamer1.0-pipewire` on Debian/Ubuntu), a ScreenCast-capable portal backend,
PipeWire and `pw-dump`. Installation must not rewrite user audio configuration.
```sh
cargo test --lib media:: # Includes actual two-peer VP8 + separate Opus delivery
VNOX_GATEWAY_BIN=/path/to/vnox-serverd cargo test --lib media::gateway_tests -- --ignored
python3 scripts/check-selected-audio.py # Temporary null sink + synthetic app, checks disconnect
cargo run --bin screen_probe -- --list-audio # Metadata only
cargo run --bin screen_probe -- --choose # Explicit portal consent, 15s local capture, no files
```
The Python integration check requires `pactl`, `gst-launch-1.0`, the GStreamer
`level` element and a running session manager. It creates and removes only its
own temporary null sink; it does not change the default device. Manual consent
and real compositor preview still require an interactive desktop check. The
two-peer transport fixture uses only synthetic video and distinct 440/880 Hz
audio, decodes both independently, and never opens microphone/output hardware.
Primary contracts: [ScreenCast portal](https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.ScreenCast.html),
[PipeWire properties](https://docs.pipewire.org/group__pw__keys.html),
[GStreamer AppSink](https://docs.rs/gstreamer-app/0.25.2/gstreamer_app/struct.AppSink.html).