80 lines
4.9 KiB
Markdown
80 lines
4.9 KiB
Markdown
# Local screen capture and separate audio
|
||
|
||
The voice view's **Screen preview** opens a native window. Choose an optional
|
||
playback application or output device, then **Choose screen or window**. The
|
||
operating system's ScreenCast portal asks which source may be recorded. **Stop**,
|
||
closing the preview, and cancelling selection revoke capture. Preview pixels are
|
||
held only in memory, with one latest RGBA frame; they do not enter chat history,
|
||
preferences, files, or logs. Microphone capture remains entirely in the existing
|
||
voice pipeline and is never enabled by preview.
|
||
|
||
## Current capability
|
||
|
||
Linux Wayland capture uses xdg-desktop-portal + PipeWire + GStreamer. Video is
|
||
limited to 1920×1080/30fps, copied out of PipeWire buffers, and displayed natively.
|
||
The application selector lists active `Stream/Output/Audio` playback streams;
|
||
applications with several playback streams may have several entries. It does not
|
||
list microphones or VNOX's own streams. The system-output choice is explicit and
|
||
may include every application and notification playing on that output.
|
||
|
||
Selected audio uses a **separate local PipeWire connection**, never the portal's
|
||
restricted video FD. Its stable `object.serial` is checked against a fresh graph
|
||
before capture. `target-object`, `node.dont-fallback`, `node.dont-reconnect`,
|
||
`node.dont-move` and `on-disconnect=error` prevent silently selecting another
|
||
stream, microphone or default output. The UI displays actual captured audio
|
||
level. Session teardown stops both pipelines before closing the portal FD.
|
||
|
||
## Broadcast and watch
|
||
|
||
After the local preview is visible, **Start broadcast** explicitly announces it
|
||
in the authenticated voice channel. The server checks current voice/guild
|
||
membership and voice-speaking permissions. Other members choose **Watch
|
||
broadcast**; receiving never starts automatically. A separate native viewer
|
||
renders actual decoded frames and can mute only the broadcast. Closing the
|
||
viewer revokes its subscription and clears queued samples. Broadcast audio uses
|
||
the same selected CPAL output device and deafen/volume controls as voice; it does
|
||
not silently open the default output. Stereo application audio stays stereo.
|
||
|
||
GStreamer `webrtcbin` negotiates ICE + DTLS-SRTP, VP8 video and independent Opus
|
||
tracks. Authenticated LNEx protobuf carries only bounded SDP/ICE signalling.
|
||
One publisher has at most four viewers; packets never use custom crypto or video
|
||
fragments. Existing voice remains an independent microphone lane by default,
|
||
respecting mute/PTT/VAD; broadcasting does not duplicate it. The transport also
|
||
supports an explicitly enabled separate microphone track through the gated,
|
||
bounded existing-capture tap for integrations that need it.
|
||
|
||
The default uses host ICE candidates, suitable for reachable LAN/public-address
|
||
peers. Administrators may explicitly set `VNOX_STUN_SERVER=stun://host:port` and
|
||
`VNOX_TURN_SERVER=turns://user:password@host:port` in the client environment.
|
||
Some NATs require TURN; no public STUN server, third-party relay or system-audio
|
||
fallback is silently selected. ICE failure times out visibly. TURN credentials
|
||
are configuration secrets and must not be committed. Windows process-loopback
|
||
and macOS capture adapters remain unsupported; those builds report capability
|
||
rather than recording a different source.
|
||
|
||
## Dependencies and verification
|
||
|
||
`screen-capture` is enabled by default. A build without it remains available with
|
||
`cargo run --no-default-features`. Linux needs GStreamer core/base development
|
||
packages the GStreamer WebRTC, libnice, VP8/Opus plugins and the PipeWire GStreamer plugin (`gst-plugin-pipewire` on Arch;
|
||
`gstreamer1.0-pipewire` on Debian/Ubuntu), a ScreenCast-capable portal backend,
|
||
PipeWire and `pw-dump`. Installation must not rewrite user audio configuration.
|
||
|
||
```sh
|
||
cargo test --lib media:: # Includes actual two-peer VP8 + separate Opus delivery
|
||
VNOX_GATEWAY_BIN=/path/to/vnox-serverd cargo test --lib media::gateway_tests -- --ignored
|
||
python3 scripts/check-selected-audio.py # Temporary null sink + synthetic app, checks disconnect
|
||
cargo run --bin screen_probe -- --list-audio # Metadata only
|
||
cargo run --bin screen_probe -- --choose # Explicit portal consent, 15s local capture, no files
|
||
```
|
||
|
||
The Python integration check requires `pactl`, `gst-launch-1.0`, the GStreamer
|
||
`level` element and a running session manager. It creates and removes only its
|
||
own temporary null sink; it does not change the default device. Manual consent
|
||
and real compositor preview still require an interactive desktop check. The
|
||
two-peer transport fixture uses only synthetic video and distinct 440/880 Hz
|
||
audio, decodes both independently, and never opens microphone/output hardware.
|
||
|
||
Primary contracts: [ScreenCast portal](https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.ScreenCast.html),
|
||
[PipeWire properties](https://docs.pipewire.org/group__pw__keys.html),
|
||
[GStreamer AppSink](https://docs.rs/gstreamer-app/0.25.2/gstreamer_app/struct.AppSink.html).
|