59 lines
3.6 KiB
Markdown
59 lines
3.6 KiB
Markdown
# Local screen capture and separate audio
|
||
|
||
The voice view's **Screen preview** opens a native window. Choose an optional
|
||
playback application or output device, then **Choose screen or window**. The
|
||
operating system's ScreenCast portal asks which source may be recorded. **Stop**,
|
||
closing the preview, and cancelling selection revoke capture. Preview pixels are
|
||
held only in memory, with one latest RGBA frame; they do not enter chat history,
|
||
preferences, files, or logs. Microphone capture remains entirely in the existing
|
||
voice pipeline and is never enabled by preview.
|
||
|
||
## Current capability
|
||
|
||
Linux Wayland capture uses xdg-desktop-portal + PipeWire + GStreamer. Video is
|
||
limited to 1920×1080/30fps, copied out of PipeWire buffers, and displayed natively.
|
||
The application selector lists active `Stream/Output/Audio` playback streams;
|
||
applications with several playback streams may have several entries. It does not
|
||
list microphones or VNOX's own streams. The system-output choice is explicit and
|
||
may include every application and notification playing on that output.
|
||
|
||
Selected audio uses a **separate local PipeWire connection**, never the portal's
|
||
restricted video FD. Its stable `object.serial` is checked against a fresh graph
|
||
before capture. `target-object`, `node.dont-fallback`, `node.dont-reconnect`,
|
||
`node.dont-move` and `on-disconnect=error` prevent silently selecting another
|
||
stream, microphone or default output. The UI displays actual captured audio
|
||
level. Session teardown stops both pipelines before closing the portal FD.
|
||
|
||
**This is local preview, not broadcasting.** The existing client/server media
|
||
protocol does not yet negotiate separate video/application-audio tracks, codec
|
||
capabilities or recipient subscriptions. No frames are sent to other members.
|
||
The next integration must use established WebRTC transport with authenticated
|
||
signalling, independent tracks, backpressure and participant authorization in the
|
||
combined server daemon. Do not send video through custom UDP fragments or reuse
|
||
the microphone encryption/packet format without a documented protocol change.
|
||
Windows process-loopback and macOS adapters are not implemented; unsupported
|
||
builds report that capability rather than recording a different source.
|
||
|
||
## Dependencies and verification
|
||
|
||
`screen-capture` is enabled by default. A build without it remains available with
|
||
`cargo run --no-default-features`. Linux needs GStreamer core/base development
|
||
packages and the PipeWire GStreamer plugin (`gst-plugin-pipewire` on Arch;
|
||
`gstreamer1.0-pipewire` on Debian/Ubuntu), a ScreenCast-capable portal backend,
|
||
PipeWire and `pw-dump`. Installation must not rewrite user audio configuration.
|
||
|
||
```sh
|
||
cargo test --lib media:: # Synthetic video/audio; no desktop recording
|
||
python3 scripts/check-selected-audio.py # Temporary null sink + synthetic app, checks disconnect
|
||
cargo run --bin screen_probe -- --list-audio # Metadata only
|
||
cargo run --bin screen_probe -- --choose # Explicit portal consent, 15s local capture, no files
|
||
```
|
||
|
||
The Python integration check requires `pactl`, `gst-launch-1.0`, the GStreamer
|
||
`level` element and a running session manager. It creates and removes only its
|
||
own temporary null sink; it does not change the default device. Manual consent
|
||
and real compositor preview still require an interactive desktop check.
|
||
|
||
Primary contracts: [ScreenCast portal](https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.ScreenCast.html),
|
||
[PipeWire properties](https://docs.pipewire.org/group__pw__keys.html),
|
||
[GStreamer AppSink](https://docs.rs/gstreamer-app/0.25.2/gstreamer_app/struct.AppSink.html).
|