feat(client): protocol v2 wire layer, screen sharing, channel webhooks and secure voice
Verified against the real gateway via tests/gateway_interop.rs.
This commit is contained in:
parent
8460fb28fd
commit
3d7dd5413b
65 changed files with 6048 additions and 1156 deletions
|
|
@ -24,26 +24,45 @@ before capture. `target-object`, `node.dont-fallback`, `node.dont-reconnect`,
|
|||
stream, microphone or default output. The UI displays actual captured audio
|
||||
level. Session teardown stops both pipelines before closing the portal FD.
|
||||
|
||||
**This is local preview, not broadcasting.** The existing client/server media
|
||||
protocol does not yet negotiate separate video/application-audio tracks, codec
|
||||
capabilities or recipient subscriptions. No frames are sent to other members.
|
||||
The next integration must use established WebRTC transport with authenticated
|
||||
signalling, independent tracks, backpressure and participant authorization in the
|
||||
combined server daemon. Do not send video through custom UDP fragments or reuse
|
||||
the microphone encryption/packet format without a documented protocol change.
|
||||
Windows process-loopback and macOS adapters are not implemented; unsupported
|
||||
builds report that capability rather than recording a different source.
|
||||
## Broadcast and watch
|
||||
|
||||
After the local preview is visible, **Start broadcast** explicitly announces it
|
||||
in the authenticated voice channel. The server checks current voice/guild
|
||||
membership and voice-speaking permissions. Other members choose **Watch
|
||||
broadcast**; receiving never starts automatically. A separate native viewer
|
||||
renders actual decoded frames and can mute only the broadcast. Closing the
|
||||
viewer revokes its subscription and clears queued samples. Broadcast audio uses
|
||||
the same selected CPAL output device and deafen/volume controls as voice; it does
|
||||
not silently open the default output. Stereo application audio stays stereo.
|
||||
|
||||
GStreamer `webrtcbin` negotiates ICE + DTLS-SRTP, VP8 video and independent Opus
|
||||
tracks. Authenticated LNEx protobuf carries only bounded SDP/ICE signalling.
|
||||
One publisher has at most four viewers; packets never use custom crypto or video
|
||||
fragments. Existing voice remains an independent microphone lane by default,
|
||||
respecting mute/PTT/VAD; broadcasting does not duplicate it. The transport also
|
||||
supports an explicitly enabled separate microphone track through the gated,
|
||||
bounded existing-capture tap for integrations that need it.
|
||||
|
||||
The default uses host ICE candidates, suitable for reachable LAN/public-address
|
||||
peers. Administrators may explicitly set `VNOX_STUN_SERVER=stun://host:port` and
|
||||
`VNOX_TURN_SERVER=turns://user:password@host:port` in the client environment.
|
||||
Some NATs require TURN; no public STUN server, third-party relay or system-audio
|
||||
fallback is silently selected. ICE failure times out visibly. TURN credentials
|
||||
are configuration secrets and must not be committed. Windows process-loopback
|
||||
and macOS capture adapters remain unsupported; those builds report capability
|
||||
rather than recording a different source.
|
||||
|
||||
## Dependencies and verification
|
||||
|
||||
`screen-capture` is enabled by default. A build without it remains available with
|
||||
`cargo run --no-default-features`. Linux needs GStreamer core/base development
|
||||
packages and the PipeWire GStreamer plugin (`gst-plugin-pipewire` on Arch;
|
||||
packages the GStreamer WebRTC, libnice, VP8/Opus plugins and the PipeWire GStreamer plugin (`gst-plugin-pipewire` on Arch;
|
||||
`gstreamer1.0-pipewire` on Debian/Ubuntu), a ScreenCast-capable portal backend,
|
||||
PipeWire and `pw-dump`. Installation must not rewrite user audio configuration.
|
||||
|
||||
```sh
|
||||
cargo test --lib media:: # Synthetic video/audio; no desktop recording
|
||||
cargo test --lib media:: # Includes actual two-peer VP8 + separate Opus delivery
|
||||
VNOX_GATEWAY_BIN=/path/to/vnox-serverd cargo test --lib media::gateway_tests -- --ignored
|
||||
python3 scripts/check-selected-audio.py # Temporary null sink + synthetic app, checks disconnect
|
||||
cargo run --bin screen_probe -- --list-audio # Metadata only
|
||||
cargo run --bin screen_probe -- --choose # Explicit portal consent, 15s local capture, no files
|
||||
|
|
@ -52,7 +71,9 @@ cargo run --bin screen_probe -- --choose # Explicit portal consent, 15s loc
|
|||
The Python integration check requires `pactl`, `gst-launch-1.0`, the GStreamer
|
||||
`level` element and a running session manager. It creates and removes only its
|
||||
own temporary null sink; it does not change the default device. Manual consent
|
||||
and real compositor preview still require an interactive desktop check.
|
||||
and real compositor preview still require an interactive desktop check. The
|
||||
two-peer transport fixture uses only synthetic video and distinct 440/880 Hz
|
||||
audio, decodes both independently, and never opens microphone/output hardware.
|
||||
|
||||
Primary contracts: [ScreenCast portal](https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.ScreenCast.html),
|
||||
[PipeWire properties](https://docs.pipewire.org/group__pw__keys.html),
|
||||
|
|
|
|||
19
docs/maintenance/protocol-interop.md
Normal file
19
docs/maintenance/protocol-interop.md
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
# Native client and gateway interoperability
|
||||
|
||||
The client uses LNEx v2 protobuf payloads generated from `protocol/lnex.proto`. This is a verbatim snapshot of the gateway's canonical schema; run `python3 scripts/sync-protocol.py --check` with the adjacent server checkout, or `--server /path/to/lnex.proto`. The client builds independently using its checked-in snapshot. Internal payload aliases exist for application compatibility, but no JSON is sent on LNEx.
|
||||
|
||||
Remote addresses use verified TLS 1.3, including certificate chain and hostname checks. Use `lnexs://host:7600` explicitly; bare remote hosts also use TLS. Development plaintext is restricted to actual connected loopback peers (`127.0.0.1:7600`, `[::1]:7600`, or explicit `lnex://localhost:7600`). Certificate failures never trigger a plaintext fallback. Self-signed remote certificates need a separately designed explicit trust workflow; verification is never bypassed.
|
||||
|
||||
The v2 handshake verifies the server's signed HELLO and binds the client's identity signature to the complete HELLO and AUTH transcript, including both ephemeral X25519 keys. Control encryption retains ChaCha20-Poly1305 and HKDF-SHA256 with the v2 domain; it authenticates the entire 12-byte header. Sequence numbers are checked in order and never wrap. A dedicated bounded reader keeps partial frames intact while other commands and timers run and is cancelled when its session closes.
|
||||
|
||||
Voice receives independent random credentials through encrypted control (`VoiceSession`, `0x0012`). UDP authenticates its full 28-byte header, explicit 64-bit sequence and protobuf frame, uses direction-specific keys, and permits bounded reordering without accepting duplicates. The receiver checks the selected relay endpoint and authenticated user/channel attribution. Empty encrypted Opus frames register the source and maintain it every five seconds. Leaving or disconnecting drops credentials and stops the receiver task. This replaces the incompatible legacy encrypted-guess-sequence client/plaintext relay combination.
|
||||
|
||||
Run focused verification:
|
||||
|
||||
```sh
|
||||
cargo test --lib --no-default-features
|
||||
VNOX_GATEWAY_BIN=/path/to/current/vnox-gateway cargo test --test gateway_interop --no-default-features -- --ignored
|
||||
cargo run --bin vnox-e2e-voice --no-default-features -- 127.0.0.1:7600
|
||||
```
|
||||
|
||||
The ignored interoperability test launches its own loopback gateway and temporary database using an in-memory identity, without reading or writing the user's profile. The voice smoke command requires the combined `vnox-serverd` daemon so gateway credentials reach the UDP relay through the in-process bridge.
|
||||
Loading…
Add table
Add a link
Reference in a new issue