add docs, protocol, plugins, README, CHANGELOG

This commit is contained in:
loki5512344 2026-07-09 12:30:55 +02:00
parent 1d2209bfac
commit f262da222b
Signed by: boba
GPG key ID: 253067914055423B
44 changed files with 9269 additions and 0 deletions

99
docs/05-features/video.md Normal file
View file

@ -0,0 +1,99 @@
# Video Chat — Architecture
## Goal
Add real-time video to voice channels.
Users can share their webcam feed alongside voice.
## Architecture
```
Current (voice only):
├─ Gateway: TCP (signaling, chat)
├─ Voice-node: UDP (Opus voice relay)
└─ Client: mic → Opus → UDP → voice-node → UDP → client → playback
With video:
├─ Gateway: TCP (signaling) — UNCHANGED
├─ Voice-node: UDP (Opus voice) — UNCHANGED
├─ Video-node: UDP (H.264/VP9 relay) — NEW
│ └─ Receives encoded frames, relays to channel members
└─ Client: mic → Opus → UDP → voice-node
webcam → H.264 → UDP → video-node ← UDP → decode → display grid
```
## Video-node
New optional component, separate binary or embedded in voice-node.
### Responsibilities
- Receive encoded video frames over UDP
- Relay to all other clients in the same voice channel
- No transcoding (relay only — CPU efficient)
- Max resolution / bitrate per channel configurable
### Packet format
```json
{
"packet_id": "VIDEO_FRAME",
"channel_id": 12345,
"sender_id": "<pubkey>",
"frame_seq": 42,
"codec": "h264", // or "vp9"
"keyframe": false,
"data": "<base64 encoded frame>"
}
```
Initially JSON (matching Phase 1 convention), binary framing in Phase 2.
## Client capture pipeline
New file: `client/src/video/capture.rs`
1. Enumerate webcam devices via `nokhwa` or `video4linux`
2. Capture frames at configurable resolution (720p default)
3. Encode to H.264 via `ffmpeg-next` or hardware encoder
4. Packetize into MTU-friendly chunks
5. Send over UDP to video-node
### Dependencies
- `nokhwa` — cross-platform camera capture (Rust)
- `ffmpeg-next` or `rav1e` — H.264/VP9 encoding
## Client UI — Video grid
New component: `client/src/ui/video.rs`
- Grid layout (max 4×4 = 16 participants visible)
- Active speaker highlight (green border)
- Self-view (small, picture-in-picture corner)
- Mute video button per participant
- Resolution/quality indicator per stream
Layout modes:
- 1 participant → full width
- 2-4 → 2×2 grid
- 5-9 → 3×3 grid
- 10-16 → 4×4 grid with scrolling
## Implementation phases
### Phase 1 (MVP)
- H.264 encoding with `nokhwa` + `ffmpeg-next`
- Single video-node binary
- 2×2 grid in UI
- 720p max resolution, 30 fps
### Phase 2
- VP9 support (better quality/bitrate)
- Adaptive resolution (auto downscale on packet loss)
- Picture-in-picture self-view
- Screen sharing (desktop capture)
### Phase 3
- Hardware encoding (NVENC/VAAPI)
- Simulcast (different resolution per stream)
- Recording support