add docs, protocol, plugins, README, CHANGELOG
This commit is contained in:
parent
1d2209bfac
commit
f262da222b
44 changed files with 9269 additions and 0 deletions
99
docs/05-features/video.md
Normal file
99
docs/05-features/video.md
Normal file
|
|
@ -0,0 +1,99 @@
|
|||
# Video Chat — Architecture
|
||||
|
||||
## Goal
|
||||
|
||||
Add real-time video to voice channels.
|
||||
Users can share their webcam feed alongside voice.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Current (voice only):
|
||||
├─ Gateway: TCP (signaling, chat)
|
||||
├─ Voice-node: UDP (Opus voice relay)
|
||||
└─ Client: mic → Opus → UDP → voice-node → UDP → client → playback
|
||||
|
||||
With video:
|
||||
├─ Gateway: TCP (signaling) — UNCHANGED
|
||||
├─ Voice-node: UDP (Opus voice) — UNCHANGED
|
||||
├─ Video-node: UDP (H.264/VP9 relay) — NEW
|
||||
│ └─ Receives encoded frames, relays to channel members
|
||||
└─ Client: mic → Opus → UDP → voice-node
|
||||
webcam → H.264 → UDP → video-node ← UDP → decode → display grid
|
||||
```
|
||||
|
||||
## Video-node
|
||||
|
||||
New optional component, separate binary or embedded in voice-node.
|
||||
|
||||
### Responsibilities
|
||||
- Receive encoded video frames over UDP
|
||||
- Relay to all other clients in the same voice channel
|
||||
- No transcoding (relay only — CPU efficient)
|
||||
- Max resolution / bitrate per channel configurable
|
||||
|
||||
### Packet format
|
||||
|
||||
```json
|
||||
{
|
||||
"packet_id": "VIDEO_FRAME",
|
||||
"channel_id": 12345,
|
||||
"sender_id": "<pubkey>",
|
||||
"frame_seq": 42,
|
||||
"codec": "h264", // or "vp9"
|
||||
"keyframe": false,
|
||||
"data": "<base64 encoded frame>"
|
||||
}
|
||||
```
|
||||
|
||||
Initially JSON (matching Phase 1 convention), binary framing in Phase 2.
|
||||
|
||||
## Client capture pipeline
|
||||
|
||||
New file: `client/src/video/capture.rs`
|
||||
|
||||
1. Enumerate webcam devices via `nokhwa` or `video4linux`
|
||||
2. Capture frames at configurable resolution (720p default)
|
||||
3. Encode to H.264 via `ffmpeg-next` or hardware encoder
|
||||
4. Packetize into MTU-friendly chunks
|
||||
5. Send over UDP to video-node
|
||||
|
||||
### Dependencies
|
||||
|
||||
- `nokhwa` — cross-platform camera capture (Rust)
|
||||
- `ffmpeg-next` or `rav1e` — H.264/VP9 encoding
|
||||
|
||||
## Client UI — Video grid
|
||||
|
||||
New component: `client/src/ui/video.rs`
|
||||
|
||||
- Grid layout (max 4×4 = 16 participants visible)
|
||||
- Active speaker highlight (green border)
|
||||
- Self-view (small, picture-in-picture corner)
|
||||
- Mute video button per participant
|
||||
- Resolution/quality indicator per stream
|
||||
|
||||
Layout modes:
|
||||
- 1 participant → full width
|
||||
- 2-4 → 2×2 grid
|
||||
- 5-9 → 3×3 grid
|
||||
- 10-16 → 4×4 grid with scrolling
|
||||
|
||||
## Implementation phases
|
||||
|
||||
### Phase 1 (MVP)
|
||||
- H.264 encoding with `nokhwa` + `ffmpeg-next`
|
||||
- Single video-node binary
|
||||
- 2×2 grid in UI
|
||||
- 720p max resolution, 30 fps
|
||||
|
||||
### Phase 2
|
||||
- VP9 support (better quality/bitrate)
|
||||
- Adaptive resolution (auto downscale on packet loss)
|
||||
- Picture-in-picture self-view
|
||||
- Screen sharing (desktop capture)
|
||||
|
||||
### Phase 3
|
||||
- Hardware encoding (NVENC/VAAPI)
|
||||
- Simulcast (different resolution per stream)
|
||||
- Recording support
|
||||
Loading…
Add table
Add a link
Reference in a new issue