meeting-avatar

Meeting Avatar — a real-person avatar that attends live meetings

A production-oriented, fully pre-trained stack: a photoreal lip-synced avatar of a real person with a cloned voice and an LLM brain, served over WebRTC, ready to bridge into Zoom / Google Meet / Teams / LiveKit.

Built on OpenAvatarChat (pinned commit 8b7b3b4, Apache-2.0) with a deployment-focused Docker image, configs, and meeting-bridge documentation.

Live demo Space: red-kit/meeting-avatar-demo (requires l40s-class Space hardware — see below).

Pipeline

browser / meeting bridge (WebRTC)
  └─ Silero VAD → SenseVoice ASR (local)
       └─ LLM brain (any OpenAI-compatible API)
            └─ CosyVoice2-0.5B zero-shot voice clone (local, streaming)
                 └─ MuseTalk real-time lip-sync over the person's footage
                      └─ avatar video+audio back over WebRTC

End-to-end speech-to-speech latency: ~0.8–2.2 s on one A10G-class GPU (upstream reports 2.2 s avg on RTX 4090). One concurrent session per GPU with the MuseTalk config; use config/meeting_avatar_lite.yaml for lighter hardware.

Verified on GPU

End-to-end boot smoke-tested on Hugging Face Jobs (NVIDIA A10G, CUDA 12.8): system deps → pinned clone → install → weights → boot → HTTP 200 + CosyVoice2 zero-shot synthesis on GPU (9 min, exit 0; handler load 22.5 s, warmup synthesis rtf 1.68, both TTS processors live). Reproducible via scripts/smoke_test.sh. Verification surfaced and fixed four real deployment bugs, all now baked into the Dockerfile: a typeguard/typing_extensions pin conflict, a pyarrow/NumPy wheel conflict, a stale upstream voice-asset path, and an upstream CosyVoice warmup-call signature bug (patched at build time).

Also verified as a Docker Space (see the demo Space): the image builds and the stack boots on an HF l40sSpace (48 GB VRAM, 62 GB system RAM). An a10g-small Space (24 GB VRAM but 15 GB system RAM) is OOM-killed at boot — the stack needs ~24 GB VRAM and >15 GB system RAM. Use l40s or a10g-large (46 GB RAM) hardware, or a 32 GB+ RAM GPU box.

Deploy

Option 1 — any CUDA 12.8 host (simplest)

git clone https://huggingface.co/redkits/meeting-avatar && cd meeting-avatar
echo "DASHSCOPE_API_KEY=sk-your-openai-compatible-key" > .env
docker compose up --build   # then open http://localhost:7860

Requires ~24 GB VRAM and ≥46 GB system RAM (A10G-large / L40S / RTX 3090 or 4090 with 32+ GB RAM). Image is ~20 GB — all weights are baked at build time; restarts never re-download.

Option 2 — Hugging Face Space

The Dockerfile is Space-ready (port 7860, non-root user). Creating a Docker Space requires a paid plan (PRO, or an org on Team/Enterprise). Once created (or by duplicating the demo Space):

1. Create a Space (Docker SDK), copy this repo's files into it.
2. Settings → Secrets → add DASHSCOPE_API_KEY (your LLM API key — the env var name is an upstream default, any OpenAI-compatible key works).
3. Settings → Hardware → attach l40s (48 GB VRAM, 62 GB RAM, $1.80/h) or a10g-large(46 GB RAM, $1.50/h).Not a10g-small — its 15 GB system RAM gets the stack OOM-killed at boot (verified).
4. Open the Space and start a conversation.

Option 3 — HF Inference Endpoints

Inference Endpoints are request/response HTTP — they do not fit a long-lived WebRTC session. Use them later for a headless render API if you outgrow the single-GPU Space; the interactive avatar itself wants a persistent container (Options 1–2).

Make it your person (no training required)

Two files, both specified in docs/footage-collection-spec.md:

• Face: 1–3 min static-camera clip of the person → set AvatarMusetalk.avatar_video_path in config/meeting_avatar.yaml.
• Voice: 5–10 s clean recording + exact transcript → set CosyVoice.ref_audio_path / ref_audio_text.

Until real footage exists, the stack runs on bundled stand-in assets. For the photoreal LoRA upgrade path (Wan-S2V person LoRA), the collection spec includes what to record for that too.

Getting into meetings

See docs/meeting-bridges.md: OBS virtual camera (zero-code, universal), Recall.ai output-media bot (production, Zoom/Meet/Teams), or LiveKit native participant (cleanest architecture). Includes the latency budget and the honest list of what nobody has solved yet.

Compliance — read before first use

Cloning a real person's face and voice requires their written consent, and many jurisdictions and meeting platforms require disclosing AI participants. Details in the collection spec. This repo ships stand-in assets only.

Repo layout

Dockerfile                     deployment image (weights baked in)
docker-compose.yml             self-host launcher
config/meeting_avatar.yaml     production config (MuseTalk)
config/meeting_avatar_lite.yaml  lightweight fallback (LiteAvatar)
docs/footage-collection-spec.md  how to collect face + voice assets
docs/meeting-bridges.md        Zoom/Meet/Teams/LiveKit integration routes
scripts/smoke_test.sh          GPU end-to-end boot verification

Upstream credits: OpenAvatarChat, MuseTalk, CosyVoice, Silero VAD, SenseVoice (FunAudioLLM).