CyberVerse: Build Self-Hosted Voice-First AI Agents

Overview

CyberVerse is a framework for building voice-first AI agents that communicate over WebRTC. It combines conversation handling with persistent persona memory, tools, and RAG, and can optionally pair an agent with a real-time digital-human avatar.

It is aimed at developers who want to assemble and operate an interactive agent platform rather than adopt a hosted chatbot service. Voice-only operation is supported, while avatar video adds a more demanding local inference setup or reliance on supported cloud providers.

Key Features

• Real-time, full-duplex voice interaction over WebRTC, including interruption during agent speech.
• Persistent character conversation history and retrieval from imported knowledge and biographical material.
• Foreground PersonaAgent conversations with asynchronous SubAgents for longer tasks such as research and summarization.
• Replaceable modules for language models, speech, hearing, tools, memory, and avatar video.
• Optional digital-human video using local FlashHead or LiveAct models, or supported cloud services.
• Web settings for configuring provider keys and endpoints, with a LiteLLM plugin for access to multiple LLM providers.

Use Cases

• Developers building a self-hosted, voice-first assistant that needs to remember past conversations.
• Teams creating character-based agents grounded in documents, biographies, or other imported knowledge.
• Prototypers exploring real-time video conversations with a digital avatar, when suitable GPU capacity or a cloud avatar provider is available.
• Engineers testing an agent architecture where long-running research tasks should not block spoken conversation.

Project Facts

• Language: Python
• License: GPL-3.0
• Stars: 1.7k
• Forks: 227
• Topics: ai-agents, ai-companion, digital-human, digital-life, jarvis-assistant, lip-sync, streaming, talking-avatar, talking-head, voice-agent, voice-assistant, webrtc
• Archived: No

Getting Started

Clone the repository and follow the README for environment setup, configuration, and service startup:

git clone https://github.com/Lynpoint/CyberVerse.git
cd CyberVerse

The project also offers a voice-only mode that disables avatar inference. See the README for prerequisites and deployment details.

Alternatives

• openai-agents-js: OpenAI Agents JS supports low-latency voice agents and tool delegation, but is a TypeScript SDK without CyberVerse’s digital-human video focus.

Considerations

• Local setup spans Python, Node, Go, Conda, FFmpeg, and system audio libraries, so this is a multi-service deployment rather than a single-command application.
• Pure voice mode does not require a local avatar GPU, but local digital-human video requires CUDA-capable hardware, model weights, and additional setup. Performance depends on the selected model and hardware.
• Cloud voice and avatar providers may require their own credentials and incur provider costs.
• The GPL-3.0 license should be reviewed before incorporating the project into a product or distribution.
• The README describes supported integrations and setup, but does not establish a universal performance guarantee across hardware and providers.