CyberVerse: Self-Hosted Real-Time Digital Human Agent Platform

Introduction

CyberVerse is an innovative, open-source, and self-hosted platform designed for creating real-time digital human agents. It empowers developers to build sophisticated, voice-first AI agents by integrating cutting-edge technologies like WebRTC, persona memory, tools, and Retrieval Augmented Generation (RAG). A key feature is its optional digital-human video integration, enabling highly engaging and lifelike interactions from just a single photo. Imagine bringing your own J.A.R.V.I.S.-like AI to life, capable of seeing, hearing, and speaking with you in real-time.

Why Use CyberVerse and Its Benefits

CyberVerse offers a compelling set of features and advantages for anyone looking to develop advanced AI agents:

Real-time Digital Human Interaction: Create video-call capable digital humans from a single photo. Experience natural, full-duplex conversations with the ability to interrupt, just like a human video call. CyberVerse integrates with local models like FlashHead and LiveAct, and cloud services such as Baidu Xiling and Xunfei Digital Human.
Multi-Agent Architecture: The platform utilizes a sophisticated multi-agent system. A PersonaAgent handles smooth user conversation and quick responses, while SubAgents asynchronously manage complex, time-consuming tasks like research, summarization, or report generation. This ensures that the main conversation flow remains uninterrupted.
Persistent Character Memory and RAG: Each character maintains a persistent conversation history, ensuring continuity across sessions. You can also ingest knowledge bases, documents, and biographical materials for characters, which the system indexes for RAG. This allows for highly contextual and personalized responses aligned with the character's background.
Plugin-Based Stack: CyberVerse is built with a modular design, where components like the brain, voice, hearing, tools, memory, and face are all pluggable modules. This flexibility allows users to easily swap providers and model configurations for LLMs, TTS, ASR, and Embedding services via configuration files or the Web UI.
Flexible Deployment: Get started quickly with pre-built cloud images on platforms like Compshare and AutoDL, or deploy locally with comprehensive setup instructions.

Installation

To get CyberVerse up and running locally, follow these simplified steps. For detailed instructions and troubleshooting, please refer to the official GitHub repository.

Prerequisites

Ensure you have the following installed:

Node 18+
Go 1.25 (with protoc-gen-go, protoc-gen-go-grpc)
Conda
Python 3.10+
FFmpeg
libopus-dev, libopusfile-dev, libsoxr-dev, pkg-config

You can verify installations with:

node --version
go version
protoc --version
ffmpeg -version
conda --version

Steps

Clone the repository:

git clone https://github.com/Lynpoint/CyberVerse.git
cd CyberVerse

Create Python environment:

conda create -n cyberverse python=3.10
conda activate cyberverse

Set environment variables:
```
cp -r infra/config config
```
Edit config/env to add your API keys (e.g., DASHSCOPE_API_KEY, DOUBAO_ACCESS_TOKEN).
Enable voice-only mode (optional, for initial setup):
Edit config/cyberverse.yaml and set inference.avatar.enabled: false if you don't want video inference initially.
Install project dependencies:
```
make setup
pip install ".[all]"
```
Start services (in three separate terminals):
- Terminal 1 (Python inference server):
```
conda activate cyberverse
make inference
```
- Terminal 2 (Go API server):
```
make server
```
- Terminal 3 (Frontend):
```
make frontend
```
Verify and access:
Check API health:
```
curl -s http://localhost:8080/api/v1/health
```
Open your browser to http://localhost:5173.

For full digital human video setup, including GPU requirements and model weights download, please refer to the detailed instructions in the official CyberVerse README.

Examples

CyberVerse showcases its capabilities through impressive demonstrations of digital human agents. The repository's README features demo images and links to YouTube videos where you can see these agents in action, demonstrating real-time video interaction and natural language processing.