WiFi-3D-Fusion: Real-Time 3D Human Pose Estimation from WiFi Signals

Introduction

WiFi-3D-Fusion is a groundbreaking open-source research project that redefines spatial awareness by estimating 3D human pose using ordinary WiFi signals. This innovative system combines Channel State Information (CSI) from WiFi with advanced deep learning and computer vision techniques. It provides real-time motion detection and visualization, allowing users to "see" human movement in 3D environments without the need for cameras. The project is designed for live local Wi-Fi sensing, offering capabilities like multi-person 3D pose estimation and neural RF radiance fields.

Installation

Getting started with WiFi-3D-Fusion requires a Linux system (Ubuntu 22.04+ recommended) and Python 3.8+. You'll also need a compatible WiFi adapter, such as a dual-band USB WiFi adapter with the Realtek RTL8812AU chipset or an ESP32 with CSI firmware.

To set up the project:

1. Clone the repository:

   git clone https://github.com/MaliosDark/wifi-3d-fusion.git
   cd wifi-3d-fusion
   

2. Install all dependencies and set up the environment:

   bash scripts/install_all.sh
   

3. Activate the Python environment:

   source venv/bin/activate
   

Detailed hardware setup instructions for ESP32-CSI and Nexmon are available in the project's README and DeepWiki documentation.

Examples

WiFi-3D-Fusion offers both web-based and traditional terminal-based execution. The web-based visualization is highly recommended for an interactive experience.

Web-Based Real-Time Visualization (Recommended)

After installation and activating your Python environment, run:

python run_js_visualizer.py

Then, open your browser to http://localhost:5000 to see the real-time 3D visualization. You can specify the data source (e.g., esp32 or nexmon) using the --source flag.

Traditional Pipeline

For a terminal-based experience, you can run:

./scripts/run_realtime.sh --source esp32
# Or for Nexmon (requires monitor-mode interface)
sudo ./scripts/run_realtime.sh --source nexmon

Model Training

The project also supports training your own detection models, including continuous learning features where the model improves automatically over time.

# Basic training with current configuration
./train_wifi3d.sh

# Train with continuous learning enabled
./train_wifi3d.sh --continuous --auto-improve

Why Use WiFi-3D-Fusion?

WiFi-3D-Fusion stands out by revealing the invisible, transforming ambient WiFi signals into actionable spatial data. This project demonstrates that technology can sense presence and movement without relying on traditional cameras, offering significant advantages in privacy-sensitive environments or conditions where visual sensors are ineffective. Imagine applications in dark rooms, burning buildings, collapsed tunnels, or deep underground, where this system could provide critical information, potentially saving lives. It's a testament to pushing the boundaries of what's considered "impossible" in sensing technology, proving that the air around us is alive with data waiting to be interpreted.

It's important to note that this is a research project, and while powerful, WiFi sensing faces challenges like signal interference and resolution limits. Users must operate it ethically and with explicit authorization on networks and environments they own or control, respecting local laws and privacy regulations.

Links

Explore the project further through these official resources:

• GitHub Repository: https://github.com/MaliosDark/wifi-3d-fusion
• Extended README (DeepWiki): https://deepwiki.org/MaliosDark/wifi-3d-fusion/
• Demo Video: A compelling demo video showcasing WiFi-3D-Fusion in action is available in the project's GitHub README.