MonoPCC: Photometric-invariant Cycle Constraint for Monocular Depth Estimation

Introduction

MonoPCC presents a novel PyTorch implementation for monocular depth estimation, primarily focusing on endoscopic images. This project introduces a photometric-invariant cycle constraint, enhancing the accuracy and robustness of depth prediction in complex medical environments. As a self-supervised learning framework, MonoPCC addresses the challenges of acquiring ground truth depth data in endoscopy by leveraging image sequences.

Installation

To set up MonoPCC, it is recommended to use a conda environment with Python 3.7. The necessary dependencies can be installed using pip and pip3 as follows:

pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0
pip install dominate==2.4.0 Pillow==6.1.0 visdom==0.1.8
pip install tensorboardX==1.4 opencv-python matplotlib scikit-image
pip3 install mmcv-full==1.3.0 mmsegmentation==0.11.0
pip install timm einops IPython

Data preparation involves datasets such as SCARED, SimCol3D, and SERV-CT. Specific training/test splits for SCARED are defined in splits/endovis, with further preprocessing details available in the AF-SfMLearner repository.

Examples

MonoPCC provides comprehensive evaluation scripts and pre-trained models.

• Depth Estimation on SCARED: To evaluate depth estimation on the SCARED dataset, first prepare ground truth depth maps following the AF-SfMLearner guide. Then, run the evaluation command:

  CUDA_VISIBLE_DEVICES=0 python evaluate_depth.py --data_path <your_data_path> --load_weights_folder <your_weight_path> \
      --eval_split endovis --dataset endovis --max_depth 150 --png --eval_mono

  MonoPCC consistently achieves state-of-the-art results, outperforming several baselines.
• Plug and Play Design: The Photometric-invariant Cycle Constraint (PCC) is designed to be plug-and-play, allowing integration into various backbone networks. For instance, when integrated with EndoDAC, PCC further enhances its performance.
• Pose Estimation: MonoPCC also supports pose estimation. After preparing ground truth pose data as described in AF-SfMLearner, you can evaluate pose estimation using the provided PoseNet models:

  CUDA_VISIBLE_DEVICES=0 python evaluate_pose.py --data_path <your_data_path> --load_weights_folder <your_weight_path>

  Visual trajectories can be generated to demonstrate the accuracy of pose estimation.
• 3D Reconstruction Demo: The project showcases a demo of 3D reconstruction using MonoPCC to estimate depth maps from video sequences, which are then used with tools like ElasticFusion to generate 3D models.
• KITTI Dataset Evaluation: To demonstrate its effectiveness in general scenarios, MonoPCC has been evaluated on the KITTI dataset. The evaluation command is:

  CUDA_VISIBLE_DEVICES=0 python evaluate_depth_kitti.py --data_path <kitti_data_path> --load_weights_folder <your_weight_path> \
      --eval_split eigen --dataset kitti --eval_mono

  MonoPCC achieves competitive results on KITTI, validating its broad applicability.

Why Use MonoPCC

MonoPCC stands out for its innovative photometric-invariant cycle constraint, which significantly improves monocular depth estimation, especially in challenging endoscopic imaging. Its self-supervised nature reduces the need for extensive labeled data, making it highly practical. The plug-and-play design allows for easy integration into existing deep learning architectures, enhancing their performance. Furthermore, its demonstrated state-of-the-art results on both medical (SCARED) and general (KITTI) datasets highlight its robustness and versatility for various computer vision tasks.

Links

• GitHub Repository: https://github.com/adam99goat/MonoPCC