{"name":"PPS-Ctrl: Controllable Sim-to-Real Translation for Colonoscopy Depth Estimation","description":"PPS-Ctrl is an innovative image translation framework designed for controllable sim-to-real translation in colonoscopy depth estimation. It leverages Stable Diffusion and ControlNet, guided by a unique Per-Pixel Shading (PPS) map. This approach provides a physics-informed structural prior, enhancing texture realism and structure preservation in medical imaging applications.","github":"https://github.com/anaxqx/PPS-Ctrl","url":"https://osrepos.com/repo/anaxqx-pps-ctrl","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/anaxqx-pps-ctrl","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/anaxqx-pps-ctrl.md","json":"https://osrepos.com/repo/anaxqx-pps-ctrl.json","topics":["Python","Sim-to-Real","Colonoscopy","Depth Estimation","Stable Diffusion","ControlNet","Medical Imaging","Image Translation"],"keywords":["Python","Sim-to-Real","Colonoscopy","Depth Estimation","Stable Diffusion","ControlNet","Medical Imaging","Image Translation"],"stars":null,"summary":"PPS-Ctrl is an innovative image translation framework designed for controllable sim-to-real translation in colonoscopy depth estimation. It leverages Stable Diffusion and ControlNet, guided by a unique Per-Pixel Shading (PPS) map. This approach provides a physics-informed structural prior, enhancing texture realism and structure preservation in medical imaging applications.","content":"## Introduction\n\nPPS-Ctrl introduces a novel image translation framework for controllable sim-to-real translation, specifically tailored for colonoscopy depth estimation. This project combines the power of Stable Diffusion and ControlNet, uniquely guided by a Per-Pixel Shading (PPS) map. Unlike traditional sim-to-real methods that rely on depth maps, PPS-Ctrl utilizes a physics-informed representation of surface-light interactions, ensuring a more faithful and geometrically consistent structural prior. This results in superior texture realism and structure preservation, crucial for accurate medical image translation.\n\n## Installation\n\nTo get started with PPS-Ctrl, follow these steps to set up your environment and prepare your data.\n\n### 1. Environment Setup\n\nIt is recommended to use Python 3.9 with PyTorch ? 2.0 and the HuggingFace `diffusers` library.\n\nbash\nconda create -n ppsctrl python=3.9\nconda activate ppsctrl\npip install -r requirements.txt\n\n\n### 2. Prepare Data\n\nDownload the necessary datasets:\n\n*   [SimCol3D](https://www.ucl.ac.uk/interventional-surgical-sciences/simcol3d-3d-reconstruction-during-colonoscopy-challenge)\n*   [C3VD](https://durrlab.github.io/C3VD/)\n*   [Colon10K](https://endoscopography.web.unc.edu/place-recognition-in-colonoscopy/)\n\nAfter downloading, precompute PPS maps using the provided utility script:\n\nbash\npip install opencv-python # Required for compute_pps.py\npython utils/compute_pps.py --depth_dir path/to/depth --output_dir path/to/pps\n\n\n## Examples\n\nPPS-Ctrl's workflow involves a two-stage training process followed by inference.\n\n### 1. Train\n\n#### Stage 1: Fine-tune Stable Diffusion\n\nbash\nbash scripts/train_sd.sh\n\n\n#### Stage 2: Train ControlNet with PPS conditioning\n\nbash\nbash scripts/train_controlnet.sh\n\n\n### 2. Inference\n\nOnce trained, you can perform inference to generate images:\n\nbash\npython scripts/infer.py --depth path/to/test/depth --output path/to/save\n\n\n## Why Use PPS-Ctrl?\n\nPPS-Ctrl offers significant advantages for researchers and developers working on medical image translation, particularly in colonoscopy. By leveraging a Per-Pixel Shading map as a structural prior, it achieves superior texture realism and maintains geometric consistency, which are critical for accurate depth estimation in complex anatomical environments. This physics-informed approach provides a robust foundation for developing more reliable sim-to-real translation models, potentially leading to advancements in surgical simulation, training, and diagnostic tools.\n\n## Links\n\n*   [GitHub Repository](https://github.com/anaxqx/PPS-Ctrl)","metrics":{"detailViews":4,"githubClicks":8},"dates":{"published":null,"modified":"2025-11-28T12:01:16.000Z"}}