co-tracker: Track Points Across Video Frames

Overview

CoTracker estimates where points move across video frames. It is designed for cases where you need trajectories for chosen pixels or many points at once, rather than only a dense optical-flow field. Points can be selected manually or sampled on a grid.

The repository includes CoTracker3 models and earlier versions, with pretrained checkpoints and inference interfaces for both complete videos and streaming chunks. It is aimed at computer-vision practitioners who need point trajectories for video analysis, research, or downstream 3D reconstruction.

Key Features

• Track arbitrary selected pixels or grid-sampled points across video.
• Process a quasi-dense set of points jointly.
• Choose offline inference or online, chunk-by-chunk processing.
• Load pretrained models through PyTorch Hub or use local checkpoints.
• Visualize predicted tracks and visibility with the included utilities.
• Includes evaluation workflows for TAP-Vid and Dynamic Replica datasets.
• Provides training code for Kubric data and pseudo-label fine-tuning on real videos.

Use Cases

• Researchers can evaluate point-tracking methods on video benchmarks using the supplied evaluation code.
• Robotics and vision teams can follow selected image points through a video as input to later motion or scene analysis.
• Developers building video-analysis tools can use grid-sampled tracks to inspect motion across a scene.
• Researchers can explore online tracking when video arrives in chunks or its total length is unknown.

Getting Started

Install the package and its basic visualization dependencies from a clone:

git clone https://github.com/facebookresearch/co-tracker
cd co-tracker
pip install -e .
pip install matplotlib flow_vis tqdm tensorboard

For pretrained inference, the README also shows PyTorch Hub examples. See the repository README for checkpoint setup, demos, and training instructions.

Alternatives

• vggt: VGGT estimates point tracks alongside cameras, depth, and 3D scene structure from image sets, while CoTracker focuses on tracking selected or sampled points through video.

Considerations

• A GPU is strongly recommended for local use, although the README notes that small tasks can run on CPU.
• Local installation requires PyTorch and TorchVision; CUDA support is recommended.
• Training is resource-intensive: the documented Kubric run used 32 GPUs, and pseudo-label fine-tuning requires a real-video dataset and a Kubric-trained model.
• The README says the majority of the project is licensed under CC-BY-NC, with some portions under separate terms. The repository metadata reports NOASSERTION, so check the relevant license files before use, especially for commercial work.
• The repository is not archived, but the input reports 102 open issues. Review current issues and compatibility details before adopting it.