{"name":"CoTracker: A Powerful Model for Tracking Any Point on a Video","description":"CoTracker is a state-of-the-art model developed by Facebook AI Research and the University of Oxford, designed for tracking any point (pixel) across video sequences. This transformer-based solution offers fast, accurate, and quasi-dense point tracking capabilities. It is an invaluable tool for researchers and developers in computer vision, enabling precise analysis of motion in videos.","github":"https://github.com/facebookresearch/co-tracker","url":"https://osrepos.com/repo/facebookresearch-co-tracker","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/facebookresearch-co-tracker","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/facebookresearch-co-tracker.md","json":"https://osrepos.com/repo/facebookresearch-co-tracker.json","topics":["optical-flow","point-tracking","track-anything","Jupyter Notebook","Computer Vision","Deep Learning","Video Analysis","AI/ML"],"keywords":["optical-flow","point-tracking","track-anything","Jupyter Notebook","Computer Vision","Deep Learning","Video Analysis","AI/ML"],"stars":null,"summary":"CoTracker is a state-of-the-art model developed by Facebook AI Research and the University of Oxford, designed for tracking any point (pixel) across video sequences. This transformer-based solution offers fast, accurate, and quasi-dense point tracking capabilities. It is an invaluable tool for researchers and developers in computer vision, enabling precise analysis of motion in videos.","content":"## Introduction\n\nCoTracker, developed by Meta AI Research and the University of Oxford, is a cutting-edge model designed to track any point or pixel within a video. Leveraging a fast transformer-based architecture, CoTracker brings the benefits of optical flow to point tracking, offering robust and accurate performance.\n\nKey features of CoTracker include:\n\n*   Tracking any individual pixel in a video.\n*   Tracking a quasi-dense set of pixels simultaneously.\n*   Flexibility to select points manually or sample them on a grid in any video frame.\n\nThe project has seen continuous development, with significant updates including CoTracker3, which offers simpler and better point tracking through pseudo-labelling real videos, and the release of the Kubric Dataset for enhanced training. CoTracker has also been instrumental in other projects, such as VGGSfM, a fully differentiable SfM framework.\n\n## Installation\n\nCoTracker can be easily integrated into your projects. A GPU is strongly recommended for optimal performance.\n\n### Via PyTorch Hub\n\nThe easiest way to get started is by loading a pretrained model from `torch.hub`.\n\n**Offline mode:**\n\nbash\npip install imageio[ffmpeg]\n\n\npython\nimport torch\nimport imageio.v3 as iio\n\nurl = 'https://github.com/facebookresearch/co-tracker/raw/refs/heads/main/assets/apple.mp4'\nframes = iio.imread(url, plugin=\"FFMPEG\")\n\ndevice = 'cuda'\ngrid_size = 10\nvideo = torch.tensor(frames).permute(0, 3, 1, 2)[None].float().to(device)\n\ncotracker = torch.hub.load(\"facebookresearch/co-tracker\", \"cotracker3_offline\").to(device)\npred_tracks, pred_visibility = cotracker(video, grid_size=grid_size)\n\n\n**Online mode:**\n\npython\nimport torch\nimport imageio.v3 as iio\n\nurl = 'https://github.com/facebookresearch/co-tracker/raw/refs/heads/main/assets/apple.mp4'\nframes = iio.imread(url, plugin=\"FFMPEG\")\n\ndevice = 'cuda'\ngrid_size = 10\nvideo = torch.tensor(frames).permute(0, 3, 1, 2)[None].float().to(device)\n\ncotracker = torch.hub.load(\"facebookresearch/co-tracker\", \"cotracker3_online\").to(device)\ncotracker(video_chunk=video, is_first_step=True, grid_size=grid_size)\n\nfor ind in range(0, video.shape[1] - cotracker.step, cotracker.step):\n    pred_tracks, pred_visibility = cotracker(\n        video_chunk=video[:, ind : ind + cotracker.step * 2]\n    )\n\n\n### Development Version\n\nTo install CoTracker from the GitHub repository, which is ideal for running local demos or for evaluation and training:\n\nbash\ngit clone https://github.com/facebookresearch/co-tracker\ncd co-tracker\npip install -e .\npip install matplotlib flow_vis tqdm tensorboard\n\n\n## Examples\n\nCoTracker provides various ways to interact with its powerful tracking capabilities.\n\n### Visualize Predicted Tracks\n\nAfter installation, you can visualize the tracks with a simple script:\n\npython\nfrom cotracker.utils.visualizer import Visualizer\n\nvis = Visualizer(save_dir=\"./saved_videos\", pad_value=120, linewidth=3)\nvis.visualize(video, pred_tracks, pred_visibility)\n\n\n### Interactive Demos\n\n*   **Hugging Face Space:** Explore the interactive demo on the [facebook/cotracker Hugging Face Space](https://huggingface.co/spaces/facebook/cotracker \"Hugging Face Space\" target=\"_blank\").\n*   **Google Colab:** Run the notebook directly in [Google Colab](https://colab.research.google.com/github/facebookresearch/co-tracker/blob/master/notebooks/demo.ipynb \"Google Colab Demo\" target=\"_blank\").\n*   **Local Gradio Demo:** Run the Gradio demo locally by installing requirements (`pip install -r gradio_demo/requirements.txt`) and executing `python -m gradio_demo.app`.\n\n### Command Line Demos\n\n*   **Offline Demo:** `python demo.py --grid_size 10` (results saved to `./saved_videos/demo.mp4`).\n*   **Online Demo:** `python online_demo.py`.\n\n## Why Use CoTracker?\n\nCoTracker stands out as a robust solution for point tracking due to several compelling reasons:\n\n*   **State-of-the-Art Performance:** CoTracker3 achieves impressive results on benchmarks like TAP-Vid, demonstrating superior accuracy compared to previous versions and other models.\n*   **Versatility:** It can track any pixel, a quasi-dense set of pixels, and supports both manual and grid-based point selection.\n*   **Efficiency:** The transformer-based architecture ensures fast processing, and the online mode offers memory-efficient tracking for longer videos.\n*   **Continuous Improvement:** The project is actively maintained with regular updates, including new models, datasets, and features.\n*   **Developer-Friendly:** With PyTorch Hub integration, clear installation instructions, and various demo options, CoTracker is accessible for both quick experimentation and in-depth development.\n\n## Links\n\n*   **GitHub Repository:** [https://github.com/facebookresearch/co-tracker](https://github.com/facebookresearch/co-tracker \"CoTracker GitHub\" target=\"_blank\")\n*   **Project Page:** [https://cotracker3.github.io/](https://cotracker3.github.io/ \"CoTracker Project Page\" target=\"_blank\")\n*   **Paper #1 (ECCV 2024):** [https://arxiv.org/abs/2307.07635](https://arxiv.org/abs/2307.07635 \"CoTracker ECCV Paper\" target=\"_blank\")\n*   **Paper #2 (arXiv 2024):** [https://arxiv.org/abs/2410.11831](https://arxiv.org/abs/2410.11831 \"CoTracker3 arXiv Paper\" target=\"_blank\")\n*   **X Thread:** [https://twitter.com/n_karaev/status/1742638906355470772](https://twitter.com/n_karaev/status/1742638906355470772 \"CoTracker X Thread\" target=\"_blank\")\n*   **Colab Demo:** [https://colab.research.google.com/github/facebookresearch/co-tracker/blob/main/notebooks/demo.ipynb](https://colab.research.google.com/github/facebookresearch/co-tracker/blob/main/notebooks/demo.ipynb \"CoTracker Colab Demo\" target=\"_blank\")\n*   **Hugging Face Space:** [https://huggingface.co/spaces/facebook/cotracker](https://huggingface.co/spaces/facebook/cotracker \"CoTracker Hugging Face Space\" target=\"_blank\")","metrics":{"detailViews":6,"githubClicks":9},"dates":{"published":null,"modified":"2026-04-11T11:21:20.000Z"}}