PyO3: Seamless Rust Bindings for the Python Interpreter

Introduction

PyO3 is a robust and widely-used library that provides Rust bindings for the Python interpreter. It empowers developers to write high-performance native Python extension modules entirely in Rust, taking advantage of Rust's speed, memory safety, and concurrency features. Beyond creating Python modules, PyO3 also facilitates embedding and interacting with Python code directly from Rust applications, offering a versatile solution for interoperability between these two powerful languages.

Installation

PyO3 supports various Python distributions, including CPython 3.7+, PyPy 7.3 (Python 3.11+), and GraalPy 25.0+ (Python 3.12+). The installation process varies slightly depending on whether you're using Rust from Python or Python from Rust.

Using Rust from Python

The easiest way to get started with PyO3 for creating native Python modules is by using maturin, a tool designed for building and publishing Rust-based Python packages. Here's a quick setup:

1. Create a project and virtual environment:

   mkdir my_rust_module
   cd my_rust_module
   python -m venv .env
   source .env/bin/activate
   pip install maturin
   

2. Initialize your project with maturin:

   maturin init
   # Select 'pyo3' when prompted for bindings.
   

   This generates Cargo.toml and src/lib.rs. A minimal src/lib.rs might look like this:

   use pyo3::prelude::*;
   
   #[pyo3::pymodule]
   fn my_rust_module(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
       m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
       Ok(())
   }
   
   #[pyfunction]
   fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
       Ok((a + b).to_string())
   }
   

3. Build and install:

   maturin develop
   

   Now you can import and use your Rust-powered module in Python:

   >>> import my_rust_module
   >>> my_rust_module.sum_as_string(5, 20)
   '25'
   

Using Python from Rust

To embed a Python interpreter within a Rust binary, you'll need the Python shared library installed on your system. For Ubuntu, use sudo apt install python3-dev, and for RPM-based distributions, install python3-devel.

Then, add pyo3 to your Cargo.toml with the auto-initialize feature:

[dependencies.pyo3]
version = "0.27.2"
features = ["auto-initialize"]

Here's an example of running Python code from Rust:

use pyo3::prelude::*;
use pyo3::types::IntoPyDict;

fn main() -> PyResult<()> {
    Python::attach(|py| {
        let sys = py.import("sys")?;
        let version: String = sys.getattr("version")?.extract()?;

        let locals = [("os", py.import("os")?)].into_py_dict(py)?;
        let code = "os.getenv('USER') or os.getenv('USERNAME') or 'Unknown'";
        let user: String = py.eval(code, None, Some(&locals))?.extract()?;

        println!("Hello {}, I'm Python {}", user, version);
        Ok(())
    })
}

Examples

PyO3 is utilized in a wide array of projects, demonstrating its versatility and impact. Some notable examples include:

• Polars: A fast multi-threaded DataFrame library available in Rust, Python, and Node.js.

• Pydantic-core: The core validation logic for the popular Pydantic library, rewritten in Rust for performance.

• Tiktoken: OpenAI's fast BPE tokeniser, used with their models.

• Tokenizers: Python bindings for Hugging Face's NLP tokenizers, implemented in Rust.

• Orjson: A high-performance Python JSON library.

Why Use PyO3?

Choosing PyO3 for Python-Rust interoperability offers several compelling advantages:

• Performance Boost: Leverage Rust's unparalleled speed for computationally intensive tasks, offloading critical sections of your Python application to native code.

• Memory Safety: Benefit from Rust's strong type system and ownership model, eliminating common memory-related bugs and enhancing application stability.

• Seamless Interoperability: PyO3 provides a natural and idiomatic way to call Rust functions from Python and vice-versa, making the integration feel native to both languages.

• Access to Rust Ecosystem: Utilize Rust's rich ecosystem of crates, bringing advanced functionalities like efficient data processing, cryptography, and more to your Python projects.

• Robust Tooling: Supported by tools like maturin and setuptools-rust, simplifying the build, packaging, and distribution of Rust-powered Python extensions.

Links

• GitHub Repository: https://github.com/PyO3/pyo3

• User Guide: https://pyo3.rs

• API Documentation: https://docs.rs/pyo3/