# hyperpb-go: Ultra-Fast Dynamic Protobuf Parsing in Go

This repository profile is provided by osrepos.com, an open source repository discovery platform.

Source: osrepos.com
Repository profile: https://osrepos.com/repo/bufbuild-hyperpb-go
Generated for open source discovery and AI-assisted research.

hyperpb-go is a highly optimized dynamic message library for Protobuf in Go, designed as a drop-in replacement for `protobuf-go`'s `dynamicpb` solution. It offers significantly faster parsing, beating `dynamicpb` by 10x and often outperforming generated code by 2-3x. This makes it ideal for read-only workloads requiring high performance with dynamic Protobuf messages.

GitHub: https://github.com/bufbuild/hyperpb-go
OSRepos URL: https://osrepos.com/repo/bufbuild-hyperpb-go

## Summary

hyperpb-go is a highly optimized dynamic message library for Protobuf in Go, designed as a drop-in replacement for `protobuf-go`'s `dynamicpb` solution. It offers significantly faster parsing, beating `dynamicpb` by 10x and often outperforming generated code by 2-3x. This makes it ideal for read-only workloads requiring high performance with dynamic Protobuf messages.

## Topics

- Go
- Protobuf
- gRPC
- Protocol Buffers
- Parsing
- Performance
- Dynamic Messages
- bufbuild

## Repository Information

Last analyzed by OSRepos: Sun Oct 19 2025 20:01:22 GMT+0100 (Western European Summer Time)
Detail views: 4
GitHub clicks: 10

## Safety Notice

OSRepos shares public repositories for knowledge and discovery only. Review source code, dependencies, licenses, and security implications before running or installing anything.

## Content

## Introduction

`hyperpb-go` is a highly optimized dynamic message library for Protobuf in Go, developed by Bufbuild. It's designed as a drop-in replacement for `protobuf-go`'s canonical `dynamicpb` solution, offering significantly enhanced performance for read-only workloads. `hyperpb-go` boasts a parser that is 10 times faster than `dynamicpb` and often 2 to 3 times faster than even generated Protobuf code, particularly for messages with many nested fields. This performance is achieved through an efficient VM for a special instruction set, a variant of table-driven parsing pioneered by the UPB project.

## Installation

To integrate `hyperpb-go` into your Go project, you can use `go get`:

bash
go get buf.build/go/hyperpb


## Examples

`hyperpb-go` requires you to pre-compile a parser using `hyperpb.Compile` at runtime, similar to how regular expressions are compiled. This allows for continuous layout optimizations without source-breaking changes.

Here's an example of compiling a type from a compiled-in descriptor and parsing data:

go
package main

import (
    "fmt"
    "log"

    "buf.build/go/hyperpb"
    "google.golang.org/protobuf/proto"

    weatherv1 "buf.build/gen/go/bufbuild/hyperpb-examples/protocolbuffers/go/example/weather/v1"
)

// Byte slice representation of a valid *weatherv1.WeatherReport.
var weatherDataBytes = []byte{
    0x0a, 0x07, 0x53, 0x65, 0x61, 0x74, 0x74, 0x6c,
    0x65, 0x12, 0x1d, 0x0a, 0x05, 0x4b, 0x41, 0x44,
    0x39, 0x33, 0x15, 0x66, 0x86, 0x22, 0x43, 0x1d,
    0xcd, 0xcc, 0x34, 0x41, 0x25, 0xd7, 0xa3, 0xf0,
    0x41, 0x2d, 0x33, 0x33, 0x13, 0x40, 0x30, 0x03,
    0x12, 0x1d, 0x0a, 0x05, 0x4b, 0x48, 0x42, 0x36,
    0x30, 0x15, 0xcd, 0x8c, 0x22, 0x43, 0x1d, 0x33,
    0x33, 0x5b, 0x41, 0x25, 0x52, 0xb8, 0xe0, 0x41,
    0x2d, 0x33, 0x33, 0xf3, 0x3f, 0x30, 0x03,
}

func main() {
    // Compile a type for your message. Make sure to cache this!
    // Here, we're using a compiled-in descriptor.
    msgType := hyperpb.CompileMessageDescriptor(
        (*weatherv1.WeatherReport)(nil).ProtoReflect().Descriptor(),
    )

    // Allocate a fresh message using that type.
    msg := hyperpb.NewMessage(msgType)

    // Parse the message, using proto.Unmarshal like any other message type.
    if err := proto.Unmarshal(weatherDataBytes, msg); err != nil {
        // Handle parse failure.
        log.Fatalf("failed to parse weather data: %v", err)
    }

    // Use reflection to read some fields. hyperpb currently only supports access
    // by reflection. You can also look up fields by index using fields.Get(), which
    // is less legible but doesn't hit a hashmap.
    fields := msgType.Descriptor().Fields()

    // Get returns a protoreflect.Value, which can be printed directly...
    fmt.Println(msg.Get(fields.ByName("region")))

    // ... or converted to an explicit type to operate on, such as with List(),
    // which converts a repeated field into something with indexing operations.
    stations := msg.Get(fields.ByName("weather_stations")).List()
    for i := range stations.Len() {
        // Get returns a protoreflect.Value too, so we need to convert it into
        // a message to keep extracting fields.
        station := stations.Get(i).Message()
        fields := station.Descriptor().Fields()

        // Here we extract each of the fields we care about from the message.
        // Again, we could use fields.Get if we know the indices.
        fmt.Println("station:", station.Get(fields.ByName("station")))
        fmt.Println("frequency:", station.Get(fields.ByName("frequency")))
        fmt.Println("temperature:", station.Get(fields.ByName("temperature")))
        fmt.Println("pressure:", station.Get(fields.ByName("pressure")))
        fmt.Println("wind_speed:", station.Get(fields.ByName("wind_speed")))
        fmt.Println("conditions:", station.Get(fields.ByName("conditions")))
    }
}


`hyperpb-go` also supports using types from a registry, enabling efficient transcoding to JSON or validation with `protovalidate` for runtime-loaded messages.

## Why Use hyperpb-go?

`hyperpb-go` offers compelling advantages for specific use cases:

*   **Exceptional Performance**: Achieve parsing speeds 10x faster than `dynamicpb` and often 2-3x faster than generated Protobuf code, especially for complex, nested messages.
*   **Dynamic Message Handling**: Ideal for generic services that download types over the network and parse messages using those types, relying on reflection.
*   **Memory Reuse**: Features a memory-reuse mechanism (`hyperpb.Shared`) to bypass the Go garbage collector, reducing allocation latency for high-throughput scenarios.
*   **Profile-Guided Optimization (PGO)**: Supports online PGO to further optimize parser performance by adapting to the average message structure, allowing for more intelligent allocations.
*   **Seamless Integration**: Works directly with existing `proto.Unmarshal`, `protojson.Marshal`, and `protovalidate.Validate` functions for non-mutating operations.

It's important to note that `hyperpb-go` currently focuses on read-only workloads and does not support mutation of parsed messages.

## Links

*   **GitHub Repository**: [bufbuild/hyperpb-go](https://github.com/bufbuild/hyperpb-go)
*   **Introducing hyperpb (Buf Blog)**: [https://buf.build/blog/hyperpb](https://buf.build/blog/hyperpb)
*   **Parsing Protobuf Like Never Before (Miguel's Blog)**: [https://mcyoung.xyz/2025/07/16/hyperpb](https://mcyoung.xyz/2025/07/16/hyperpb)
*   **Cheating the Reaper in Go (Miguel's Blog)**: [https://mcyoung.xyz/2025/04/21/go-arenas/](https://mcyoung.xyz/2025/04/21/go-arenas/)