hyperpb-go: Ultra-Fast Dynamic Protobuf Parsing in Go

Introduction

hyperpb-go is a highly optimized dynamic message library for Protobuf in Go, developed by Bufbuild. It's designed as a drop-in replacement for protobuf-go's canonical dynamicpb solution, offering significantly enhanced performance for read-only workloads. hyperpb-go boasts a parser that is 10 times faster than dynamicpb and often 2 to 3 times faster than even generated Protobuf code, particularly for messages with many nested fields. This performance is achieved through an efficient VM for a special instruction set, a variant of table-driven parsing pioneered by the UPB project.

Installation

To integrate hyperpb-go into your Go project, you can use go get:

go get buf.build/go/hyperpb

Examples

hyperpb-go requires you to pre-compile a parser using hyperpb.Compile at runtime, similar to how regular expressions are compiled. This allows for continuous layout optimizations without source-breaking changes.

Here's an example of compiling a type from a compiled-in descriptor and parsing data:

package main

import (
    "fmt"
    "log"

    "buf.build/go/hyperpb"
    "google.golang.org/protobuf/proto"

    weatherv1 "buf.build/gen/go/bufbuild/hyperpb-examples/protocolbuffers/go/example/weather/v1"
)

// Byte slice representation of a valid *weatherv1.WeatherReport.
var weatherDataBytes = []byte{
    0x0a, 0x07, 0x53, 0x65, 0x61, 0x74, 0x74, 0x6c,
    0x65, 0x12, 0x1d, 0x0a, 0x05, 0x4b, 0x41, 0x44,
    0x39, 0x33, 0x15, 0x66, 0x86, 0x22, 0x43, 0x1d,
    0xcd, 0xcc, 0x34, 0x41, 0x25, 0xd7, 0xa3, 0xf0,
    0x41, 0x2d, 0x33, 0x33, 0x13, 0x40, 0x30, 0x03,
    0x12, 0x1d, 0x0a, 0x05, 0x4b, 0x48, 0x42, 0x36,
    0x30, 0x15, 0xcd, 0x8c, 0x22, 0x43, 0x1d, 0x33,
    0x33, 0x5b, 0x41, 0x25, 0x52, 0xb8, 0xe0, 0x41,
    0x2d, 0x33, 0x33, 0xf3, 0x3f, 0x30, 0x03,
}

func main() {
    // Compile a type for your message. Make sure to cache this!
    // Here, we're using a compiled-in descriptor.
    msgType := hyperpb.CompileMessageDescriptor(
        (*weatherv1.WeatherReport)(nil).ProtoReflect().Descriptor(),
    )

    // Allocate a fresh message using that type.
    msg := hyperpb.NewMessage(msgType)

    // Parse the message, using proto.Unmarshal like any other message type.
    if err := proto.Unmarshal(weatherDataBytes, msg); err != nil {
        // Handle parse failure.
        log.Fatalf("failed to parse weather data: %v", err)
    }

    // Use reflection to read some fields. hyperpb currently only supports access
    // by reflection. You can also look up fields by index using fields.Get(), which
    // is less legible but doesn't hit a hashmap.
    fields := msgType.Descriptor().Fields()

    // Get returns a protoreflect.Value, which can be printed directly...
    fmt.Println(msg.Get(fields.ByName("region")))

    // ... or converted to an explicit type to operate on, such as with List(),
    // which converts a repeated field into something with indexing operations.
    stations := msg.Get(fields.ByName("weather_stations")).List()
    for i := range stations.Len() {
        // Get returns a protoreflect.Value too, so we need to convert it into
        // a message to keep extracting fields.
        station := stations.Get(i).Message()
        fields := station.Descriptor().Fields()

        // Here we extract each of the fields we care about from the message.
        // Again, we could use fields.Get if we know the indices.
        fmt.Println("station:", station.Get(fields.ByName("station")))
        fmt.Println("frequency:", station.Get(fields.ByName("frequency")))
        fmt.Println("temperature:", station.Get(fields.ByName("temperature")))
        fmt.Println("pressure:", station.Get(fields.ByName("pressure")))
        fmt.Println("wind_speed:", station.Get(fields.ByName("wind_speed")))
        fmt.Println("conditions:", station.Get(fields.ByName("conditions")))
    }
}

hyperpb-go also supports using types from a registry, enabling efficient transcoding to JSON or validation with protovalidate for runtime-loaded messages.

Why Use hyperpb-go?

hyperpb-go offers compelling advantages for specific use cases:

Exceptional Performance: Achieve parsing speeds 10x faster than dynamicpb and often 2-3x faster than generated Protobuf code, especially for complex, nested messages.
Dynamic Message Handling: Ideal for generic services that download types over the network and parse messages using those types, relying on reflection.
Memory Reuse: Features a memory-reuse mechanism (hyperpb.Shared) to bypass the Go garbage collector, reducing allocation latency for high-throughput scenarios.
Profile-Guided Optimization (PGO): Supports online PGO to further optimize parser performance by adapting to the average message structure, allowing for more intelligent allocations.
Seamless Integration: Works directly with existing proto.Unmarshal, protojson.Marshal, and protovalidate.Validate functions for non-mutating operations.