{"name":"hyperpb-go: Ultra-Fast Dynamic Protobuf Parsing in Go","description":"hyperpb-go is a highly optimized dynamic message library for Protobuf in Go, designed as a drop-in replacement for `protobuf-go`'s `dynamicpb` solution. It offers significantly faster parsing, beating `dynamicpb` by 10x and often outperforming generated code by 2-3x. This makes it ideal for read-only workloads requiring high performance with dynamic Protobuf messages.","github":"https://github.com/bufbuild/hyperpb-go","url":"https://osrepos.com/repo/bufbuild-hyperpb-go","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/bufbuild-hyperpb-go","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/bufbuild-hyperpb-go.md","json":"https://osrepos.com/repo/bufbuild-hyperpb-go.json","topics":["Go","Protobuf","gRPC","Protocol Buffers","Parsing","Performance","Dynamic Messages","bufbuild"],"keywords":["Go","Protobuf","gRPC","Protocol Buffers","Parsing","Performance","Dynamic Messages","bufbuild"],"stars":null,"summary":"hyperpb-go is a highly optimized dynamic message library for Protobuf in Go, designed as a drop-in replacement for `protobuf-go`'s `dynamicpb` solution. It offers significantly faster parsing, beating `dynamicpb` by 10x and often outperforming generated code by 2-3x. This makes it ideal for read-only workloads requiring high performance with dynamic Protobuf messages.","content":"## Introduction\n\n`hyperpb-go` is a highly optimized dynamic message library for Protobuf in Go, developed by Bufbuild. It's designed as a drop-in replacement for `protobuf-go`'s canonical `dynamicpb` solution, offering significantly enhanced performance for read-only workloads. `hyperpb-go` boasts a parser that is 10 times faster than `dynamicpb` and often 2 to 3 times faster than even generated Protobuf code, particularly for messages with many nested fields. This performance is achieved through an efficient VM for a special instruction set, a variant of table-driven parsing pioneered by the UPB project.\n\n## Installation\n\nTo integrate `hyperpb-go` into your Go project, you can use `go get`:\n\nbash\ngo get buf.build/go/hyperpb\n\n\n## Examples\n\n`hyperpb-go` requires you to pre-compile a parser using `hyperpb.Compile` at runtime, similar to how regular expressions are compiled. This allows for continuous layout optimizations without source-breaking changes.\n\nHere's an example of compiling a type from a compiled-in descriptor and parsing data:\n\ngo\npackage main\n\nimport (\n    \"fmt\"\n    \"log\"\n\n    \"buf.build/go/hyperpb\"\n    \"google.golang.org/protobuf/proto\"\n\n    weatherv1 \"buf.build/gen/go/bufbuild/hyperpb-examples/protocolbuffers/go/example/weather/v1\"\n)\n\n// Byte slice representation of a valid *weatherv1.WeatherReport.\nvar weatherDataBytes = []byte{\n    0x0a, 0x07, 0x53, 0x65, 0x61, 0x74, 0x74, 0x6c,\n    0x65, 0x12, 0x1d, 0x0a, 0x05, 0x4b, 0x41, 0x44,\n    0x39, 0x33, 0x15, 0x66, 0x86, 0x22, 0x43, 0x1d,\n    0xcd, 0xcc, 0x34, 0x41, 0x25, 0xd7, 0xa3, 0xf0,\n    0x41, 0x2d, 0x33, 0x33, 0x13, 0x40, 0x30, 0x03,\n    0x12, 0x1d, 0x0a, 0x05, 0x4b, 0x48, 0x42, 0x36,\n    0x30, 0x15, 0xcd, 0x8c, 0x22, 0x43, 0x1d, 0x33,\n    0x33, 0x5b, 0x41, 0x25, 0x52, 0xb8, 0xe0, 0x41,\n    0x2d, 0x33, 0x33, 0xf3, 0x3f, 0x30, 0x03,\n}\n\nfunc main() {\n    // Compile a type for your message. Make sure to cache this!\n    // Here, we're using a compiled-in descriptor.\n    msgType := hyperpb.CompileMessageDescriptor(\n        (*weatherv1.WeatherReport)(nil).ProtoReflect().Descriptor(),\n    )\n\n    // Allocate a fresh message using that type.\n    msg := hyperpb.NewMessage(msgType)\n\n    // Parse the message, using proto.Unmarshal like any other message type.\n    if err := proto.Unmarshal(weatherDataBytes, msg); err != nil {\n        // Handle parse failure.\n        log.Fatalf(\"failed to parse weather data: %v\", err)\n    }\n\n    // Use reflection to read some fields. hyperpb currently only supports access\n    // by reflection. You can also look up fields by index using fields.Get(), which\n    // is less legible but doesn't hit a hashmap.\n    fields := msgType.Descriptor().Fields()\n\n    // Get returns a protoreflect.Value, which can be printed directly...\n    fmt.Println(msg.Get(fields.ByName(\"region\")))\n\n    // ... or converted to an explicit type to operate on, such as with List(),\n    // which converts a repeated field into something with indexing operations.\n    stations := msg.Get(fields.ByName(\"weather_stations\")).List()\n    for i := range stations.Len() {\n        // Get returns a protoreflect.Value too, so we need to convert it into\n        // a message to keep extracting fields.\n        station := stations.Get(i).Message()\n        fields := station.Descriptor().Fields()\n\n        // Here we extract each of the fields we care about from the message.\n        // Again, we could use fields.Get if we know the indices.\n        fmt.Println(\"station:\", station.Get(fields.ByName(\"station\")))\n        fmt.Println(\"frequency:\", station.Get(fields.ByName(\"frequency\")))\n        fmt.Println(\"temperature:\", station.Get(fields.ByName(\"temperature\")))\n        fmt.Println(\"pressure:\", station.Get(fields.ByName(\"pressure\")))\n        fmt.Println(\"wind_speed:\", station.Get(fields.ByName(\"wind_speed\")))\n        fmt.Println(\"conditions:\", station.Get(fields.ByName(\"conditions\")))\n    }\n}\n\n\n`hyperpb-go` also supports using types from a registry, enabling efficient transcoding to JSON or validation with `protovalidate` for runtime-loaded messages.\n\n## Why Use hyperpb-go?\n\n`hyperpb-go` offers compelling advantages for specific use cases:\n\n*   **Exceptional Performance**: Achieve parsing speeds 10x faster than `dynamicpb` and often 2-3x faster than generated Protobuf code, especially for complex, nested messages.\n*   **Dynamic Message Handling**: Ideal for generic services that download types over the network and parse messages using those types, relying on reflection.\n*   **Memory Reuse**: Features a memory-reuse mechanism (`hyperpb.Shared`) to bypass the Go garbage collector, reducing allocation latency for high-throughput scenarios.\n*   **Profile-Guided Optimization (PGO)**: Supports online PGO to further optimize parser performance by adapting to the average message structure, allowing for more intelligent allocations.\n*   **Seamless Integration**: Works directly with existing `proto.Unmarshal`, `protojson.Marshal`, and `protovalidate.Validate` functions for non-mutating operations.\n\nIt's important to note that `hyperpb-go` currently focuses on read-only workloads and does not support mutation of parsed messages.\n\n## Links\n\n*   **GitHub Repository**: [bufbuild/hyperpb-go](https://github.com/bufbuild/hyperpb-go)\n*   **Introducing hyperpb (Buf Blog)**: [https://buf.build/blog/hyperpb](https://buf.build/blog/hyperpb)\n*   **Parsing Protobuf Like Never Before (Miguel's Blog)**: [https://mcyoung.xyz/2025/07/16/hyperpb](https://mcyoung.xyz/2025/07/16/hyperpb)\n*   **Cheating the Reaper in Go (Miguel's Blog)**: [https://mcyoung.xyz/2025/04/21/go-arenas/](https://mcyoung.xyz/2025/04/21/go-arenas/)","metrics":{"detailViews":4,"githubClicks":10},"dates":{"published":null,"modified":"2025-10-19T19:01:22.000Z"}}