Are you an LLM? Read llms.txt for a summary of the docs, or llms-full.txt for the full context.
Skip to content

Codecs

A codec controls how a topic's messages cross the wire. Every topic carries a Codec associated type that decides how T::Message is encoded on publish and decoded on consume. The default is JSON, which means existing code keeps working without any changes; opting into a different encoding is a one-line edit on the topic definition.

The codec slot is per-topic, not per-broker. Two topics on the same backend may speak entirely different wire formats. This is the right boundary: a topic models a single event flow, and the encoding is a property of that flow, not of the transport.

When to swap the default

Three situations come up in practice:

  • Protobuf, when you need a compact, schema-evolution-aware wire format, or when you share message contracts with services written in other languages.
  • Raw bytes, when the payload is already encoded by an upstream component (Confluent Schema Registry's framed Avro, opaque blobs from a legacy system, a third-party format with its own framing).
  • A custom codec, when none of the above fit — for example, CBOR or a domain-specific framing.

Outside those situations, stick with JSON; it is the default, requires no extra dependencies, and is the easiest format to debug.

The Codec trait

The trait is short:

pub trait Codec<M>: Send + Sync + 'static {
    const NAME: &'static str;
    fn encode(value: &M) -> Result<Vec<u8>>;
    fn decode(bytes: &[u8]) -> Result<M>;
}

encode and decode are associated functions, not methods on a value. The codec carries no state; the macros plug the type in directly. NAME is a stable label used in logs.

JsonCodec — the default

JsonCodec is what you get when define_topic! is invoked without a codec = … clause:

define_topic!(
    OrderSettlement,
    SettlementEvent,
    TopologyBuilder::new("order-settlement").dlq().build()
);

This compiles down to a topic with type Codec = JsonCodec;. Payloads ride the wire as serde_json::to_vec output, and serde_json::from_slice is called on the consumer side. SettlementEvent must implement Serialize and DeserializeOwned.

ProtobufCodec

Enable the protobuf cargo feature, derive prost::Message on the payload, and pass the codec to the macro:

#[derive(Clone, PartialEq, ::prost::Message)]
struct OrderEvent {
    #[prost(string, tag = "1")]
    order_id: String,
    #[prost(double, tag = "2")]
    amount: f64,
}
 
shove::define_topic!(
    Orders,
    OrderEvent,
    TopologyBuilder::new("kafka-orders-proto").dlq().build(),
    codec = ProtobufCodec
);

ProtobufCodec is a single marker that implements Codec<M> for every M: prost::Message + Default. One topic can use codec = ProtobufCodec for OrderEvent, another for UserCreated, and so on — each call site picks the matching impl through <ProtobufCodec as Codec<T::Message>>. Encoding pre-sizes the buffer via encoded_len, so a single allocation covers the common case. Decoding failures surface as ShoveError::Codec { codec: "protobuf", .. } rather than Serialization, so handlers and middleware can distinguish protobuf-decode errors from JSON ones.

Runnable example: examples/kafka/protobuf_pubsub.rs.

Generating Rust types from a .proto file

The example above defines OrderEvent inline with #[derive(prost::Message)]. In production you usually drive the schema from a .proto file and let prost-build generate the Rust types at compile time. shove sees the generated type the same as a hand-written one — anything implementing prost::Message + Default works.

Add prost-build as a build dependency:

# Cargo.toml
[dependencies]
shove = { version = "0.11", features = ["kafka", "protobuf"] }
prost = "0.13"
 
[build-dependencies]
prost-build = "0.13"

Write the schema:

// proto/order.proto
syntax = "proto3";
package myservice;
 
message OrderEvent {
  string order_id = 1;
  double amount = 2;
}

Compile it at build time:

// build.rs
fn main() -> Result<(), Box<dyn std::error::Error>> {
    prost_build::compile_protos(&["proto/order.proto"], &["proto/"])?;
    Ok(())
}

Then include the generated module and bind it to a topic:

use shove::{define_topic, ProtobufCodec, TopologyBuilder};

// The module name matches the `.proto`'s `package` declaration.
mod myservice {
    include!(concat!(env!("OUT_DIR"), "/myservice.rs"));
}

define_topic!(
    Orders,
    myservice::OrderEvent,
    TopologyBuilder::new("orders").dlq().build(),
    codec = ProtobufCodec
);

For gRPC services, swap prost-build for tonic-build — the generated message types still implement prost::Message, so ProtobufCodec works on them unchanged. If you already depend on a separate crate that publishes generated protobuf types, depend on it as a normal dependency and skip the build.rs step.

shove deliberately does not bundle a .proto compiler — prost-build and tonic-build are the standard tools and would only duplicate them. The protobuf feature in shove is the runtime codec only.

RawBytesCodec — the escape hatch

RawBytesCodec is a passthrough for payloads that are already encoded. The topic carries Vec<u8> and the handler owns every wire-format decision:

shove::define_topic!(
    RawTopic,
    Vec<u8>,
    TopologyBuilder::new("raw-bytes").dlq().build(),
    codec = RawBytesCodec
);

The canonical use case is Confluent Schema Registry. Records on that wire arrive as [magic_byte, schema_id (4 bytes), avro_payload...]. The library doesn't ship a Schema Registry client, but RawBytesCodec lets the handler strip the 5-byte framing header, look up the schema, and decode the Avro payload itself. Re-publishing works the same way in reverse: prepend the framing header before publishing.

Reach for RawBytesCodec only when the format genuinely isn't serde-friendly. Once the bytes leave the handler, the type system can no longer help you.

Runnable example: examples/nats/raw_bytes.rs.

Migrating hand-rolled Topic impls

If you implement Topic by hand instead of via define_topic!, you now need an explicit type Codec line:

impl Topic for OrderSettlement {
    type Message = SettlementEvent;
    type Codec = JsonCodec;
    fn topology() -> &'static QueueTopology { /* ... */ }
}

JsonCodec preserves the old behaviour exactly. Macro-generated topics already do this for you.

Custom codecs

Implement Codec<M> for your own type when neither JSON, Protobuf, nor raw bytes fits. The contract is round-trip safety: decode(encode(m).unwrap()).unwrap() == m for every m the codec supports. Encode and decode errors should surface as ShoveError::Codec { codec: "<name>", source } so consumers see a consistent error variant.