Kafka — Stress

Use this to measure Kafka throughput on your workload and compare it against the InMemory baseline and NATS results. The numbers quantify Kafka's write-ahead log cost relative to lighter backends — useful data when choosing between Kafka and NATS, or when sizing partitions and consumer counts for a production deployment.

Prerequisites

Docker (an Apache Kafka testcontainer is started automatically)
Cargo feature: kafka

Run

cargo run --example kafka_stress --features kafka

Narrow to one tier or handler profile:

cargo run --example kafka_stress --features kafka -- --tier moderate --handler fast

Release mode for representative numbers:

cargo run -q --release --example kafka_stress --features kafka

Expected output

Non-deterministic. Look for these characteristic markers:

shove stress benchmarks — kafka
scenarios: 60

[1/60] moderate | 20000msg | 1c | fast (1-5ms) ...
  -> 8200.4 msg/s | dispatch p50=0.3ms p99=1.8ms | e2e p50=2.9ms p99=5.1ms | cpu=62% rss=44.1MB | 2.4s
...

Backend: kafka
TIER         MSGS     C  HANDLER   MSG/SEC  ...
moderate    20000     1  fast       8200    ...
moderate    20000     4  fast      25000    ...
...

Throughput is typically in the thousands to tens of thousands of msg/s — higher than RabbitMQ for zero handlers due to Kafka's batch-oriented producer, but with a hard ceiling at the partition count.

Source

//! Stress benchmarks for the Kafka backend.
//!
//! Spins up a Kafka testcontainer for the lifetime of the process. Requires a
//! running Docker daemon.
//!
//!     cargo run -q --example kafka_stress --features kafka
//!     cargo run -q --example kafka_stress --features kafka -- --tier moderate
 
#[path = "../common/stress_test.rs"]
mod harness;
 
use std::time::Duration;
 
use rdkafka::ClientConfig;
use rdkafka::admin::{AdminClient, AdminOptions};
use rdkafka::client::DefaultClientContext;
use rdkafka::consumer::{BaseConsumer, Consumer};
use shove::kafka::{KafkaConfig, KafkaConsumerGroupConfig};
use shove::{Broker, Kafka};
use testcontainers::runners::AsyncRunner;
use testcontainers_modules::kafka::apache::{self, Kafka as KafkaImage};
 
use harness::{HarnessConfig, run_all_scenarios};
 
const TOPIC_NAME: &str = "shove-stress-bench";
const DLQ_NAME: &str = "shove-stress-bench-dlq";
 
#[tokio::main]
async fn main() {
    let container = KafkaImage::default()
        .start()
        .await
        .expect("failed to start Kafka container");
    let port = container
        .get_host_port_ipv4(apache::KAFKA_PORT)
        .await
        .expect("failed to read Kafka port");
    let bootstrap = format!("127.0.0.1:{port}");
 
    wait_until_ready(&bootstrap).await;
 
    let purge_bootstrap = bootstrap.clone();
    let purge: harness::PurgeFn = Box::new(move || {
        let bootstrap = purge_bootstrap.clone();
        Box::pin(async move {
            // Delete the topics AND the consumer groups derived from them.
            // The topic delete on its own resets storage and lets the next
            // scenario re-declare with a fresh partition count sized to its
            // own `max_consumers`, but it leaves the consumer group state
            // (offsets, rebalance epoch, dead members) sitting in Kafka's
            // group coordinator. With many short scenarios that residue
            // accumulates and the next scenario eats a long rebalance
            // before steady-state, flattening apparent throughput. Wiping
            // the group too gives each scenario a clean baseline.
            let Ok(admin): Result<AdminClient<DefaultClientContext>, _> = ClientConfig::new()
                .set("bootstrap.servers", &bootstrap)
                .create()
            else {
                return;
            };
            let _ = admin
                .delete_topics(&[TOPIC_NAME, DLQ_NAME], &AdminOptions::new())
                .await;
            let main_group = format!("{TOPIC_NAME}-consumer");
            let dlq_group = format!("{DLQ_NAME}-consumer");
            let _ = admin
                .delete_groups(&[&main_group, &dlq_group], &AdminOptions::new())
                .await;
        })
    });
 
    let hcfg = HarnessConfig::<Kafka>::new("kafka").with_purge(purge);
    run_all_scenarios(
        hcfg,
        || {
            let bootstrap = bootstrap.clone();
            async move {
                Broker::<Kafka>::new(KafkaConfig::new(&bootstrap))
                    .await
                    .expect("connect Kafka")
            }
        },
        |consumers, prefetch, concurrent| {
            KafkaConsumerGroupConfig::new(consumers..=consumers)
                .with_prefetch_count(prefetch)
                .with_concurrent_processing(concurrent)
        },
    )
    .await;
 
    drop(container);
}
 
/// Poll the broker until a metadata fetch succeeds. Testcontainers' Kafka
/// image returns from `.start()` as soon as the process logs "started", but
/// the broker may still be coming up internally and reject the first
/// connection attempts. Without this wait, the first scenario eats the
/// startup latency inside its measurement window.
async fn wait_until_ready(bootstrap: &str) {
    let probe: BaseConsumer = ClientConfig::new()
        .set("bootstrap.servers", bootstrap)
        .set("group.id", "shove-stress-probe")
        .create()
        .expect("build Kafka probe consumer");
 
    let deadline = std::time::Instant::now() + Duration::from_secs(60);
    loop {
        match probe.fetch_metadata(None, Duration::from_secs(2)) {
            Ok(md) if !md.brokers().is_empty() => return,
            _ if std::time::Instant::now() >= deadline => {
                panic!("Kafka broker did not become ready within 60s")
            }
            _ => tokio::time::sleep(Duration::from_millis(200)).await,
        }
    }
}

Walkthrough

Topic delete-and-recreate between scenarios

with_purge(purge) injects a closure that uses an rdkafka AdminClient to delete the main topic and DLQ topic between scenarios. Unlike NATS (which deletes the stream) or RabbitMQ (which purges the queue), Kafka deletes and recreates topics because ensure_partitions can only expand partition counts, not shrink them. Each scenario declares its own topology with a partition count matched to consumers, so the previous scenario's partition layout must be cleared to avoid idle consumers.

Partition count and consumer ceiling

The make_cfg closure produces:

KafkaConsumerGroupConfig::new(consumers..=consumers)
    .with_prefetch_count(prefetch)

On Kafka, declare provisions exactly consumers partitions for the main topic. When the stress harness steps through consumer counts like [1, 4, 8, 16, 32], each scenario first purges and then redeclares with the correct partition count for that scenario's consumer level. The consumer count ceiling for scaling_efficiency is thus the partition count — adding more consumers than partitions would leave some idle and cap throughput.

`KafkaConsumerGroupConfig` with concurrent processing

with_concurrent_processing(concurrent) enables parallel handler dispatch within a single consumer partition assignment. When --concurrent is passed, each consumed record spawns a tokio task, allowing up to prefetch records to be processed simultaneously within one Kafka consumer instance. This is effective for slow and heavy handler profiles but provides little benefit for zero (no-op) handlers where CPU is the bottleneck.

Interpreting the results

Key metrics:

disp p50 — time from published_at_ns (embedded in the message) to handler entry. For Kafka this includes batch accumulation in the rdkafka producer, network transmission, and consumer fetch latency. Typically 0.2–2 ms at moderate load with default batch settings.
scaling_efficiency — near-linear scaling up to the partition count; flat beyond it because additional consumers get no partition assignment.
CPU% — Kafka's log-structured storage and zero-copy reads keep CPU lower than AMQP backends for high-throughput scenarios.

What to try next

Compare --tier moderate --handler zero throughput here against NATS and InMemory results to see how Kafka's write-ahead log affects raw scheduling cost.
Add --concurrent --handler slow — measure the throughput improvement as concurrent dispatch overlaps the handler I/O within each partition.
Try --tier extreme with 256 consumers — throughput will plateau far below the InMemory extreme because the partition count per scenario is also 256, and Kafka consumer-group rebalance overhead becomes significant.
See the Kafka backend overview for TLS and SASL configuration options relevant to production deployments.