Kafka Performance Testing

Generate a JMeter test plan to load test a Kafka producer using the JMeter Kafka plugin (pepper-box or kafka-jmeter):

Kafka cluster: [broker1:9092, broker2:9092]
Topic: [topic-name]
Number of partitions: [N]
Replication factor: [N]
Message format: [JSON / Avro / Plain text]
Sample message payload:
[Paste message JSON or schema]

Requirements:
- Thread Group: [N] producer threads, [N]s ramp-up
- Target throughput: [N] messages/second
- Message size: ~[N] KB
- Parameterize key fields using CSV or Groovy
- Acks configuration: [0 / 1 / all]
- Compression: [none / gzip / snappy / lz4]
- Linger.ms and batch.size tuning

Provide:
- Full JMX XML with Kafka Sampler configuration
- Producer properties (key.serializer, value.serializer, acks, retries)
- How to measure producer latency and throughput in JMeter
- Expected Kafka broker metrics to monitor during test

Design a performance test for a Kafka consumer group:

Topic: [topic-name]
Consumer group: [group-id]
Expected message rate: [N] messages/second
Consumer lag SLA: < [N] messages behind

Provide:
- How to measure consumer lag using kafka-consumer-groups.sh
- k6 or JMeter approach to simulate concurrent consumers
- Kafka consumer benchmark using kafka-consumer-perf-test.sh
- Key consumer config tuning: fetch.min.bytes, max.poll.records, max.poll.interval.ms, session.timeout.ms
- Grafana / Prometheus metrics to monitor: kafka_consumer_group_lag, kafka_consumer_records_consumed_rate
- Alert thresholds for consumer lag during a load test
- How to identify a slow consumer vs a slow broker

I need to measure end-to-end latency through a Kafka pipeline:

Flow: [Producer] → [Topic A] → [Consumer/Service] → [Topic B] → [Final Consumer]

Target: p99 end-to-end latency < [N]ms at [N] messages/second

Design a test that:
- Embeds a timestamp in message payload at producer side
- Measures time-to-consume at the final consumer
- Calculates end-to-end latency per message
- Accounts for clock skew between producer and consumer machines
- Plots latency distribution under increasing message rates

Provide:
- JMeter / k6 producer script with timestamp injection
- Consumer-side measurement approach (custom consumer app or kafka-streams)
- Latency calculation formula and methodology
- Results analysis: how to identify which stage in the pipeline adds the most latency
- Common Kafka latency bottlenecks: network, disk I/O, GC, partition imbalance

Help me capacity plan a Kafka cluster for the following workload:

Peak message rate: [N] messages/second
Average message size: [N] KB
Retention period: [N] hours / days
Replication factor: [N]
Number of consumer groups: [N]
Expected peak network throughput: [N] MB/s

Calculate:
- Required disk space per broker (with formula)
- Required network bandwidth per broker
- Recommended number of partitions per topic
- Recommended number of brokers
- JVM heap size for each broker
- os.page.cache sizing recommendation
- Key broker configs to tune: log.retention.bytes, num.io.threads, num.network.threads, socket.send.buffer.bytes
- How to run kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh to validate capacity

My Kafka producers use Avro serialization with Confluent Schema Registry.

Schema Registry URL: [http://schema-registry:8081]
Avro schema:
[Paste Avro schema JSON]

Performance concerns:
- Schema Registry lookup adds latency on first message per schema
- Serialization/deserialization CPU cost at high throughput

Provide:
- JMeter Kafka sampler configuration for Avro messages
- How to pre-warm schema registry cache before test
- Benchmark: Avro vs JSON serialization throughput comparison approach
- Schema Registry performance tuning: caching, replication
- Monitoring Schema Registry health during load test
- Common Avro serialization errors under load and fixes

My Kafka load test shows the following symptoms:
[Choose / describe: high producer latency / consumer lag growing / under-replicated partitions / broker CPU spike / OOM on broker / network saturation]

Walk me through diagnosis:
1. Kafka broker JMX metrics to inspect: UnderReplicatedPartitions, RequestHandlerAvgIdlePercent, NetworkProcessorAvgIdlePercent
2. OS-level checks: disk throughput (iostat), network (sar, netstat), file descriptor limits
3. JVM GC analysis for Kafka brokers: which GC, heap sizing, GC pause impact on produce latency
4. Producer-side investigation: record-error-rate, record-retry-rate, batch-size-avg
5. Consumer-side investigation: fetch-latency-avg, records-lag-max
6. Partition rebalancing impact during test
7. Step-by-step fix recommendations with specific config property changes