Kafka Consumer Lag Isn't What You Think

I thought I'd caught the villain in the last post. I was wrong, or at least only half right.

After fixing the logging bottleneck, Event Lab still showed lag spiking to 0–400 messages, at only 100 EPS, on a consumer that was demonstrably keeping up. The consumer wasn't behind. The metric was.

What I thought lag was

If you'd asked me to define consumer lag off the top of my head, I'd have said "messages sitting in the queue that the consumer hasn't gotten to yet." It's the intuitive answer, and it's wrong.

What lag actually is

lag = topicHighWatermark − consumerCommittedOffset

The load-bearing word is committed, not consumed. Lag measures the gap to the last offset the consumer told Kafka about, not the last message it actually handled.

Kafka's auto-commit behavior

By default, a consumer commits on a timer, not after each message:

enable.auto.commit: true
auto.commit.interval.ms: 5000 (five seconds)

So the consumer reads and processes a message immediately, but doesn't report that progress to Kafka for up to five seconds. The lag metric shows uncommitted messages, not unprocessed ones; the consumer is keeping up perfectly the whole time.

The timeline that explains the spikes

0.0s  producer: 10 msgs → consumer reads + processes instantly
0.1s  producer: 10 msgs → consumer reads + processes instantly
...
2.0s  metrics poll → lag shows ~200 (processed, but not yet committed)
...
5.0s  auto-commit fires → Kafka updated → lag drops to ~0

The math that proved it

Here's the part that actually felt good: I didn't just fix it, I predicted the number first. If the metric is really just uncommitted messages, then the most lag you could ever see is the commit interval times the produce rate. So:

Before tuning (5s commit, 100 EPS): theoretical max lag = 5 × 100 = 500. Observed: 0–400. ✅
After tuning (auto.commit.interval.ms: 1000, 100 EPS): theoretical max = 1 × 100 = 100. Observed: 0–10. ✅

this.consumer = kafkaInstance.consumer({
  'group.id': this.consumerGroupId,
  'bootstrap.servers': process.env.BOOTSTRAP_SERVERS,
  'auto.commit.interval.ms': 1000, // tighter commits → honest lag metric
})

Why the residual spikes to ~10

The producer micro-batches: 10 events every 100ms (see the architecture post). A metrics poll that lands right after a batch sees 10 uncommitted messages, which the consumer clears in milliseconds. That's the system's natural rhythm, not a problem.

Takeaways

What I'd want you to walk away with:

Lag has multiple meanings: uncommitted-offset lag (what the metric shows) vs. actual processing lag (what you care about). They can diverge a lot.
Defaults shape your dashboards. A 5-second commit interval is fine operationally but makes lag misleading as a real-time signal.
Micro-batching creates natural fluctuation. Bounded spikes are expected; unbounded growth is the real alarm.
Know what you're measuring before you panic about it. "0–400 lag" sounded bad and meant nothing was wrong.

Lag of 0–10 at 100 EPS with one-second commits is, for all practical purposes, real time. The lesson wasn't "tune your commit interval." It was to know what a metric is actually counting before you let its number scare you.

Part of the Event Lab case-study series.