Rakeem Thomas

Senior Software Engineer

Writing

Designing a Real-Time Kafka Load Simulator

I've run Kafka in production, but a lot of the engineers I work with hadn't touched it, and the same questions kept coming up. What's consumer lag? What are partitions? Can lag be different across partitions? Wait, there are consumer groups too? And the one that really lands in practice: "I can see this message was produced, but this consumer's database still has stale data. What's going on?" That gap between "produced" and "actually reflected downstream" is exactly what lag is, and it's hard to explain in the abstract. I wanted something I could point people to: a way to watch consumer lag build and see how it ripples into the systems reading behind it.

Event Lab is an interactive Kafka load simulator. You drag a slider to set a target throughput and watch producer rate, per-partition consumer lag, and backpressure react in real time. This post is the architecture deep dive: how the pieces fit together, and the handful of decisions that turned out to matter.

The problem with most Kafka demos

Most explanations of Kafka stop at a static architecture diagram and a hello-world producer. You can read all of it and still have no feel for what lag actually is: how fast it grows, what makes it recede, why it can spike while a consumer is keeping up perfectly fine. I wanted the behavior on a screen and moving, so those questions answer themselves.

The goal was the opposite of a diagram: make the behavior of an event-driven system legible. Turn the load up, turn it down, and see exactly where the system bends.

The shape of the system

Event Lab is a Turborepo monorepo with four services, plus Redpanda as the Kafka-compatible broker. Everything comes up with a single docker compose up.

Browser
  │  HTTP (set throughput, poll metrics)
  ▼
Web dashboard (Next.js)
  │  POST /control/throughput
  ▼
Control API (Express)
  │  publish → control.producer topic
  ▼
Redpanda  ──topics──▶  control.producer (1p)   events.input (3p)
  │                    events.input.dlq (1p)    metrics.events (1p)
  ├──▶ Event Producer (Node)  consumes control.producer, produces events.input
  └──▶ Event Consumer (Node)  consumes events.input, tracks lag

The two backend services are the interesting ones; the API is a thin control surface and the dashboard is a poller. Stack: Next.js + Tailwind on the front, Express + Node producer/consumer on the back, all in TypeScript, talking to Redpanda through @confluentinc/kafka-javascript.

Control plane / data plane separation

I reached for two topics on instinct and justified it afterward, but the instinct holds up: control traffic and data traffic shouldn't have to fight over the same pipe. A command to slow down is worthless if it's stuck in line behind the flood it's trying to stop.

Control commands and data events travel through separate topics. When you move the slider, the API publishes a small command to control.producer:

// What the API drops onto the control topic
{ commandType: "SET_THROUGHPUT_TARGET", targetEps: 500, issuedAt: "..." }

The producer consumes that topic and reconfigures itself. Meanwhile the actual synthetic events flow through events.input (3 partitions). The two never share a pipe, so a flood of data events can never delay a control command, the same reason real systems keep their control plane separate.

What happens when you drag the slider

There's a deliberate amount of machinery between the slider and Kafka:

  1. The slider updates local state immediately and shows a pending badge.
  2. A 500ms debounce waits for you to stop dragging before firing anything; otherwise dragging 0 → 1000 would spam the API with every intermediate value.
  3. The debounced value POSTs to the control API; the badge flips to syncing.
  4. The API validates it and publishes a command to control.producer.
  5. The producer consumes the command and retunes its send loop.
// The whole debounce hook, small, but it's what keeps the API from
// being hammered on every pixel of slider movement.
export function useDebounce<T>(value: T, delay: number): T {
  const [debounced, setDebounced] = useState<T>(value)
  useEffect(() => {
    const handler = setTimeout(() => setDebounced(value), delay)
    return () => clearTimeout(handler)
  }, [value, delay])
  return debounced
}

Micro-batching: hitting a target rate without melting the broker

The naive way to produce 1,000 events/second is a loop that sends one event a millisecond. That's 1,000 individual broker round-trips per second, and the overhead dominates. Instead the producer works in 100ms ticks and sends a batch each tick:

const tickMs = 100
const eventsPerTickAvg = targetEps * (tickMs / 1000) // 1000 EPS → 100 / tick
let pendingEvents = 0

this.producerInterval = setInterval(async () => {
  // Carry the fractional remainder so low targets still average out:
  // 5 EPS → 0.5 events/tick → a batch of 1 every other tick, not 0 forever.
  pendingEvents += eventsPerTickAvg
  const eventsThisTick = Math.floor(pendingEvents)
  pendingEvents -= eventsThisTick
  if (eventsThisTick === 0) return

  const batch = Array.from({ length: eventsThisTick }, () => ({
    key: 'test-key',
    value: JSON.stringify({ key: 'test-key', value: 'hello this is a test' }),
  }))
  await this.producer.send({ topic: EVENTS_TOPIC, messages: batch })
  this.metricsAccumulator.recordAcknowledged(batch.length)
}, tickMs)

I found this the irritating way. I dragged the slider down to 5 EPS to check the low end and got nothing, dead silence. 5 × 0.1 = 0.5, Math.floor(0.5) = 0, every tick, forever. Carrying the leftover fraction between ticks was a two-line fix for a bug that would otherwise have quietly rounded every small target down to zero.

Two details worth noting. The producer runs with acks: -1 (wait for all in-sync replicas), so "sent" means "durably acknowledged," not "fired and forgot." And the fractional-remainder carry means a target of 5 EPS produces the right rate on average instead of flooring to zero events every tick.

Measuring throughput honestly

On the first pass I counted send attempts, because that was the number sitting in front of me. But "I called .send()" and "the broker durably has this" are two different claims, and only the second one is throughput. Counting acknowledgments keeps the dashboard from flattering itself.

The "actual EPS" number on the dashboard counts Kafka acknowledgments, not send attempts, over a 2-second rolling window that resets for instant feedback when you change the target:

const elapsedSeconds = (now - this.windowStartTime) / 1000
this.lastCalculatedEps = Math.round(this.windowEventCount / elapsedSeconds)
// then reset the window

A cumulative average was the obvious first instinct and the wrong one. It lags badly when you adjust the target, because old data points refuse to let go. The rolling window snaps to the new rate within a couple of seconds.

Consumer lag, and the round-trip that didn't fit

Lag is the headline metric: for each partition, highWatermark − committedOffset, summed across partitions. The honest definition matters here: it's the committed offset, not the last message the consumer touched.

const [topicOffsets, groupOffsets] = await Promise.all([
  this.adminClient.fetchTopicOffsets(EVENTS_TOPIC),
  this.adminClient.fetchOffsets({ groupId, topics: [EVENTS_TOPIC] }),
])

// per partition: high watermark minus where the group has committed
const lag = Math.max(0, Number(high.high) - consumerOffset)

The catch: a Kafka admin round-trip against single-broker Redpanda takes roughly 3–5 seconds. The dashboard polls every 2. Computing lag inside the request path would mean every metrics call blocks on a broker call slower than the poll interval itself. So lag is computed on a background poller (every 1.5s) and the /metrics endpoint serves the cached snapshot, HTTP latency decoupled from broker latency.

There's something genuinely funny about discovering that your measurement is slower than the thing it measures: a 3–5 second admin round-trip reporting on a number that moves every 100ms. Pushing it onto a background poller wasn't a clever trick. It was the only move left once real broker latency met a real poll interval, and it's exactly the kind of lesson a diagram can't hand you.

Keeping a public demo cheap to run

Personal projects almost never have to survive a stranger. A public demo does. Someone will drag the slider to max, close the laptop, and forget the tab exists for three days. Designing for that abandoned tab, rather than the happy path I'd click through myself, quietly changed what "finished" meant.

A public, interactive demo has a failure mode personal projects usually don't: someone sets it to max, walks away, and leaves it running. Two guardrails:

  • A hard MAX_EPS clamp in the producer. Even if the API validation is bypassed, the producer refuses to exceed the cap. Locally the slider goes to 10K, but the deployed demo is clamped well below that.
  • An auto-pause watchdog. If no control message arrives for 60 seconds, the producer stops producing on its own. An abandoned tab winds down instead of billing throughput forever.
// Defensive clamp: never trust that upstream validation ran
const maxEps = Number(process.env.MAX_EPS ?? 200)
const targetEps = Math.min(Math.max(0, Number(requestedEps) || 0), maxEps)

Two things the system taught me

Two detours were interesting enough to earn their own posts:

  • A console.log was my bottleneck. Throughput stalled around 200 msgs/sec on a consumer that should handle 10,000+. The culprit wasn't Kafka. It was logging every message. Console writes are milliseconds; at that rate the CPU spends its life waiting on I/O. Batching logs to one every 10,000 messages erased the lag. (Full write-up.)

  • Lag isn't what you think it is. After fixing I/O, lag still spiked to 0–400 at only 100 EPS. The metric is highWatermark − committed, and Kafka's default auto-commit interval is 5 seconds, so the consumer was keeping up perfectly and the metric was just 5 seconds stale. Dropping auto.commit.interval.ms to 1000 made the dashboard tell the truth. (Full write-up.)

What's next

Event Lab is very much a work in progress, and honestly that's the point: it's built to keep misbehaving in new ways as I add to it. The events.input.dlq topic is wired but not yet used; the next milestone is consumer-side failure injection and dead-letter routing, so you can watch poison messages and retries the same way you watch throughput today. Metrics polling is also slated to become Server-Sent Events for sub-second updates.

Source on GitHub · Live demo at event-streaming-dashboard-web.vercel.app.