Your Producer Is a Batching Engine, Not a send() Call

Here's a line of code you've written a hundred times:

producer.send(new ProducerRecord<>("orders", orderId, payload));

And here's what it does not do: send anything.

It serializes your key and value, asks a partitioner where the record goes, appends the resulting bytes onto the end of an in-memory batch, and returns a Future. No socket was touched. No broker heard about it. By the time send() returns, your record is sitting in a ByteBuffer in your own JVM's heap, waiting for a different thread - one you never created and probably can't name - to decide it's worth sending.

The Kafka producer is a batching engine with a send()-shaped front door. Every producer behavior that feels random makes mechanical sense once you accept that: the p99 latency that parks itself on a suspiciously round number, the async API that inexplicably blocks, the BufferExhaustedException that isn't really about memory, the throughput that doubles when you make things slower, and the retry that quietly reorders your log.

This post takes the machine apart.

Two threads, and only one of them is yours

The producer is two threads with a queue between them.

Your application thread, inside send(), does four things: run interceptors, serialize the key and value, choose a partition, and append the bytes into the RecordAccumulator. Then it returns. That's the whole job.

The Sender thread is the one doing real work. Its class doc is refreshingly plain about it:

It loops forever: ask the accumulator which partitions have data worth sending, group those partitions by the broker that leads them, build one ProduceRequest per broker, write it to a socket, and complete the futures when responses come back.

So the Future that send() hands you isn't "in flight." It's a claim ticket for work that hasn't started. And the queue between the two threads - the RecordAccumulator - is where every interesting decision happens. Its class doc:

Two promises in one paragraph, and they're in tension: bounded memory, and blocking appends. Hold onto that - it's the whole story of BufferExhaustedException later.

The accumulator: batch.size is an allocation, not a count

Inside the accumulator, every topic-partition gets its own Deque<ProducerBatch>. Appending means: find the deque for this partition, look at the batch at the tail, and try to write the record into it. If it fits, great - the record is now bytes inside an existing buffer. If it doesn't fit, close that batch (it's now sendable) and allocate a new one.

Which brings us to the most misread config in the client:

batch.size=16384    # 16 KB - the DEFAULT

batch.size is not "how many records to batch." It's how many bytes to allocate for one batch buffer. The consequences follow immediately:

• It's a per-partition ceiling, not a global one. Producing to 50 partitions means up to 50 batches of batch.size live at once.
• It's an upper bound, not a target. A batch is sent when it's full or when linger expires, whichever comes first. Under light load, batches leave nearly empty - batch.size never enters the picture.
• A record larger than batch.size doesn't fail. It gets its own oversized batch, allocated outside the pool's fast path. A workload of large records makes batch.size almost irrelevant while quietly wrecking buffer reuse.

The only four reasons a batch gets sent

This is the heart of the machine, and the ready() javadoc states it exactly. A destination broker is ready to receive data if there's at least one partition not backing off, and those partitions aren't muted, and any of the following are true:

That's it. Four triggers. flush() and close() are the fourth. Memory pressure is the third - and note how aggressive it is: when the pool is exhausted, every partition becomes instantly sendable, because the fastest way to free memory is to ship everything and let the responses release the buffers. Under memory pressure the producer abandons batching entirely and flushes in a panic. That's not a bug; it's the release valve. But it means a producer under memory pressure sends small, inefficient batches at exactly the moment it's least able to afford them - a throughput cliff that feeds itself.

For steady-state traffic, only the first two matter: full, or lingered. Every throughput and latency question you have about the producer is really a question about which of those two fires first.

linger.ms: the config that Kafka 4.0 changed out from under you

linger.ms is the artificial delay. When a batch isn't full, the producer waits up to linger.ms for more records to show up so they can travel together.

The classic advice is that linger.ms=0 is the low-latency setting: don't wait, send immediately. It's intuitive, it was the default for over a decade, and it's wrong often enough that Kafka 4.0 changed the default.

linger.ms=5     # Kafka 4.0+ default. It was 0 for the previous decade.

That's KIP-1030, and the reasoning is the good part: the efficiency gains from larger batches typically result in similar or lower producer latency, despite the added linger.

Sit with the shape of that argument, because it's counterintuitive in a useful way. With linger.ms=0, a busy producer emits a flood of tiny requests. Each one costs a request slot, a broker-side request-handler thread, a log append, a replication round, and a response. Queue up enough of them and the broker becomes the bottleneck - so your records wait anyway, just in a queue you can't see instead of a 5ms linger you can. Waiting 5ms to send 10× fewer requests means each request is serviced faster, and the record often comes out ahead end-to-end.

This also means: if you upgraded to Kafka 4.0 and your producer p99 moved, this is why. No code changed. The default did. If you genuinely need sub-millisecond produce latency and you're nowhere near a throughput bound, set linger.ms=0 explicitly - but set it knowing you're trading request efficiency for it, not because zero looks like the fast number.

That branch is the tuning insight most guides bury: linger.ms and batch.size are never both live at the same time. One of them is always dead config. Figure out which trigger is actually firing before you touch either - batch-size-avg tells you in one glance.

The sticky partitioner: how round-robin was silently destroying your batching

Now the part that quietly matters more than either knob, for anyone producing records without keys.

For years, the default partitioner round-robined per record. Record 1 to partition 0, record 2 to partition 1, record 3 to partition 2. It looks fair. It's perfectly balanced. And it is catastrophic for batching.

Batches are per-partition. Round-robin across 30 partitions means 30 concurrent batches, each collecting one record every 30 arrivals. Every batch takes 30× longer to fill, so every batch ships on the linger timer holding a handful of records. The partitioner's "fairness" defeated the entire batching engine sitting behind it - you got 30 tiny requests where you should have had one good one.

The fix (KIP-480, refined by KIP-794) is stickiness, and the partitioner.class doc spells out the current default:

Stick to one partition until you've filled a batch worth of bytes, then switch. You still get even distribution over time - just measured in batches instead of records. Same balance, and batching works again.

But BuiltInPartitioner goes further, with adaptive partitioning (partitioner.adaptive.partitioning.enable, default true). Rather than switching to the next partition in sequence, it switches with probability weighted by how drained each partition's queue is - so faster brokers get more traffic. The source even includes the worked example:

Partition 0 (empty queue) gets 4/8 of the traffic. Partition 1 (backed up with 3 batches) gets 1/8. The producer routes around a slow broker automatically, in proportion to how slow it is. This is load balancing you didn't configure and probably didn't know you had.

The catch to all of this: stickiness only applies to records without keys. A keyed record goes wherever hash(key) says, always, because that's the ordering guarantee you're paying for. If you produce keyed records to 200 partitions at a low rate, you get the round-robin batching problem back and no partitioner can save you - the keys have already decided.

The BufferPool: why your async producer blocks

buffer.memory (default 32 MB) is the total memory the accumulator may hold. It's managed by BufferPool, which has two properties worth knowing:

The poolable size is batch.size. That's the hidden coupling between these two configs: allocate a batch.size buffer and you get a recycled one off the free list, free. Allocate anything else - because a record was bigger than batch.size - and you take a fresh allocation and the GC churn that comes with it. batch.size isn't just a batching knob; it's the size at which your producer's memory is free to reuse.

Now the part that surprises people. BufferPool.allocate:

Your "asynchronous" producer blocks. Right there, inside send(), on your application thread. When the accumulator is full, send() waits for the Sender to ship batches and release buffers. It waits up to max.block.ms (default 60 seconds - that's a full minute your request thread can be parked inside a call you believe is non-blocking). When that expires:

org.apache.kafka.clients.producer.BufferExhaustedException:
  Failed to allocate 16384 bytes within the configured max blocking time 60000 ms.
  Total memory: 33554432 bytes. Available memory: 0 bytes. Poolable size: 16384 bytes.

And here is the thing to understand about that exception: it is almost never about memory. BufferExhaustedException extends TimeoutException, and that inheritance is the real diagnosis. Memory fills up because the Sender isn't draining it fast enough. The Sender isn't draining fast enough because the brokers aren't acking fast enough - or the network is slow, or a partition leader is down, or you're blocked on metadata for a topic that doesn't exist yet.

BufferExhaustedException is backpressure. It is the producer telling you your application is producing faster than your cluster is accepting, sustained long enough to fill a 32MB buffer and then wait a further 60 seconds. Raising buffer.memory when you see it is treating a symptom: you'll buy a slightly longer runway and hit the same wall, later, with more data at risk in a JVM that can still crash. Go find out why the Sender is slow.

Ordering: max.in.flight=5 and the guardrail that makes it safe

Last mechanism, and it's the one with teeth.

max.in.flight.requests.per.connection defaults to 5: the producer will have up to five unacknowledged ProduceRequests on a single connection at once. That's a big part of producer throughput - waiting for each response before sending the next request would put a full network round-trip between every batch.

But pipelining plus retries is a reordering machine. Requests 1 and 2 are both in flight. Request 1 fails (a transient NOT_LEADER_OR_FOLLOWER during a leader election). Request 2 succeeds. The producer retries request 1, which now lands after request 2. Your records are in the log backwards, permanently, and nobody errored. The max.in.flight doc says it plainly:

The historical advice - "set max.in.flight=1 if you need ordering" - works by making the pipeline one deep, so there's nothing to reorder against. (That's what the muted set in the accumulator implements: a partition with a request in flight is muted so no second batch can be drained for it.) But it costs you most of your throughput.

Idempotence is the modern answer, and it's on by default. The idempotent producer tags every batch with a producer ID and a per-partition sequence number. The broker tracks the last sequence it accepted per partition and rejects anything out of order. If request 1 is retried after request 2 arrived, the broker sees a sequence gap and refuses request 2 rather than writing it out of order - and the producer sorts it out. Ordering is enforced at the broker, so the client is free to pipeline. This is why the limit is exactly 5: it's the deepest pipeline the broker's sequence-tracking window supports. Ask for more and the config throws:

ConfigException: To use the idempotent producer, max.in.flight.requests.per.connection
must be set to at most 5. Current value is 10.

That's a loud, immediate failure. Good. But its neighbors in the same validation method are the real trap:

if (retries == 0) {
    if (userConfiguredIdempotence) throw new ConfigException("Must set retries to non-zero...");
    log.info("Idempotence will be disabled because {} is set to 0.", RETRIES_CONFIG);
    shouldDisableIdempotence = true;
}
final short acks = Short.parseShort(acksStr);
if (acks != (short) -1) {
    if (userConfiguredIdempotence) throw new ConfigException("Must set acks to all...");
    log.info("Idempotence will be disabled because {} is set to {}, not set to 'all'.", ACKS_CONFIG, acks);
    shouldDisableIdempotence = true;
}

Debugging the batching engine

Every producer symptom maps to one of the mechanisms above. The metrics live in kafka.producer:type=producer-metrics.

The two-question diagnostic that resolves most cases:

"Producer latency is high." Compare record-queue-time-avg against request-latency-avg. If queue time dominates, the records are waiting in your JVM - that's linger, or backpressure from a full buffer, and it's yours to fix. If request latency dominates, the records are waiting at the broker - and linger.ms is not your problem. People tune linger to fix broker-side latency constantly and are baffled when nothing improves.

"Throughput is too low." Look at batch-size-avg against your batch.size. Far below it? You're linger-bound: records aren't arriving fast enough to fill a batch, so raise linger.ms (yes, raise it) or check whether an unkeyed workload is being spread thin across partitions. Pinned at batch.size? You're size-bound: raise batch.size and enable compression, because you're already filling every batch and the win is in making each request carry more.

The reframe

Stop reading producer.send() as "send this record." Read it as "hand this record to a batching engine and let it decide."

Once you do, the configs stop being a list of tuning knobs and become a description of one machine. batch.size is how big a batch buffer is, and also the size at which memory gets recycled. linger.ms is how long the engine waits for a batch to fill. buffer.memory is how much unsent data may pile up before send() starts blocking. The partitioner decides how many batches you're filling in parallel, which decides whether any of the others matter. And max.in.flight decides how deep the pipeline runs, which is safe only because idempotence enforces order at the far end.

None of them are independent, because none of them are really configs. They're dimensions of a single buffering system that Kafka put a send() in front of - and the front door is the only part of it that ever looked simple.

This is the client-side half of the produce path. For what happens after the bytes arrive - how the broker decides a write is actually safe - see acks=all and the ISR. The sequence numbers that make max.in.flight=5 safe are the first of the three mechanisms in exactly-once semantics. For the consumer side, see consumer group rebalancing, and for how the cluster tracks its own state, see KRaft. The foundations underneath all of it are in the ingestion and transport curriculum.