Everyone treats Parquet as 'the fast columnar format' and stops there. But two files with identical rows can differ tenfold in scan cost, and the reason is never in the data - it's in the layout. Why readers open the file at the end, row group and page sizing, dictionary encoding and its silent fallback, whether min/max statistics can actually be trusted, and whether a page index exists at all: these are the dials that decide your query bill. This is a tour of the bytes most people never look at, and a free, 100%-in-browser way to look at your own.
The single most useful thing you can learn about the Kafka producer is that send() is not a send. It's an append to an in-memory buffer, plus a promise. A background thread you never created decides when bytes actually hit a socket, and it decides using rules that have nothing to do with your call site. That gap is where every confusing producer symptom lives: the p99 that sits suspiciously close to a round number, the async API that mysteriously blocks, the BufferExhaustedException that isn't really about memory, and the retry that quietly reorders your log. This post takes the producer apart - RecordAccumulator, Sender, BufferPool, BuiltInPartitioner - and rebuilds it as what it actually is: a batching engine with a send()-shaped front door. Includes the Kafka 4.0 change that flipped linger.ms from 0 to 5 and what it says about the whole design.
Kafka spent fourteen years telling everyone that a replicated, ordered log was the right way to move state between systems - and the whole time, it kept its own metadata in a ZooKeeper tree, pushed to brokers over RPCs that could half-arrive and leave the cluster in a divergent state. KRaft is the moment Kafka started eating its own dog food: the controller quorum is a Kafka topic, the active controller is just the leader of that topic's only partition, and every broker is a consumer of it. That one idea collapses a whole category of problems - failover stops being a reload and becomes a replay - but it also means metadata now has offsets, epochs, a high watermark, and lag, and you debug it with the same instincts you use for any other Kafka log. This post builds KRaft from the log up: why the source code calls it 'a Kafkaesque version of the Raft protocol,' why replication is driven by fetch requests, how snapshots put a floor under an infinite log, and what to look at when a controller election goes wrong.
Most explanations of Kafka transactions stop at 'atomic writes across partitions,' which is true and tells you almost nothing about what is happening. A transaction is not a buffer the broker holds until commit, and it is not a lock. Your records are appended to their partitions the instant you send them, interleaved with everyone else's, visible on disk. What makes them a transaction is a separate machine: a transaction coordinator running a two-phase-commit state machine whose state lives in an internal replicated log, and a single control marker it stamps into each partition at the end. The commit never touches your records. It writes one marker, and the reader does the rest. This post builds that machine from the source up: the transactional.id and coordinator selection, the protocol RPC by RPC, the full state enum including the epoch-fence state, control markers and coordinator epochs, the Last Stable Offset and aborted-transaction index that make it visible, and what KIP-890's Transaction V2 changed about all of it.
Everyone wants exactly-once, most people think it's a checkbox, and almost nobody can say what the checkbox actually does - because there is no checkbox. "Exactly-once" in Kafka is three separate mechanisms that share a name: an idempotent producer that dedupes its own retries within a single session and partition; transactions that make writes to many partitions plus the consumer's offset commit atomic across sessions and fence zombie producers; and read-committed isolation that stops consumers from ever seeing uncommitted or aborted records. They solve different problems, fail in different ways, and - critically - are not all on by default. Idempotence is on; transactions and read-committed are opt-in, so the out-of-the-box producer is exactly-once in a much narrower sense than the phrase implies. This post builds all three from the protocol up - producer IDs, epochs, sequence numbers, the two-phase commit state machine, control markers, the Last Stable Offset - and then walks the failures each one throws, because knowing which of the three you're actually relying on is the whole game.
Choosing between SNAPPY and ZSTD is the first Parquet tuning decision most people make and the one they understand least. The usual advice ranks codecs on a ratio-vs-speed chart, as if the codec were compressing your dataset. It isn't. It sees roughly one megabyte at a time, of one column, already dictionary-encoded and run-length-encoded by the time it arrives. That constraint explains why the ratios disappoint, why an encoding exists whose entire purpose is to make the codec's job easier, and why parquet-java's own default codec is none at all.
Almost everyone ships the same 'safe' config: replication.factor=3, acks=all, and a comfortable feeling that a committed message survives a broker dying. Then a broker dies and the message is gone anyway - and the config looks fine in the postmortem. The trap is that acks=all is a promise about the in-sync replica set (the ISR), not about your three replicas, and the ISR is a live set that shrinks whenever a follower lags. With the default min.insync.replicas=1, a shrunk ISR of just the leader still satisfies acks=all - so you're running acks=1 without a single config saying so. This post builds the real durability model from the replication protocol up: the high watermark that defines 'committed', min.insync.replicas as the actual floor, leader epochs that stop log divergence, unclean leader election as the availability-vs-safety switch, and the newest fix, Eligible Leader Replicas. Durability was never one setting - it's how five mechanisms interact.
Two years ago, Iceberg vs Delta Lake was a feature bake-off: who has row-level deletes, who has time travel, who has schema evolution. That comparison is dead. Both formats now have deletion vectors, row lineage, column mapping, and merge-on-read - they arrived at the same place from opposite directions, Iceberg through a version dial and Delta through table features. So the real decision has moved. It's not 'which format has feature X' anymore; it's which ecosystem you want to live in, which catalog you bet on, how much of your stack is Databricks, and how much the new interop layer lets you defer the choice entirely. This is that decision guide - what genuinely differs, what doesn't, and how to pick without regret.
Most engineers meet Delta Lake as 'the thing that gives Spark ACID transactions' and stop there. But underneath is one idea - an append-only transaction log of atomic commits - and a version story that looks nothing like Iceberg's clean dial. Delta grew by a ladder of protocol versions (writer versions 1 to 7, reader versions 1 to 3), bolting one capability onto each rung, until the ladder itself became the bottleneck. Writer 7 and reader 3 replaced it with table features: an a-la-carte set of named capabilities a table opts into individually. This is how the log works, how the ladder was climbed, and how deletion vectors, row tracking, column mapping, clustering, variant, type widening, and catalog-managed tables all hang off that pivot.