Back-of-the-Envelope for Data

"How big is this?" the interviewer asked. Priya said "a lot" and watched the room deflate. She knew the system cold, but she'd skipped the one move that turns a story into engineering: a number. Without it, every later choice ("we'll stream it", "we'll partition by hour") was a guess. With it, the choices make themselves.

Estimation isn't trivia and it isn't about being right to two decimals. It's about finding the single number that changes the design. Sometimes that number says "this fits on one machine, stop overengineering." Sometimes it says "streaming this would cost more than the team, do it in batch." You're hunting for that number.

Memorize five unit conversions

You don't need a calculator. You need a handful of conversions you can chain in your head. These five carry almost every data estimate:

• 1 event/sec ≈ 86,400/day ≈ 2.6M/month. (Round 86,400 to "about 100k/day" in your head.)
• 1 KB × 1 million = 1 GB. So a million 1 KB events is a gigabyte. Clean.
• 1 GB/day ≈ 365 GB/year ≈ roughly a third of a TB a year.
• 1 MB/s ≈ 86 GB/day ≈ 31 TB/year. (Throughput to storage, the conversion people forget.)
• Columnar compression ≈ 3x to 10x over raw JSON. Use 5x as a default when you don't know.

Chain them. "6,000 events/sec" times "1 KB" is "6 MB/s", which by the fourth conversion is "about 500 GB/day", which by the third is "roughly 180 TB/year raw." All in your head, all good enough.

The volume and velocity walk

Always go in the same order: events per second, bytes per event, then derive the rest. Worked for PhoenixWorld's clickstream:

• Velocity. 500 million events/day. Divide by ~86,400 → about 5,800 events/sec average. Real traffic isn't flat, so assume a peak of 4 to 5x → ~25,000 events/sec at peak. That peak number, not the average, sizes your ingestion.
• Size. A clickstream event as JSON is maybe 1 KB. (JSON is fluffy. The same event in a packed binary format is often a third of that.)
• Raw per day. 500M × 1 KB = 500 GB/day raw.
• Columnar per day. At 5x compression in Parquet → ~100 GB/day.
• Per year. 100 GB/day × 365 → ~36 TB/year stored, growing daily.

Five numbers, two minutes, no calculator. Now every architecture claim has a foundation. "25k events/sec at peak" tells you how many Kafka partitions. "100 GB/day" tells you whether this is a laptop problem or a cluster problem (it's a cluster problem). "36 TB/year" tells you the storage bill is real but not scary.

Why columnar changes the number

That 5x is not a footnote. It's often the difference between "affordable" and "no". Columnar formats like Parquet store each column together, so values that look alike sit next to each other and compress hard. They also let a query read only the columns it needs and skip whole chunks of rows using per-chunk min/max stats.

A worked intuition. Say each clickstream event has 40 fields but the funnel dashboard only reads 4 of them. In row storage, a scan reads all 40 fields of every row to get the 4. In Parquet, the reader opens the file's metadata, sees the 4 columns it wants, and reads only those column chunks. Then within each chunk, the min/max stats let it skip row groups whose values can't match the filter. You can touch a fraction of the bytes for the same answer.

Partitions and the small-files trap

Once you know it's ~100 GB/day, you decide the physical layout. Two numbers fight each other:

• Target file size: aim for 128 MB to 512 MB per Parquet file. Big enough that the per-file overhead (open, read footer, plan) is amortized; small enough to parallelize.
• Partitioning: split data into directories by a column you filter on, usually date. 100 GB/day partitioned by date is one day's worth per partition, which at 256 MB files is about 400 files/day. Healthy.

Now watch what over-partitioning does. Partition by date and country and event_type, and that 100 GB shatters across thousands of tiny combinations. A partition that should be 256 MB becomes 200 KB. Now a query opens ten thousand tiny files, and the planning overhead dwarfs the reading. This is the small-files problem, and it's the single most common self-inflicted performance wound in data engineering.

The rule of thumb: partition by the column you filter on most (almost always a date), and stop. Reach for a second partition column only when a single partition is still too big to scan and you reliably filter on that second column too.

Throughput sizes the moving parts

Storage is the easy half. The harder half is whether the pipes are wide enough.

A handy rule: one Kafka partition comfortably handles ~10 MB/s, and a consumer reads one partition at a time, so partitions set your parallelism. PhoenixWorld's peak is ~25k events/sec × 1 KB = ~25 MB/s. For raw throughput that's only 3 partitions, but you'll want more (say 12 to 24) so consumers can parallelize and you have headroom for a hot key or a replay. Partition count is a throughput-and-parallelism decision, not a guess.

The same logic sizes a Spark job. Spark's parallelism is its partition count, and each task wants a chunk it can hold in memory. 100 GB shuffled across 200 tasks is 500 MB per task, comfortable. Across 20 tasks it's 5 GB per task and you're spilling to disk or dying. When someone asks "how many partitions", the honest answer is "total size divided by a target task size of a few hundred MB."

Reference numbers worth carrying

Order-of-magnitude, for sizing in your head. Don't quote them as precise; use them to know what's cheap and what's expensive.

The two lessons buried in that table: per-request latency to object storage is high, but bulk throughput is enormous (so prefer few big files), and cross-region is 100x slower than same-AZ (so design for replication, never for synchronous cross-region reads).

A cost model you can do out loud

You rarely need exact pricing. You need to know which line item dominates. Four levers:

• Storage: roughly $0.02 per GB-month for standard object storage. PhoenixWorld's year of clickstream, ~36 TB, is about $700 to $800/month at full year. Real, but not the thing that sinks you.
• Compute: cluster hours. A nightly Spark job over 100 GB might be an hour on a modest cluster. This usually dominates if you reprocess a lot.
• Query/scan: serverless query engines often bill ~$5 per TB scanned. This is where columnar and partitioning pay rent: a query that scans 5 GB instead of 500 GB is 100x cheaper, every time it runs.
• Egress: moving data out of a cloud or across regions. Small until it isn't. Cross-region replication of 100 GB/day adds up.

The senior move is to say which lever dominates and design against it. For PhoenixWorld clickstream, storage is cheap and scan is the risk, so you spend your design effort on partitioning and pre-aggregation, not on shaving storage.

Practice the reflex

Before you read another problem page, build the habit: whenever you hear a data system described, silently run the walk. Events/sec, bytes/event, GB/day, TB/year, peak throughput, dominant cost. Thirty seconds, no calculator. Do it enough and "how big is this?" stops being a question that deflates the room and becomes the move that wins it.

Next up, The Building Blocks: the dozen primitives you compose into any of these systems, and the honest price tag on each.