Why Distributed Computing?

Priya pulls up the orders file her data team dropped to S3 last night. The CEO wants a daily revenue report by category, top sellers by region, a 7-day click-to-purchase funnel. She's done this kind of work a hundred times.

import pandas as pd
df = pd.read_csv("shopstream_orders_all_time.csv")

The fans spin up. The memory bar fills. A minute later the kernel dies and her browser tabs are gone. The file is 50GB. Her laptop has 16.

Priya isn't bad at her job. The machine isn't broken. The tool she reached for was built for a world where the data fits on one box, and ShopStream has outgrown that world. What does it actually take to handle data that doesn't?

2. Concept Explanation

The Analogy: One Chef vs. a Brigade

Imagine you're catering a wedding for 2,000 guests. You hire one chef. This chef is extraordinarily skilled, the best in the country. But they're one person with two hands. They can't chop 500 onions in parallel. They can't simultaneously boil 40 pots. No matter how talented they are, there's a hard ceiling on throughput: one brain, one body, one set of hands.

Now imagine instead you hire a brigade of 50 cooks, a head chef (the coordinator), and a prep kitchen with 50 stations. Each station handles a portion of the work. Onions get split 50 ways. The head chef coordinates, assigns tasks, and combines the results into the final plates.

This is exactly what distributed computing does. Your laptop is the single chef. A cluster of machines is the brigade.

Why a Single Machine Hits a Wall

There are three fundamental physical bottlenecks on a single machine:

1. RAM (Memory)

RAM is fast, reads happen in nanoseconds. But it's finite. A typical server might have 256GB, 512GB, or even 2TB of RAM at the extreme high end. But data grows faster than hardware. ShopStream's order history is 50GB today. In two years, with user growth, it could be 2TB. You cannot buy your way out of this forever. Even if you could, 2TB servers are extraordinarily expensive and represent a single point of failure.

When data exceeds RAM, the OS starts using virtual memory, it pages data to disk. Disk I/O is 100,000x slower than RAM access. Your 50GB CSV analysis doesn't run at memory speed; it crawls at disk speed. This is what Priya experienced.

2. CPU Parallelism Limits

Modern CPUs have multiple cores, 8, 16, 32, sometimes 64 on high-end workstations. But even 64 cores is a ceiling. If your computation needs to scan 500GB of data, sort it, join it, and aggregate it, a 64-core machine is still constrained by the laws of physics: heat dissipation limits clock speed, memory bandwidth limits how fast data can flow to the cores, and Amdahl's Law limits the speedup you get from parallelism based on the sequential portions of your program.

With a cluster of 100 machines each with 32 cores, you have 3,200 parallel workers. With 1,000 machines, you have 32,000. This doesn't scale linearly due to coordination overhead, but it scales far better than buying a bigger single machine.

3. Disk I/O

Reading 50GB from a local SSD might take 2-3 minutes at peak throughput (~500MB/s). But a distributed file system like HDFS (Hadoop Distributed File System) or cloud object stores like S3, when accessed by 100 machines in parallel, can achieve aggregate throughput of hundreds of GB/s. The data is physically striped across disks on many machines, and they all read simultaneously.

Vertical vs. Horizontal Scaling

Vertical scaling (scale-up): Buy a bigger machine. Add more RAM, faster CPUs, larger disks. This is simple to reason about, it's still one machine, one OS, one process. But it's expensive, has hard physical limits, and a single machine is a single point of failure. If it crashes, everything stops.

Horizontal scaling (scale-out): Add more machines. Connect them with a network. Distribute the work across all of them. This is what Spark does. The machines can be commodity hardware, nothing exotic. If one machine fails, you still have 99 others. You can add capacity incrementally.

Here's a concrete comparison:

What "Distributed" Actually Means

When we say "distributed computing," we mean:

1. The data is split across multiple machines. A 1TB dataset might be split into 200 chunks of 5GB each, spread across 20 machines (10 chunks per machine).
2. Computation runs on each machine against its local chunk. No single machine sees all the data.
3. A coordinator collects and merges partial results. The "driver" (in Spark terms) assembles the final answer.

This works beautifully for what's called embarrassingly parallel workloads: "compute the revenue for each order independently" requires no communication between machines during the computation phase. Each machine just processes its chunk.

It gets harder for operations that require data from multiple chunks, like sorting the entire dataset, or joining two large datasets together. These require a shuffle: moving data between machines over the network. We'll cover shuffles in depth in Lesson 05.

A Brief Mention: The CAP Theorem

When you have multiple machines storing data, you face a fundamental tradeoff described by the CAP theorem (Brewer, 2000):

You cannot simultaneously guarantee all three of:

• Consistency: Every read sees the most recent write
• Availability: Every request receives a response
• Partition tolerance: The system keeps working despite network failures between nodes

Spark is a compute engine, it doesn't store data long-term, so it doesn't need to make this tradeoff directly. But the storage systems it reads from (HDFS, S3, Cassandra, Kafka) do. Understanding CAP helps you reason about why distributed systems sometimes return stale data or fail in unexpected ways.

Why Spark Beats Hadoop MapReduce

Hadoop MapReduce (released by Yahoo in 2006, open-sourced shortly after) was the first widely-used distributed compute framework. It solved the horizontal scaling problem. But it had a critical design flaw: every intermediate result was written to disk.

Consider a multi-step computation:

1. Parse log lines (Map)
2. Filter for errors (Map)
3. Count by error type (Reduce)
4. Sort by count (another MapReduce job)
5. Take top 10 (another MapReduce job)

In Hadoop MapReduce, each step writes its output to HDFS before the next step reads it. For a 5-step pipeline, you're doing 10 full disk reads/writes of potentially terabytes of data. Disk I/O becomes the bottleneck again.

Spark's key insight (from the 2010 RDD paper by Matei Zaharia at UC Berkeley): keep intermediate results in memory. In Spark, the output of step 1 feeds directly into step 2 in RAM, no disk involved. The entire pipeline is represented as a DAG (Directed Acyclic Graph) of operations, and Spark executes it intelligently, materializing to disk only when RAM is exhausted or the computation explicitly requires it.

For iterative algorithms (machine learning, graph algorithms where you run the same computation many times), this is transformative. MLlib benchmarks from Berkeley showed Spark running logistic regression 100x faster than Hadoop MapReduce because it kept the training data in memory across iterations rather than reading it from HDFS each time.

3. Worked Example

Let's make this concrete with ShopStream. Suppose the orders CSV has the schema:

order_id, customer_id, product_id, category, seller_id, amount, timestamp

What Priya's pandas code tries to do (and why it fails):

import pandas as pd

# This attempts to load the ENTIRE 50GB file into RAM at once
df = pd.read_csv("shopstream_orders_all_time.csv")

# pandas builds a single DataFrame in contiguous memory
# A 50GB CSV typically expands to 3-5x in memory due to Python object overhead
# So we're looking at ~150-250GB RAM needed
# Priya's 16GB laptop never had a chance

revenue_by_category = df.groupby("category")["amount"].sum()

What Spark does differently:

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("ShopStream Revenue Analysis")
  .master("local[*]")  // We'll explain this in Lesson 03
  .getOrCreate()

// Spark does NOT load the entire file here
// It reads the schema, notes the file location, creates a plan
val orders = spark.read
  .option("header", "true")
  .option("inferSchema", "true")
  .csv("shopstream_orders_all_time.csv")

// Still no data loaded. Spark builds a logical plan
val revenueByCategory = orders
  .groupBy("category")
  .sum("amount")

// NOW data is read, in chunks, processed, results aggregated
revenueByCategory.show()

The critical difference: Spark never loads all 50GB into one machine's RAM simultaneously. It reads the file in partitions (say, 200 chunks of 250MB each), processes each partition, computes a partial sum per partition, then merges partial sums. At peak, only a few partitions are in memory at once.

On a cluster of 10 machines with 32GB RAM each, Spark can process all 200 partitions in parallel across machines, with 20 partitions per machine. Total wall time: a fraction of what single-machine processing would take, and no crashes.

4. Your Turn

Scenario: ShopStream's data engineering team has told you the raw data is growing at 15GB per month. You currently have a 128GB RAM single server.

Task: Answer these questions in writing (or in code comments):

1. How many months before your single server's RAM is completely saturated by the raw data alone (assuming no other processes)?

2. If you upgraded to a 1TB RAM server instead, how many additional months does that buy you? Is that a good investment strategy?

3. Assume you instead deploy a Spark cluster of 10 nodes, each with 64GB RAM. What is your total cluster memory? At 15GB/month data growth, how long before you'd need to add more nodes?

4. Sketch (in pseudocode or plain English) what happens to a groupBy("category").sum("amount") operation when the data is split across 10 Spark nodes. How does each node contribute to the final answer?

Success criteria: You can explain why RAM + data growth = inevitable vertical scaling failure. You understand that distributed aggregations work by computing partial results locally and merging them. You can articulate the difference between vertical and horizontal scaling in concrete terms.

5. Real-World Application

Uber processes over 7 petabytes of data daily across their data platform. Their entire ride pricing, surge detection, and driver supply matching runs on distributed Spark clusters. A single server, even the most powerful one money can buy, couldn't scan 7PB in the time window needed for real-time pricing decisions.

Netflix uses Spark to process its entire viewing history (hundreds of billions of events) for recommendation model training. The Viewing History dataset is so large that even their largest single machines can't hold it. Their ML pipelines run across thousands of Spark executors nightly.

LinkedIn originally built their analytics on Hadoop MapReduce (they were actually one of the early contributors). When they migrated to Spark, their data pipeline runtimes dropped from hours to minutes, because of the in-memory processing advantage. The team that maintains this is called the Data Infrastructure team, and "Big Data Engineer" or "Distributed Systems Engineer" roles at companies like LinkedIn specifically require understanding why distributed computing exists.

Amazon (AWS EMR) offers managed Spark clusters. The fact that one of the world's largest cloud providers built an entire product category around Spark tells you how central distributed compute is to modern data engineering.

For job seekers: roles titled Data Engineer, Analytics Engineer, or ML Engineer at companies with significant data volume will almost universally require Spark. Understanding why distributed computing exists, not just how to write Spark code, separates candidates who can debug and optimize from those who just copy-paste examples.

6. Recap + Bridge

Single machines have hard physical limits on RAM, CPU, and I/O that data growth will inevitably exceed, making horizontal scaling across a cluster of commodity machines both cheaper and more reliable than vertical scaling. In the next lesson, we'll open up Spark's hood and understand exactly how it coordinates work across those machines, who's in charge, who does the work, and how a single Spark job becomes thousands of parallel tasks.