What's Actually Inside Your Parquet File

Parquet has a reputation problem, and it's the good kind: it's so reliably faster than CSV that most engineers stop thinking about it the moment they switch. The file is columnar, it compresses well, the warehouse reads it quickly - what else is there to know?

Quite a lot, as it turns out. A Parquet file isn't a table. It's a layout - a set of decisions about how rows are grouped, how each column's values are encoded, which codec squeezes them, and what summary statistics get written alongside. Those decisions are made by whatever wrote the file, often with defaults nobody chose deliberately. And they're the difference between a query that reads 4 MB and the exact same query, over the exact same rows, reading 4 GB.

This post opens the format up. Not the spec - the parts that change your bill. And because the only convincing way to learn this is on a file you recognize, everything here is something you can see for yourself in the free Parquet Viewer: drop a file in, nothing uploads, and the layout is right there.

A file is a tree, not a grid

The mental model that gets people into trouble is the spreadsheet: rows down, columns across, one flat grid. Parquet's real shape is a tree.

At the top, the file splits into row groups - horizontal slabs of rows, each a self-contained unit. Inside a row group, each column is stored separately as a column chunk. Each chunk is divided into pages, the smallest unit the reader actually decompresses. And pinned to the end of the file is the footer: the schema, the byte offsets of everything, and - this is the part that earns its keep - per-chunk statistics like the min and max value in each column.

Every performance property of Parquet falls out of this structure. The reader opens the footer first, decides which row groups and pages it can skip entirely based on the statistics, and only then touches data. The grid model can't explain why one query is cheap and another expensive. The tree can.

The footer: why every reader starts at the end

Parquet's map is at the back of the book. A column chunk's byte offset is not stored next to the column chunk - it's stored in the footer, which means a reader cannot find a single value until it has read the last few kilobytes of the file first.

[PAR1][ row group 0 ][ row group 1 ]…[ FileMetaData ][ footer_len 4B ][PAR1]
  ▲                                   ▲
  magic, 4 bytes                      file_size - 8 - footer_len

One common open sequence is:

1. Read the first 4 bytes and check they say PAR1.
2. Seek to file_size - 8 and read 8 bytes: a little-endian int32 footer length, followed by PAR1 again.
3. Seek back footer_len bytes and deserialize the Thrift FileMetaData - the schema, the row count, and for every column chunk its byte offset, compressed size, encodings, and statistics.
4. Only now touch data, seeking directly to the chunks the query actually needs.

This design is why a writer can stream row groups without knowing final offsets: it appends data, then writes the footer at close with every offset filled in. Three consequences are worth carrying around:

Metadata is nearly free, even on object storage. Row count, schema, and column list for a 4 GB file on S3 cost two small ranged GETs and no data pages at all. That's how a catalog can summarize thousands of files in seconds.

The footer is a single point of failure. Lose the trailing bytes to a truncated upload and you don't lose a few rows, you lose the file. Every byte of data is still sitting there, but with no map, no reader can decode one value.

The footer can get fat. Its size isn't bounded by the spec, and it carries per-chunk statistics for every column of every row group. Ten thousand row groups times a thousand columns produces a footer measured in tens or hundreds of megabytes, re-read on every single open. That's the hidden cost shared by tiny row groups and very wide tables.

Row groups: the unit nobody sizes on purpose

The row group is the granularity of skipping. When your query has a WHERE clause, the engine checks each row group's column statistics and asks: could any row in here match? If the answer is no, the whole slab is skipped without being read. This is predicate pushdown, and it's most of why Parquet feels fast.

Which means row group size is a genuine tradeoff, not a detail:

• Too large (say, one row group for the whole file) and skipping becomes all-or-nothing. The statistics cover so many rows that the min/max range is wide, almost everything "could match," and you read the file end to end.
• Too small and you drown in overhead - a footer entry per group, a decompression setup per page, and statistics so granular the metadata starts to rival the data.

The usual advice is row groups in the 128 MB–1 GB range, but the number that actually matters is rows-per-group relative to your query patterns. The trap is that you rarely chose your row group size. Spark picked it. Pandas picked it. A COPY INTO picked it. The Parquet Viewer shows you the row group count and the rows in each, which is usually the first time anyone looks.

Pages: the unit that actually gets decompressed

One level below the column chunk is the page, and it's the level most tuning conversations skip. A page is the unit of encoding and compression: to read a single value, a reader must decompress and decode the entire page that value sits in. The default is roughly 1 MB.

Three page types show up in a normal file:

• Dictionary page. Written first in a column chunk when dictionary encoding is active, holding the table of distinct values.
• Data page (v1). Encoded values plus the definition and repetition levels that describe nulls and nesting, all compressed together.
• Data page (v2). Same content, but the levels stay uncompressed even when the values are compressed, so a reader can inspect nulls and nesting without paying to decompress the data.

Page size is its own dial (data_page_size in PyArrow), independent of row group size, and it's the one that decides how fine-grained skipping can get. Hold that thought - it becomes the whole point when we get to the page index.

Encodings: where the columnar magic actually lives

Here's the thing most "Parquet vs CSV" explanations skip. Parquet isn't fast just because it's columnar - it's fast because storing a column together lets it encode that column cleverly, in ways that are impossible when values from different columns are interleaved row by row.

The two that carry most of the weight:

• Dictionary encoding. When a column has low cardinality - a country, a status, an event_type - Parquet builds a dictionary of the distinct values and stores each cell as a small integer index. And "small" is literal: the index width is ceil(log2(distinct values)), so four distinct countries need two-bit dictionary IDs, not two-byte IDs. The IDs themselves use a hybrid run-length/bit-packed encoding, so headers and padding mean the actual storage is not always exactly two bits per row. A column of "checkout" / "view" / "add_cart" repeated a million times collapses to a three-entry dictionary plus a compact stream of two-bit IDs. This happens automatically - until the dictionary grows too big, at which point Parquet silently falls back to plain encoding for the rest of the chunk.
• Run-length & bit-packing (RLE). Sorted or repetitive columns compress to almost nothing: "the value 1 appears 50,000 times" is a handful of bytes.

That silent dictionary fallback is the one to watch. A high-cardinality column you thought was dictionary-encoded - a user_id, a request_id, a free-text field - pushes the dictionary past its threshold (about 1 MB by default), the writer gives up and writes the remaining pages as plain, and your file is suddenly far larger and slower than you assumed.

The good news is that it can leave a fingerprint in the optional encoding_stats metadata, which counts data pages by encoding and can reveal a mix of dictionary-encoded and plain data pages. The chunk's basic encodings list alone is not enough: a fully dictionary-encoded chunk can still list both RLE_DICTIONARY and PLAIN, because the dictionary page itself uses PLAIN. You can't feel a fallback from the outside, but when the writer emits encoding statistics you can read it straight off the per-chunk metadata.

Compression: the codec is a tradeoff, not a default

After encoding, each page gets compressed with a codec - and "default" is doing a lot of unexamined work here. The common ones:

The mistake isn't picking the "wrong" codec - it's not knowing which one your files use, and therefore not knowing whether you're paying for ratio you don't need or speed you're not getting. A warehouse serving interactive dashboards off GZIP is leaving latency on the table; a cold archive on SNAPPY is leaving storage money on the table. Both are invisible until you look at the codec recorded in the file.

The bigger point, and the reason a table like that oversells itself: the codec never sees your dataset. It sees one page, of one column, that the encodings above already squeezed. That's why switching Snappy to Zstd rarely delivers the ratio the benchmarks promise. The next post in this series, Parquet Compression Codecs, walks the write path and works out what the codec is actually doing to your bytes.

Statistics: the feature that does nothing if it's missing

Min/max statistics are what make predicate pushdown work. The engine reads "this row group's ts ranges from 09:00 to 09:15," sees your query wants ts > 14:00, and skips the whole slab. No statistics, no skip - the engine falls back to reading everything and filtering after the fact.

Four ways this quietly breaks:

1. The writer didn't emit statistics. Some writers, or some configurations, skip them. Your beautifully partitioned file gets scanned in full because the engine has nothing to prune on.
2. The data isn't sorted on the column you filter by. Statistics still exist, but if ts is scattered randomly across every row group, every group's min/max range spans the whole day. Every group "could match." Nothing gets skipped. Sorting on your common filter column before writing is one of the highest-leverage things you can do to a Parquet dataset, and it costs nothing at read time.
3. The bounds are truncated. To keep the footer small, a writer is allowed to store min_value = "B" rather than "Blart Versenwald III", flagging it with is_min_value_exact = false. That's still a valid lower bound, so range pruning still works whenever the predicate falls outside those conservative bounds. Exact bounds can sometimes prune more values near a truncated edge, but even exact min/max statistics do not prove that a value exists in the data; they are bounds, not a membership index.
4. null_count is missing. Without it an engine can't prove that a row group has no nulls and prune an IS NULL predicate, or prove that every row is null and prune an IS NOT NULL predicate. Writers should emit null_count = 0 rather than omitting it when there are no nulls, and plenty don't.

One more piece of archaeology lives here. The Statistics struct has two pairs of bounds: the original min/max, now deprecated because they only ever used signed comparison (a UINT32 column sorts wrong above 2^31), and min_value/ max_value, which sort according to the column's declared order. Old files, and old writers, still populate the deprecated pair.

The page index: statistics, one level deeper

Row group statistics skip whole slabs. But inside a surviving 128 MB row group there are still hundreds of pages, and by default the only way to get per-page statistics is to walk the DataPageHeader of each page in sequence - which means reading the compressed data you were trying to avoid reading.

The page index, added in format version 2.5, fixes exactly that. It lifts per-page metadata out of the data stream and into the footer region as two companion structures per column chunk:

• ColumnIndex - min, max, and null count for every page, plus a boundary_order flag. If the pages are ascending, a reader can binary-search for the matching ones instead of checking every page linearly.
• OffsetIndex - the byte offset, compressed size, and first row index of every page.

With both, the reader applies the predicate to the ColumnIndex, gets byte offsets for the surviving pages from the OffsetIndex, and seeks straight to them. first_row_index is what makes late materialization possible: once you know which rows of the filter column matched, you can fetch only those rows from the other columns.

The page index is opt-in in most writers (write_page_index=True in PyArrow), so plenty of files in the wild simply don't have one. Whether yours does is a fact recorded in the footer, and it's worth checking before you conclude that Parquet "can't" serve your narrow time-window queries.

Look at your own file

You can read every paragraph above and still not know what's true of your data - because all of it is per-file, per-column, and chosen by whatever wrote it. So the honest end to this post is: go look.

The Parquet Viewer opens a file entirely in your browser - the bytes never leave your machine, there's no upload and no account - and shows you the schema, the row groups and their sizing, the encoding and codec per column, and the statistics. Then it runs a findings pass: it flags the tiny-row-group sprawl, the high-cardinality column that lost its dictionary, the missing statistics, the codec mismatch - and links each finding to the lesson that explains the internal.

That's the loop worth building: stop treating Parquet as a black box that's "just fast," open the layout, and fix the three or four decisions that are actually costing you.

If you want the full ground-up treatment, it lives in the Storage & File Formats track, split in two: Parquet, Part 1 covers the layout, the physical types, and every encoding down to the wire format, and Parquet, Part 2 covers statistics, the page index, bloom filters, encryption, and how Parquet plugs into Spark, Iceberg, Delta Lake, and DuckDB. Both come with Docker labs, so you're reading your own footers rather than mine.

But the fastest way to care about any of it is to point the inspector at a file you already ship and find out what's really in there.