Parquet Compression Codecs: What Actually Happens to Your Bytes

Every Parquet tuning conversation starts in the same place. Someone notices the storage bill, someone else says "switch from Snappy to Zstd," a benchmark gets run, and the table gets 20% smaller instead of the 60% everyone expected. Then the conversation stops, because nobody knows what to look at next.

The reason the answer disappoints is that the question is asked at the wrong altitude. The usual framing treats the codec as if it compresses your dataset - as if switching from Snappy to Zstd on a 400 GB table is like re-tarring a 400 GB directory with zstd instead of gzip. It is not remotely the same operation.

Parquet's codecs never see your dataset. They never see your table, your row group, or even a whole column. They see a page: by default about one megabyte of a single column, which by the time the codec is invoked has already been dictionary-encoded, run-length-encoded, or delta-encoded into something that barely resembles your data. The codec is a second compression pass over bytes that a first compression pass already squeezed. Whatever redundancy was easy to find is gone before the codec starts.

Once you see the write path clearly, the codec question re-forms into something answerable. The spec is small enough to read in an afternoon, and the reference implementation is explicit about every step. So let's walk it.

Where compression actually happens

Parquet's own specification is unusually blunt about the scope. From Compression.md in the apache/parquet-format repo:

Data pages and dictionary pages. That's the whole list. Not files, not row groups, not column chunks. The footer is never compressed. The page headers are never compressed. The Thrift metadata that tells you what's in the file is never compressed. Only the payload inside individual pages.

And the unit is genuinely independent. Each page is compressed on its own, with its own call into the codec library, and can be decompressed without touching any other page. That's what makes page-level predicate pushdown possible: a reader that consults the Page Index and decides it only needs page 7 out of 300 can decompress page 7 alone. It's a real feature, and the compression window is the price you pay for it.

Here is the second blunt thing in the same document:

No framing. No container. Parquet hands the library a buffer and stores the two sizes it needs to get it back. You can see both fields in parquet.thrift:

struct PageHeader {
  /** the type of the page: indicates which of the *_header fields is set **/
  1: required PageType type

  /** Uncompressed page size in bytes (not including this header) **/
  2: required i32 uncompressed_page_size

  /** Compressed (and potentially encrypted) page size in bytes, not including this header **/
  3: required i32 compressed_page_size
  ...
}

Two integers. uncompressed_page_size so the reader can allocate the output buffer exactly, compressed_page_size so it knows how many bytes to hand the decompressor. That's the entire contract between Parquet and every codec it supports. The codec doesn't get told what column this is, what type the values are, what the cardinality looks like, or what the previous page contained. It gets a byte buffer.

The write path, in order

The ordering matters more than the codec choice, so let's be precise about it. This is the sequence in ColumnChunkPageWriteStore from apache/parquet-java, the reference implementation:

The code is as plain as the diagram. For a V1 data page:

long uncompressedSize = bytes.size();
// ...
BytesInput compressedBytes = compressor.compress(bytes);

One call. bytes at that point is the concatenation of repetition levels, definition levels, and encoded values for this page, and the whole concatenated blob goes to the codec together. The dictionary page takes the identical path:

int uncompressedSize = (int) dictionaryBytes.size();
BytesInput compressedBytes = compressor.compress(dictionaryBytes);

Which is worth pausing on. The dictionary page is compressed too, with the same codec. If you dictionary-encode a high-cardinality string column, your dictionary page holds every distinct value - and that page is where a strong codec earns its keep, because a dictionary of URLs or country names is exactly the long-string-redundancy workload that Zstd is good at. The data pages that reference it are just small integers and won't compress much further. So a column's response to a codec change depends heavily on how its bytes are split between dictionary and data.

The seven codecs

The full set, from the CompressionCodec enum in parquet.thrift:

enum CompressionCodec {
  UNCOMPRESSED = 0;
  SNAPPY = 1;
  GZIP = 2;
  LZO = 3;
  BROTLI = 4;  // Added in 2.4
  LZ4 = 5;     // DEPRECATED (Added in 2.4)
  ZSTD = 6;    // Added in 2.4
  LZ4_RAW = 7; // Added in 2.9
}

Eight values, one of which is "no", one of which is deprecated. The comments carry real history: Parquet shipped with Snappy, Gzip, and LZO. Brotli, LZ4, and Zstd all arrived together in format version 2.4, and LZ4_RAW came later in 2.9 to clean up the mess LZ4 made. More on that shortly.

What the spec says about each one is thinner than you'd expect, and deliberately so:

Parquet does not define compression. It defers, and each codec section is essentially a pointer plus a tiebreak rule - for Snappy, "the implementation provided by the Snappy compression library is authoritative"; for Zstd, RFC 8878 with the reference library as authority. This is a good design decision. It means Parquet never has to version a compression algorithm, and it means the codecs improve without the format changing.

Two entries in that table need their own explanation.

GZIP is gzip, not zlib. The spec calls this out specifically: "the GZIP format (not the closely-related 'zlib' or 'deflate' formats) defined by RFC 1952." Three formats, nearly the same compressed stream, different headers. Getting this wrong produces a file that some readers accept and others reject, and the failure is a header parse error that looks nothing like a compression problem. There's a second gzip footgun in the spec too:

A conforming reader must handle a page holding several concatenated gzip members. Many real readers don't. So the spec's advice to writers is: technically legal, practically don't.

LZO needs a library you probably can't ship. The codec entry itself is neutral, but the practical constraint shows up in CompressionCodecName in parquet-java, where LZO points at com.hadoop.compression.lzo.LzoCodec - a class that lives outside the project because the LZO library is GPL. In an Apache-licensed world this means LZO is the codec that is in the spec, in the enum, and not on your classpath. If you encounter LZO Parquet in the wild, it's almost always old Hadoop-era output.

The LZ4 saga

The one genuinely broken thing in the codec list is worth telling properly, because it is the clearest example of what happens when a format defers to an implementation instead of a specification.

Notice that LZ4 is the single exception to the "fed as-is, without any additional framing or padding" rule that governs every other codec. The rule was written because of LZ4. And the fix could not be "correct LZ4," because the broken files already existed and readers still have to open them - so the fix had to be a new enum value.

If you are writing Parquet today and you want LZ4 speed, you want LZ4_RAW. If your writer's API offers you a codec literally named lz4, check what it maps to; some engines have quietly repointed the friendly name at LZ4_RAW, and others have not.

What .snappy.parquet actually tells you

If most of your Parquet arrived from Spark, you have seen filenames like part-00000-a1b2c3.snappy.parquet and reasonably concluded the file is Snappy-compressed. It usually is. But the extension is a naming convention, not metadata, and it is worth knowing exactly how much it is promising you.

The CompressionCodecName enum carries the extension as a third field, next to the Hadoop class and the Thrift enum value:

UNCOMPRESSED(null, CompressionCodec.UNCOMPRESSED, ""),
SNAPPY("org.apache.parquet.hadoop.codec.SnappyCodec", CompressionCodec.SNAPPY, ".snappy"),
GZIP("org.apache.hadoop.io.compress.GzipCodec", CompressionCodec.GZIP, ".gz"),
LZO("com.hadoop.compression.lzo.LzoCodec", CompressionCodec.LZO, ".lzo"),
BROTLI("org.apache.hadoop.io.compress.BrotliCodec", CompressionCodec.BROTLI, ".br"),
LZ4("org.apache.hadoop.io.compress.Lz4Codec", CompressionCodec.LZ4, ".lz4hadoop"),
ZSTD("org.apache.parquet.hadoop.codec.ZstandardCodec", CompressionCodec.ZSTD, ".zstd"),
LZ4_RAW("org.apache.parquet.hadoop.codec.Lz4RawCodec", CompressionCodec.LZ4_RAW, ".lz4raw");

Two things fall out of that list immediately. UNCOMPRESSED's extension is the empty string, so an uncompressed file is named plain .parquet. And the LZ4 mess has leaked into the filesystem: the deprecated codec's extension is .lz4hadoop, naming the undocumented Hadoop framing as its distinguishing feature, while the clean one gets .lz4raw.

The name gets built in exactly one place, ParquetOutputFormat:

CompressionCodecName codec = getCodec(taskAttemptContext);
String extension = codec.getExtension() + ".parquet";
Path file = getDefaultWorkFile(taskAttemptContext, extension);

This is a Hadoop MapReduce output convention - the same habit that gives you part-00000.gz. Spark doesn't inherit that method, but it reimplements the identical line in ParquetUtils.scala, reusing parquet-java's own CodecConfig:

override def getFileExtension(context: TaskAttemptContext): String = {
  CodecConfig.from(context).getCodec.getExtension + ".parquet"
}

Nothing reads it back. There is no fromExtension method anywhere in parquet-format or parquet-java, and no reader parses the path. Every call site that needs a decompressor takes the codec from the footer, per column chunk:

// ParquetFileReader.readDictionary(ColumnChunkMetaData meta)
BytesInputDecompressor decompressor = options.getCodecFactory().getDecompressor(meta.getCodec());

// ParquetFileReader, data page path
BytesInputDecompressor decompressor =
    options.getCodecFactory().getDecompressor(descriptor.metadata.getCodec());

Rename foo.snappy.parquet to foo.bin and every engine still reads it perfectly. Rename a Zstd file to .snappy.parquet and it still reads perfectly, as Zstd.

Which sets up the interesting failure. The filename is built from CodecConfig.getCodec(), and that reads the bare config key:

public static CompressionCodecName getParquetCompressionCodec(Configuration configuration) {
  return CompressionCodecName.fromConf(configuration.get(ParquetOutputFormat.COMPRESSION, UNCOMPRESSED.name()));
}

But per-column codecs, from the tuning section above, are resolved through a completely different key. ColumnConfigParser's javadoc: "Parses the specified key-values in the format of root.key#column.path", and it registers helpers as rootKey + '#'. So the file-level codec lives at parquet.compression and the per-column overrides live at parquet.compression#column.path - two different keys, and only the first one reaches the filename.

The filename therefore cannot describe a file with per-column codecs. A file legitimately named .snappy.parquet can contain Zstd column chunks, because the format stores the codec per column chunk and the name has exactly one slot.

What the extension does tell you reliably is provenance. A .snappy.parquet almost certainly came out of Spark or Hive, and since Spark's SQLConf sets spark.sql.parquet.compression.codec with .createWithDefault("snappy"), the guess is usually right. Just don't audit compression with ls.

The default nobody chose

Here's something that surprises most people. This is ParquetWriter in parquet-java:

public static final int DEFAULT_BLOCK_SIZE = 128 * 1024 * 1024;
public static final int DEFAULT_PAGE_SIZE = ParquetProperties.DEFAULT_PAGE_SIZE;
public static final CompressionCodecName DEFAULT_COMPRESSION_CODEC_NAME = CompressionCodecName.UNCOMPRESSED;

The reference Parquet writer's default compression codec is none. Not Snappy. Snappy's status as "the Parquet default" comes entirely from engines layered on top - Spark ships spark.sql.parquet.compression.codec set to snappy, and most people have only ever created Parquet through an engine that made this choice for them.

The neighbouring defaults in ParquetProperties matter just as much:

public static final int DEFAULT_PAGE_SIZE = 1024 * 1024;
public static final int DEFAULT_DICTIONARY_PAGE_SIZE = DEFAULT_PAGE_SIZE;
public static final boolean DEFAULT_IS_DICTIONARY_ENABLED = true;
public static final boolean DEFAULT_IS_BYTE_STREAM_SPLIT_ENABLED = false;
public static final WriterVersion DEFAULT_WRITER_VERSION = WriterVersion.PARQUET_1_0;
public static final int DEFAULT_PAGE_ROW_COUNT_LIMIT = 20_000;

Read those together and the shape of a default Parquet file appears:

• 1 MiB pages. This is your compression window. Not 128 MiB, not the file size. One mebibyte.
• 20,000 rows per page, whichever comes first. A narrow column - a boolean, a small int - will hit the row limit long before the byte limit, so its pages are far smaller than 1 MiB. On such a column the codec is compressing a few kilobytes at a time, which is close to the worst case for any dictionary-based compressor.
• Dictionary encoding on. The first pass is enabled by default and it is the one doing the heavy lifting.
• Byte-stream-split off.
• Writer version 1.0, which means V1 data pages by default. V2 pages, ten years after being specified, are still opt-in.

Why the encoder gets the credit

The reason page-level compression works at all despite that tiny window is that it isn't doing the important work. The encoding pass is.

Consider a country column with 200 distinct values across 100 million rows. Dictionary encoding writes a dictionary page with 200 strings and then represents every data page as RLE/bit-packed integers - and since 200 values fit in 8 bits, each row costs one byte before RLE even starts collapsing runs. You've already achieved something like a 15x reduction and no codec has run yet. When the codec finally sees those bit-packed integers, there is very little redundancy left to find. It might get you another 1.5x. This is the normal case, and it's why codec changes so often move the needle less than expected: the codec is fighting for the scraps the encoder left.

The proof that Parquet's own designers think about it this way is BYTE_STREAM_SPLIT, encoding number 9. Here is its description in Encodings.md, and it's one of the strangest sentences in the spec:

An encoding that makes the data exactly the same size. Its entire purpose is to rearrange bytes so the codec does better. The mechanism is simple: for a sequence of floats, instead of storing them as AA BB CC DD | 00 11 22 33 | A3 B4 C5 D6, it scatters each value's bytes into per-position streams and concatenates them:

Original:  AA BB CC DD    00 11 22 33    A3 B4 C5 D6
Split:     AA 00 A3  BB 11 B4  CC 22 C5  DD 33 D6

Why this helps: in an array of similar-magnitude floats, the high-order bytes (exponent and top mantissa bits) are nearly identical across values while the low-order bytes are close to random. Interleaved, every 4-byte value mixes compressible and incompressible bytes, and no codec can find the pattern. Separated into streams, the exponent stream becomes long runs a codec devours, and the noise stream stays noise - but at least it's quarantined noise that doesn't poison the rest.

That's a whole encoding that exists purely to hand-feed the compressor. If the codec were the star of the show, BYTE_STREAM_SPLIT would make no sense at all.

V1 vs V2 pages: what gets compressed

The V2 data page changes what the codec receives, and it's the one page-format decision with a direct compression consequence. From the DataPageHeaderV2 docstring in the thrift file:

The V2 write path in parquet-java shows the split exactly:

int rlByteLength = toIntWithCheck(repetitionLevels.size());
int dlByteLength = toIntWithCheck(definitionLevels.size());
int uncompressedSize = toIntWithCheck(data.size() + repetitionLevels.size() + definitionLevels.size());
boolean compressed = false;
BytesInput compressedData = BytesInput.empty();
if (data.size() > 0) {
  // TODO: decide if we compress
  compressedData = compressor.compress(data);
  compressed = true;
}

Only data goes to compress(). The levels are measured, recorded in the header, and written in the clear. And note that uncompressedSize still counts all three sections, while compressedSize is computed a few lines later as compressedData.size() + repetitionLevels.size() + definitionLevels.size() - the uncompressed levels are included in both totals. Which means your ratio math on a V2 file with deeply nested data is measuring something different than on a V1 file, since a chunk of the payload was never eligible for compression in the first place.

That // TODO: decide if we compress is my favourite line in the codebase. The format supports an is_compressed flag precisely so a writer can decide, per page, that compression isn't worth it - a page of random UUIDs or an already-compressed blob will come back from the codec larger than it went in. The spec defines the escape hatch. The reference writer has never implemented the decision, and unconditionally sets compressed = true whenever the data section is non-empty.

Tuning, in the order that matters

Given all of the above, here's the sequence that actually moves numbers.

1. Check the encoding first, not the codec. If a column isn't dictionary encoded when it should be, no codec will save you. The usual cause is the dictionary hitting DEFAULT_DICTIONARY_PAGE_SIZE (1 MiB) and the writer falling back to PLAIN for the rest of the chunk - a fallback that is silent, permanent for that chunk, and visible only in the metadata.

2. Check the page size. The codec's window is the page. If your pages are 4 KB because of the 20,000-row limit on a narrow column, you're asking a dictionary compressor to work with no dictionary.

3. Then pick a codec. With encoding and page size correct, the real choice is narrow:

• ZSTD when bytes cost more than milliseconds - cold data, archival partitions, anything scanned rarely or read over a network you pay for.
• LZ4_RAW when decompression latency is on the critical path of an interactive query and the data is hot.
• SNAPPY when interop with an older reader matters more than either. It is the most universally supported codec that isn't UNCOMPRESSED.
• UNCOMPRESSED for columns that are already compressed - image blobs, pre-gzipped payloads, high-entropy hashes. Running a codec over these burns CPU on both write and read to make the data slightly bigger.

4. Set the level deliberately. Zstd's level in parquet-java:

public static final String PARQUET_COMPRESS_ZSTD_LEVEL = "parquet.compression.codec.zstd.level";
public static final int DEFAULT_PARQUET_COMPRESS_ZSTD_LEVEL = 3;

Level 3 - Zstd's own default, out of a range that runs to 22. Raising it is the single most common "we tuned compression" change, and on 1 MiB pages it delivers much less than the level number suggests, because high Zstd levels buy their ratio through long-range matching and a large window. There is no long range inside a 1 MiB page. You are paying level-19 CPU for something closer to level-6 benefit. There's also parquet.compression.codec.zstd.workers (default 0, meaning no worker threads), which trades memory for write throughput and doesn't change the output bytes.

5. Vary the codec per column. This is the most underused knob in Parquet, and it exists: parquet-java resolves parquet.compression per column path, and ParquetOutputFormat wires both the codec and the level through a column config mechanism, keyed as parquet.compression#<column.path> and parquet.compression.level#<column.path>. The codec lives in ColumnMetaData, not in the file metadata:

4: required CompressionCodec codec

Per column chunk. Which means one file can legally hold a Zstd-compressed description column, an LZ4_RAW-compressed event_time column, and an uncompressed thumbnail_blob column, and every conforming reader handles it without knowing you did anything clever. Most tables have two or three columns holding most of the bytes; those are the ones worth a strong codec, and the rest can stay cheap.

What to look at when it doesn't work

The metadata carries everything you need, and it is exact rather than estimated, which makes this one of the few storage problems you can debug without running an experiment. Per column chunk, ColumnMetaData gives you:

/** total byte size of all uncompressed pages in this column chunk (including the headers) **/
6: required i64 total_uncompressed_size

/** total byte size of all compressed, and potentially encrypted, pages
 *  in this column chunk (including the headers) **/
7: required i64 total_compressed_size

total_uncompressed_size / total_compressed_size is your true per-column ratio, and it's the number to look at - not the file-level one, which averages your best and worst columns into a figure that tells you nothing about which to fix. Pull it for every column chunk and sort descending by total_compressed_size. The top three columns are your storage bill. The rest is noise.

Three specific readings and what they mean:

A ratio near 1.0 with a large total_compressed_size. The codec found nothing. Either the column is genuinely high-entropy (hashes, UUIDs, encrypted fields, already-compressed blobs), in which case set it to UNCOMPRESSED and stop paying CPU for nothing, or the encoding is wrong and you're compressing PLAIN-encoded values that should have been dictionary encoded. The encodings list in the same ColumnMetaData tells you which. If you see PLAIN where you expected RLE_DICTIONARY, you found a dictionary fallback.

A ratio above 1.0 - compressed larger than uncompressed. Rare, but it happens on small pages of random data, and it's the case that // TODO: decide if we compress was meant to catch. The codec's own framing exceeds what it saved.

A great ratio and a slow query. You optimized the wrong axis. Every byte you saved is being paid back on every read, and if the table is scanned hourly you'll pay that CPU thousands of times to save storage once. This is the trade that Zstd-at-a-high-level makes and it is frequently a bad one in a cloud where compute is billed by the second.

You can read all of this without a cluster. parquet-cli meta prints per-column sizes and encodings, DuckDB's parquet_metadata() table function exposes total_compressed_size and total_uncompressed_size as queryable columns, and our Parquet viewer reads the footer entirely in your browser without uploading the file anywhere.

The reframe

The ratio-vs-speed chart isn't wrong. It's just answering a question about compression libraries, and you have a question about a file format.

Parquet's codec is the last step in a pipeline that has already done the important work, operating on a window the format chose for reasons that have nothing to do with compression - 1 MiB pages exist so readers can skip pages, not so codecs can do their best. Every structural decision in Parquet that touches compression is a decision to give the codec less: less context, less window, less type information, less continuity. In exchange you get random page access, parallel decode, and predicate pushdown. That is a trade the format makes deliberately, and on balance a good one.

Which means the honest way to state the codec question is not "which codec compresses best," but: given that my compressor is going to see one megabyte of one already-encoded column at a time, what is the most useful thing I can hand it? Encode first so it doesn't waste effort on redundancy that's already gone. Size pages so it has something to work with. Split bytes so noise doesn't contaminate signal. Then, and only then, choose between Zstd and LZ4_RAW.

And when the answer for a given column turns out to be "nothing useful, it's random bytes" - the right codec is UNCOMPRESSED, and choosing it on purpose is tuning too.

This constraint is also the seam that the newer columnar formats are pushing on. Once you accept that the encoder does the real work and the codec mops up, the natural next question is why the encoder is limited to a fixed menu of five encodings chosen in 2013 - which is more or less the thesis of BtrBlocks' cascading compression and of Vortex's pluggable encoding framework.

Related reading: What's actually inside your Parquet file · Why Parquet is showing its age · BtrBlocks: cascading lightweight compression · Vortex: the compressed Arrow file format · Storage and file formats

Sources: apache/parquet-format - Compression.md, Encodings.md, and src/main/thrift/parquet.thrift (CompressionCodec, PageHeader, DataPageHeaderV2, ColumnMetaData). apache/parquet-java - ColumnChunkPageWriteStore, ParquetWriter, ParquetProperties, ParquetOutputFormat, ParquetFileReader, CompressionCodecName, CodecFactory, ColumnConfigParser, codec/CodecConfig, and codec/ZstandardCodec. Filename-convention claims cross-checked against apache/spark (SQLConf.scala, ParquetUtils.scala) and apache/arrow (python/pyarrow/dataset.py, python/pyarrow/parquet/core.py). Codec specifications are external to Parquet by design: Snappy's format description, RFC 1952 (GZIP), RFC 7932 (Brotli), RFC 8878 (Zstandard), and the LZ4 block format.