Series
Parquet and the Next Formats
Parquet has been the default columnar file format for over a decade, and the research has moved on. This series opens a Parquet file to see how it works, covers the formats challenging it - BtrBlocks, FastLanes, Lance, and Vortex - and asks where Parquet still wins.
2 posts
What's Actually Inside Your Parquet File
Everyone treats Parquet as 'the fast columnar format' and stops there. But two files with identical rows can differ tenfold in scan cost, and the reason is never in the data - it's in the layout. Why readers open the file at the end, row group and page sizing, dictionary encoding and its silent fallback, whether min/max statistics can actually be trusted, and whether a page index exists at all: these are the dials that decide your query bill. This is a tour of the bytes most people never look at, and a free, 100%-in-browser way to look at your own.
Read more →Parquet Compression Codecs: What Actually Happens to Your Bytes
Choosing between SNAPPY and ZSTD is the first Parquet tuning decision most people make and the one they understand least. The usual advice ranks codecs on a ratio-vs-speed chart, as if the codec were compressing your dataset. It isn't. It sees roughly one megabyte at a time, of one column, already dictionary-encoded and run-length-encoded by the time it arrives. That constraint explains why the ratios disappoint, why an encoding exists whose entire purpose is to make the codec's job easier, and why parquet-java's own default codec is none at all.
Read more →