Blog

The latest in data engineering, what's happening across the field.

· #duckdb #analytics #quack #motherduck

The Duck Is Plural Now: A Quack Deep Dive

DuckDB has always been a single-process, embedded engine - extraordinary at what it does, and silently solitary. Quack puts a small network surface on top of that engine so two (or twenty) DuckDB processes can mount each other's catalogs and query each other's tables as if they were local. The defining choice is that it does this inside DuckDB, not around it: same wire format, same type system, same SQL. We stood up two servers, pointed them at each other, and went looking for where that one decision quietly decides everything else.

Read more →
· #data-governance #pii #security #masking #compliance

Stop Hand-Writing PII Masking Policies

Most PII masking goes wrong long before the SQL. A hash without a salt is a lookup table for anyone with a rainbow table. A token vault stored next to the tokens defeats tokenization. A redacted ZIP that keeps five digits still re-identifies people under HIPAA Safe Harbor. The hard part isn't writing Snowflake or Databricks or BigQuery masking DDL - it's choosing the right technique per column and not tripping the gotchas that turn 'anonymized' back into identifiable. Here's the decision framework, the traps, and a free generator that produces the policy.

Read more →
· #data-modeling #scd #dimensional-modeling #warehouse

Slowly Changing Dimensions, Actually Explained

Slowly Changing Dimensions get taught as trivia: Type 1 overwrites, Type 2 keeps history, Type 3 keeps the previous value, and so on. Memorizing the list misses the only thing that matters - which version of a dimension your fact rows join to when someone asks 'what was revenue by region last quarter.' Get that wrong and your dashboard reports a number that was never true. This is SCD framed around that question, with a free playground that lets you replay a change timeline and watch facts attach to the right (or wrong) version.

Read more →
· #parquet #storage #file-formats #performance

What's Actually Inside Your Parquet File

Everyone treats Parquet as 'the fast columnar format' and stops there. But two files with identical rows can differ tenfold in scan cost, and the reason is never in the data - it's in the layout. Row group sizing, dictionary vs plain encoding, the compression codec, and whether min/max statistics actually let the engine skip pages: these are the dials that decide your query bill. This is a tour of the bytes most people never look at, and a free, 100%-in-browser way to look at your own.

Read more →