Series
Misc
Standalone deep dives that don't belong to a longer series: the latest in data engineering, from what an engine is really doing under the hood to what a new framework feature actually buys you.
3 posts
Stop Hand-Writing PII Masking Policies
Most PII masking goes wrong long before the SQL. A hash without a salt is a lookup table for anyone with a rainbow table. A token vault stored next to the tokens defeats tokenization. A redacted ZIP that keeps five digits still re-identifies people under HIPAA Safe Harbor. The hard part isn't writing Snowflake or Databricks or BigQuery masking DDL - it's choosing the right technique per column and not tripping the gotchas that turn 'anonymized' back into identifiable. Here's the decision framework, the traps, and a free generator that produces the policy.
Read more →The Duck Is Plural Now: A Quack Deep Dive
DuckDB has always been a single-process, embedded engine - extraordinary at what it does, and silently solitary. Quack puts a small network surface on top of that engine so two (or twenty) DuckDB processes can mount each other's catalogs and query each other's tables as if they were local. The defining choice is that it does this inside DuckDB, not around it: same wire format, same type system, same SQL. We stood up two servers, pointed them at each other, and went looking for where that one decision quietly decides everything else.
Read more →Spark Declarative Pipelines: What It Actually Does Under the Hood
Every Spark ETL job grows the same crust over time: checkpoint wiring you copy between projects, incremental-ingest logic nobody wants to touch, a hand-maintained order of operations that breaks the moment someone adds a table. Spark Declarative Pipelines, the framework Databricks built as Delta Live Tables and then donated to Apache Spark, where it shipped in 4.1, tries to delete that crust. You declare the tables that should exist and what they contain; SDP figures out the order, the parallelism, the checkpoints, the retries. We went through the model piece by piece - flows, streaming tables, materialized views, the dataflow graph - to see what it genuinely handles for you and what it quietly hands back.
Read more →