Your Snowflake bill is mostly overhead
If you're paying thousands a month to query tens of gigabytes, you're renting a freight train to carry a backpack. Here's how to tell, and what to do instead.
snowflakecostduckdbdata-engineeringanalytics
Blog
Notes on data engineering
Pipelines, analytics infrastructure, ML systems, and lessons from building them in production.
DuckDB in production: what it's actually good at (and what it isn't)
DuckDB is having a moment, and the hype outruns the nuance. Here's an honest map of where an in-process analytical engine belongs in a real production system — and where it doesn't.
duckdbproductiondata-engineeringparquetanalytics
Do you actually need a data warehouse?
A short, honest decision framework for the question every data team eventually faces — before you sign a five-figure annual contract for infrastructure your data might not need.
data-warehousearchitectureduckdbdata-engineeringcost
Building data pipelines that don't page you at 3am
Most financial data pipelines break for boring, predictable reasons. Here are the five habits that separate a pipeline you babysit from one you forget about.
data-engineeringpipelinesreliabilityfinancial-data
Ask your database in plain English — locally
A small SQL-tuned model running on your machine can convert plain English questions into correct SQL. No API keys, no data leaving your laptop, no vector database.
text-to-sqlduckdblocal-llmollamadata-engineering
Local-first analytics in practice: DuckDB, Parquet, and killing the round-trip
A concrete walkthrough of the local-first pattern: query Parquet directly with DuckDB, no warehouse, no server, no per-query bill — from a notebook or the browser.
duckdbparquetlocal-first-analyticsdata-engineeringpython
Calling an LLM from inside a SQL query
Run a model directly in SQL to classify rows, extract text fields, or summarize data without exporting. Here's where it works, what it costs, and when not to bother.
llmsqlduckdbdata-engineeringailocal-first
Why I wrote Local-First Analytics
Most analytics stacks are cloud round-trips solving problems that fit on a laptop. Local-first analytics is the case for bringing the compute back to the data — and to the user.
local-first-analyticsduckdbdata-engineeringanalytics
Turning messy PDFs into clean tables, on your own machine
Parse PDFs to markdown, extract structured data with a local model and strict validation schema, no API needed.
pdf-extractiondata-engineeringlocal-llmpydanticdocling
Connecting an AI assistant to your database with MCP — without letting it do damage
Use the Model Context Protocol to let Claude explore your data in plain English. The whole game is permissions: read-only role, standard server, 90% of the value with almost none of the risk.
mcpdatabaseclaudeduckdbpostgrespermissions
Search your documents without a vector database
Most teams reach for Pinecone or Weaviate the moment they hear semantic search. For a team-sized knowledge base, you don't need one. Your database already has what you need.
local-firstsemantic-searchragduckdbsqliteembeddings
Letting Claude Code write your dbt models — and why you still read every line
Claude Code can run dbt in a loop, read errors, and fix them. It needs you to steer grain, business logic, and naming — and to review the tests it writes.
dbtclaude-codedata-engineeringsqlai-assisted-development
Using an LLM to catch the data quality problems your tests can't
Structural tests catch NOT NULLs and schema errors. Semantic problems—does this description match its category?—need a judge. How to use one responsibly.
data-qualityllmsemantic-validationpipelinecost-control
Should there be an LLM in your data pipeline?
An LLM is probabilistic, slow, and costly. Here's how to decide whether it belongs in your pipeline — and where it clearly doesn't.
llmdata-pipelineengineeringover-engineeringdeterminism
Local model or paid API? The honest math for data work
When does running an open-weight model on your own hardware beat paying per token? The answer depends on volume, frequency, and whether you have real privacy constraints.
llmcost-analysisdata-engineeringbudgeting
Generating realistic test data with a local model
How to harden a data pipeline by feeding it carefully generated edge cases, using a local LLM when simpler generators aren't enough.
test-datalocal-modelsdata-qualitypipeline-testing
When local-first runs out of road
Local-first, file-based data has a real ceiling. Most teams never reach it. When you do, here's what actually changes.
local-firstdata-warehousesscalingduckdbdbtanti-over-engineering
Running a local model over millions of rows without it falling over
Batching, checkpointing, and idempotency are what make local inference scale. Get these right and a single GPU chews through millions of rows overnight.
batch-inferencelocal-modelsmlopsdata-engineeringcheckpointing
Meet AEGIS: My Weird Little Operating Layer
Every week there’s another agent demo, another workflow canvas, another pitch about software running software for us. OpenClaw, browser agents, Zapier, n8n, Claude Code, MCP s
llmautomationknowledge-graphlocal-llm
Building with Claude Code: Faster Iteration, Faster Drift
I have been rebuilding a personal project called AEGIS into its third version, and on paper this rewrite should have dragged on for weeks. It did not. Claude Code helped me mo
aideveloper-toolsclaude-codeengineering
Building AEGIS: How Prolog + a Knowledge Graph Make Local LLMs Actually Useful
I’ve been building a personal AI orchestration system. The first version — Jarvis — worked, but became a 20,000-line monolith I dreaded touching. So I rebuilt it from scratch
llmprologknowledge-graphlocal-llm
Market Regime Detection: From Hidden Markov Models to Wasserstein Clustering
Financial markets move through distinct phases — bullish rallies, sharp crashes, quiet consolidations, and volatile swings. These market regimes differ not just in price direc
machine-learningquantitative-financedata-sciencepython