Blog
Pipelines, analytics infrastructure, ML systems, and lessons from building them in production.
Start here: the 18-check platform teardown — the questions I work through in a $3,000 audit, published in full. No signup.
I wrote a book — Local-First Analytics: warehouse-class analytics on your laptop with DuckDB, Parquet and Arrow. The book →
I ran my personal AI system's thinking on an 8GB GPU at home for months, then moved it to Bedrock. Not because local models are bad — because my GPU is small and the work is bursty. Here are the production numbers, and why the embeddings stayed home.
Eight open tasks for one failing probe, and thirteen dedupe schemes that disagreed. How I rebuilt AEGIS's alerting so a problem has a record of its own before any ticket exists — and the three things production changed about it afterwards.
My research agent had indexed 20,400 documents. Prompts had used 78 of 10,284 PDFs. Here's what measuring retrieval instead of ingestion changed — and why the fix wasn't a better filter, it was storing less.
My finance agent read 200 characters of each email and called it accounting. Rebuilding it on a plain-text double-entry journal, with Postgres demoted to an index, fixed the numbers, cut the lane's model calls by 71% and caught a bug that kept paid bills open forever.
Lazy execution, no hidden copies, and parallelism by default. What each one changes in practice, why the speed claims are softer than the marketing, and when DuckDB is the better answer than either.
Parquet is smaller and faster than CSV for three mechanical reasons, not because it is magic. What the compression ratio depends on, why the speed gap collapses on a warm cache, and when CSV is still the right answer.
DuckDB is an engine inside your process; ClickHouse is a server built for many concurrent queries on fresh data. How to choose, and why teams run both.
A three-state hidden Markov model runs daily inside my trading system. Three features, four indices, a monthly retrain, and one rule that decides whether the regime is allowed to touch a position at all. This is what it does, and what it is not allowed to do.
DuckDB works as an LLM response cache when one process owns it, and the SQL over the cache is the reason to pick it. The three kinds of cache, the single-writer rule, the key that stops you serving a retired model's answers, and measured numbers for exact, brute-force and HNSW lookups.
How to set up DuckDB on a server you control for faster queries — memory and temp-dir settings, Parquet layout, the read-only fan-out pattern, containers, and the mistakes that make a single node look slow.
DuckDB is production-ready for a specific shape of workload and genuinely unsafe for another. The single-writer model, durability semantics, memory behaviour and version compatibility — what actually bites, and how to design around it.
For a small SaaS, a single-node engine over Parquet handles more than founders expect. The three questions that decide whether it fits, what customer-facing analytics needs, and the specific point where you outgrow it.
What drives the build cost of a data pipeline, what it costs to run and maintain afterwards, how managed-ETL pricing models compare with building your own, and the shapes I see by funding stage.
The full rate card — audit, build, AI workflow, fractional — with the numbers in USD, plus why builds are scoped ranges instead of a day rate.
DuckDB wins when the working set fits one machine and few people query at once. Snowflake wins on concurrency and governance. What switching really costs.
The real fully-loaded cost of a full-time data engineer, when a consultant is the better call, and how to tell which one your situation actually needs.
The four ways to wire an LLM to your data — MCP, text-to-SQL, a semantic layer, function calling — and the guardrails that stop the demo becoming an incident.
The short answer: ClickHouse for steady, high-volume, sub-second analytics — if you'll operate it. Snowflake for mixed ad-hoc SQL across many teams with zero ops. Cost models, SQL gaps, streaming and migration, from running ClickHouse in production.
What it takes to make natural-language querying actually useful for non-technical teams — the schema semantics, guardrails and verification the demos skip.
A teardown of a macro data product: 171 countries scored daily from 10 free public data sources, point-in-time correct, with the licensing and cost work done.
Four LLM agents, 28 Temporal workflows, every risky action behind a human approval gate. An architecture teardown of AEGIS — and what it teaches about production AI automation.
NSE and crypto ingestion, ClickHouse, LightGBM, backtests and a Prolog compliance layer — a self-hosted data platform built and run by one person.
Three months ago I wrote about my personal AI orchestration system and ended with a section titled 'Why It's Not Open Source Yet.' That's fixed. AEGIS is on GitHub, MIT-licensed — here's what it actually took to get there.
The biggest refactor of the AEGIS open-sourcing sprint: removing every branch on an agent's identity so behavior lives in the database as capability tags — resolved at runtime, edited from a UI, and safe even when a tag has no owner.
Scheduling a social post doesn't need a new UI. A to-do already has copy, a time, and labels — so in AEGIS, a Todoist task with a publish label is a scheduled post, and nothing goes out until a card in chat gets a human tap.
For a self-hosted system that's meant to be forked, there's exactly one honest way to handle infrastructure credentials: the user brings their own, the system stores them encrypted, and nothing in the code assumes a vendor. Here's how AEGIS got there — and what it cost me to cut my own vendors out.
The day Next and Someday stopped being Todoist projects and became labels — and why that small modelling decision is what made AEGIS's GTD layer, and its bidirectional sync, actually work.
AEGIS is going open source, and today the first public commits landed. Day one of turning a private system into a shippable one: twenty-seven migrations squashed into a baseline, credentials evicted from the build, and an admin panel redesigned around decisions.
If you're paying thousands a month to query tens of gigabytes, you're renting a freight train to carry a backpack. Here's how to tell, and what to do instead.
AEGIS watches my homelab and GitHub, but the interesting part is what happens between an alert firing and me hearing about it. Most of the time, the correct answer is nothing.
Is DuckDB production-ready? Where it shines, the workloads it fits, and the deployment patterns that work — from someone who ships it.
A short, honest decision framework for the question every data team eventually faces — before you sign a five-figure annual contract for infrastructure your data might not need.
A small SQL-tuned model running on your machine can convert plain English questions into correct SQL. No API keys, no data leaving your laptop, no vector database.
A concrete walkthrough of the local-first pattern: query Parquet directly with DuckDB, no warehouse, no server, no per-query bill — from a notebook or the browser.
Run a model directly in SQL to classify rows, extract text fields, or summarize data without exporting. Here's where it works, what it costs, and when not to bother.
Most analytics stacks are cloud round-trips solving problems that fit on a laptop. Local-first analytics is the case for bringing the compute back to the data — and to the user.
Parse PDFs to markdown, extract structured data with a local model and strict validation schema, no API needed.
Use the Model Context Protocol to let Claude explore your data in plain English. The whole game is permissions: read-only role, standard server, 90% of the value with almost none of the risk.
Most teams reach for Pinecone or Weaviate the moment they hear semantic search. For a team-sized knowledge base, you don't need one. Your database already has what you need.
Claude Code can run dbt in a loop, read errors, and fix them. It needs you to steer grain, business logic, and naming — and to review the tests it writes.
Structural tests catch NOT NULLs and schema errors. Semantic problems—does this description match its category?—need a judge. How to use one responsibly.
An LLM is probabilistic, slow, and costly. Here's how to decide whether it belongs in your pipeline — and where it clearly doesn't.
When does running an open-weight model on your own hardware beat paying per token? The answer depends on volume, frequency, and whether you have real privacy constraints.
How to harden a data pipeline by feeding it carefully generated edge cases, using a local LLM when simpler generators aren't enough.
Local-first, file-based data has a real ceiling. Most teams never reach it. When you do, here's what actually changes.
Batching, checkpointing, and idempotency are what make local inference scale. Get these right and a single GPU chews through millions of rows overnight.
The design decision in AEGIS I'm proudest of isn't an agent or a model — it's a single interactions primitive. How one Postgres table, five kinds of card, and a Temporal workflow replaced every per-domain approval pattern.
AEGIS is my personal operating layer: four agents that watch my inbox, repos, alerts and feeds, and bring a proposed next step instead of a notification.
Claude Code made my AEGIS v3 rewrite fast. The honest part is what got harder: drift, verification and knowing when to stop. The developer stays hands-on.
Prolog and a knowledge graph take routing, classification and fact lookup off the LLM, so most of a personal AI system runs on a cheap local model.
Market regimes differ in volatility and correlation, not just direction. Detecting them with hidden Markov models and Wasserstein clustering, with code.
Pipelines, warehouse cost, and what actually breaks in production. A few a month, never padded to hit a schedule.
No sequence, no pitch deck. Reply 'stop' once and you're off — it reaches me, not a queue.