Blog
Pipelines, analytics infrastructure, ML systems, and lessons from building them in production.
The real drivers behind the cost of a data pipeline or platform, honest market ranges, and why the recurring bill matters more than the build price.
Why a day rate is the wrong way to price data-engineering work, how I actually structure engagements, and how to think about what a data platform is worth to you.
A practical comparison, not a benchmark war. When a single-node engine like DuckDB is enough, when you actually need Snowflake, and how to tell the difference.
The real fully-loaded cost of a full-time data engineer, when a consultant is the better call, and how to tell which one your situation actually needs.
The demo is easy; the production version is where people get hurt. The guardrails that let an LLM query your database without dropping a table or leaking data.
Managed elastic warehouse vs. a columnar engine you run yourself. When ClickHouse wins, when Snowflake is worth the bill, and what changes when you actually operate it.
What it takes to make natural-language querying actually useful for non-technical teams — the schema semantics, guardrails and verification the demos skip.
A teardown of a macro data product: 171 countries scored daily from 10 free public data sources, point-in-time correct, with the licensing and cost work done.
Four LLM agents, 28 Temporal workflows, every risky action behind a human approval gate. An architecture teardown of AEGIS — and what it teaches about production AI automation.
NSE and crypto ingestion, ClickHouse, LightGBM, backtests and a Prolog compliance layer — a self-hosted data platform built and run by one person.
Three months ago I wrote about my personal AI orchestration system and ended with a section titled 'Why It's Not Open Source Yet.' That's fixed. AEGIS is on GitHub, MIT-licensed — here's what it actually took to get there.
The biggest refactor of the AEGIS open-sourcing sprint: removing every branch on an agent's identity so behavior lives in the database as capability tags — resolved at runtime, edited from a UI, and safe even when a tag has no owner.
Scheduling a social post doesn't need a new UI. A to-do already has copy, a time, and labels — so in AEGIS, a Todoist task with a publish label is a scheduled post, and nothing goes out until a card in chat gets a human tap.
For a self-hosted system that's meant to be forked, there's exactly one honest way to handle infrastructure credentials: the user brings their own, the system stores them encrypted, and nothing in the code assumes a vendor. Here's how AEGIS got there — and what it cost me to cut my own vendors out.
The day Next and Someday stopped being Todoist projects and became labels — and why that small modelling decision is what made AEGIS's GTD layer, and its bidirectional sync, actually work.
AEGIS is going open source, and today the first public commits landed. Day one of turning a private system into a shippable one: twenty-seven migrations squashed into a baseline, credentials evicted from the build, and an admin panel redesigned around decisions.
If you're paying thousands a month to query tens of gigabytes, you're renting a freight train to carry a backpack. Here's how to tell, and what to do instead.
AEGIS watches my homelab and GitHub, but the interesting part is what happens between an alert firing and me hearing about it. Most of the time, the correct answer is nothing.
DuckDB is having a moment, and the hype outruns the nuance. Here's an honest map of where an in-process analytical engine belongs in a real production system — and where it doesn't.
A short, honest decision framework for the question every data team eventually faces — before you sign a five-figure annual contract for infrastructure your data might not need.
Most financial data pipelines break for boring, predictable reasons. Here are the five habits that separate a pipeline you babysit from one you forget about.
A small SQL-tuned model running on your machine can convert plain English questions into correct SQL. No API keys, no data leaving your laptop, no vector database.
A concrete walkthrough of the local-first pattern: query Parquet directly with DuckDB, no warehouse, no server, no per-query bill — from a notebook or the browser.
Run a model directly in SQL to classify rows, extract text fields, or summarize data without exporting. Here's where it works, what it costs, and when not to bother.
Most analytics stacks are cloud round-trips solving problems that fit on a laptop. Local-first analytics is the case for bringing the compute back to the data — and to the user.
Parse PDFs to markdown, extract structured data with a local model and strict validation schema, no API needed.
Use the Model Context Protocol to let Claude explore your data in plain English. The whole game is permissions: read-only role, standard server, 90% of the value with almost none of the risk.
Most teams reach for Pinecone or Weaviate the moment they hear semantic search. For a team-sized knowledge base, you don't need one. Your database already has what you need.
Claude Code can run dbt in a loop, read errors, and fix them. It needs you to steer grain, business logic, and naming — and to review the tests it writes.
Structural tests catch NOT NULLs and schema errors. Semantic problems—does this description match its category?—need a judge. How to use one responsibly.
An LLM is probabilistic, slow, and costly. Here's how to decide whether it belongs in your pipeline — and where it clearly doesn't.
When does running an open-weight model on your own hardware beat paying per token? The answer depends on volume, frequency, and whether you have real privacy constraints.
How to harden a data pipeline by feeding it carefully generated edge cases, using a local LLM when simpler generators aren't enough.
Local-first, file-based data has a real ceiling. Most teams never reach it. When you do, here's what actually changes.
Batching, checkpointing, and idempotency are what make local inference scale. Get these right and a single GPU chews through millions of rows overnight.
The design decision in AEGIS I'm proudest of isn't an agent or a model — it's a single interactions primitive. How one Postgres table, five kinds of card, and a Temporal workflow replaced every per-domain approval pattern.
Every week there’s another agent demo, another workflow canvas, another pitch about software running software for us. OpenClaw, browser agents, Zapier, n8n, Claude Code, MCP s
I have been rebuilding a personal project called AEGIS into its third version, and on paper this rewrite should have dragged on for weeks. It did not. Claude Code helped me mo
I’ve been building a personal AI orchestration system. The first version — Jarvis — worked, but became a 20,000-line monolith I dreaded touching. So I rebuilt it from scratch
Financial markets move through distinct phases — bullish rallies, sharp crashes, quiet consolidations, and volatile swings. These market regimes differ not just in price direc