Building a data platform solo: the Ansaar teardown

Here are some numbers from a data platform I built and run on my own.

• 4,912 partitions backfilled in a single campaign — 99.9% success (4,909 of 4,912), in about four hours.
• Roughly 95 data assets, 50-plus ClickHouse tables, around 1,700 tests, and 56 architecture decision records.
• Indian equities and global crypto, pulled daily, run through machine-learning forecasts, backtests, portfolio construction, and a parallel compliance reasoner that checks its own work.

And here's the number that decides whether any of this is interesting: the marginal infrastructure cost is roughly zero. The only recurring cash cost is a ₹500-a-month broker API I haven't even switched on yet.

No team. No cloud warehouse bill. One person, boring tools, and a lot of tests.

This is an engineering teardown of that platform — I call it Ansaar — and of what it proves about building a data platform solo without a six-person team and a five-figure monthly invoice. It is a write-up about architecture and reliability, not investment advice, and nothing here is a recommendation to buy or sell anything.

What "a data platform" actually means here

People say "data platform" and mean very different things. Here it means the whole path, end to end: raw market data comes in at one side, and structured, queryable, tested outputs come out the other — with ML, backtesting and a decision layer in between, and a public site on top.

The stack is deliberately unglamorous:

• Ingestion and ML — Python 3.13, Polars (not Pandas), Dagster for orchestration, Playwright for scraping, LightGBM for the models.
• Storage — PostgreSQL for pipeline metadata, ClickHouse as the warehouse, and plain Parquet on network storage for the raw layer.
• Serving — a Node/Express REST API over ClickHouse, with JWT auth and a fail-open Redis cache.
• Frontend — a Next.js app that reads the API through a cached public proxy.
• Infrastructure — a self-hosted Docker Swarm cluster at home. Push to main, GitHub Actions builds and deploys. There is no cloud provider in the critical path.

Nine asset domains — equities, crypto, ETFs, ML, backtest, execution, summaries, common, and newsletter — spread across roughly 120 source files. That's the whole platform, and one person can hold it in their head.

The dataflow

The shape matters more than any single component. Data moves in one direction, and every stage is re-runnable:

Raw sources (NSE bhavcopy & F&O, Binance, yfinance, Screener.in,
             BSE, plus halal-screening feeds)
  → Polars consolidation  (quality checks, dedup)
  → validation
  → ClickHouse            (ReplacingMergeTree tables)
  → ML train / inference  (LightGBM, walk-forward retraining)
  → forecasts
  → backtest
  → plain-language summaries  (a local LLM, no API cost)
  → execution layer       (convex portfolio optimizer
                           + risk gates + compliance audit)
  → Node/Express API       (ClickHouse + JWT + Redis)
  → Next.js frontend       (cached public proxy)

Two choices in there do a lot of quiet work.

ClickHouse with ReplacingMergeTree. A columnar warehouse you can self-host on hardware you already own, and a table engine that makes re-ingesting the same partition idempotent by design — a re-run replaces rather than duplicates. That property is what makes a backfill campaign safe to run at all. A ClickHouse data platform like this handles years of daily bars and tens of millions of rows without a per-query meter attached.

Polars, not Pandas. For a consolidation step that runs across the whole universe every day, the difference in memory pressure and speed is the difference between "fits on the box I have" and "needs a bigger box." Boring, measurable, worth it.

The unusual part: a platform that audits its own decisions

Most data platforms stop at "serve the numbers." This one has a layer I haven't seen elsewhere, and it's the part I'm proudest of engineering.

Every trade decision the Python execution logic makes is independently re-derived by a Prolog reasoner (Janus-SWI) from the same ClickHouse facts. It runs in parallel to the main logic and never overrides it — its job is to produce an auditable proof tree for each decision, answer counterfactual "what if this input changed?" queries, and run formal consistency checks. The go/no-go gate is blunt: 100% agreement between the Prolog derivation and the Python result, or the decision doesn't ship.

Alongside it sits a small RDF/SPARQL knowledge graph — about 15,000 triples, built from scratch in under two seconds — that encodes Shariah screening rules (AAOIFI Standard 21 thresholds) with a citation attached to each prohibited category. A boolean is_halal flag can't tell you that a company's debt-to-assets ratio is approaching the 33% limit. The graph can, so it raises an "amber zone" alert before a hard breach, not after.

That's the "unusual depth" of the platform: two independent reasoning systems checking the same facts, one imperative and one declarative, with proof trees you can actually read. It's the kind of thing you build when correctness is the product, not a feature.

The execution layer also carries my favorite scoping story. The original plan was a 10–14 week build. A careful read of the existing backtest code showed its position selector and returns calculator already were the strategy engine — so execution became a thin adapter around them, plus the parts that genuinely didn't exist yet: entry gates, kill switches, tiered drawdown handling, and a portfolio optimizer that replaced a greedy rank-and-clamp selector with a single convex optimization solving all position, exposure and sector constraints jointly. Five to seven weeks instead of fourteen, with 87 tests around the risky parts. The cheapest code is the code you notice you already have.

The backfill campaign: 4,912 partitions, 99.9%, four hours

Backfilling is where data platforms go to die. You change a feature, fix a bug, or add a derived column, and now you have to reprocess three years of history without a human babysitting every step.

Because every stage keys its output by input partition and writes idempotently, "today's run" and "the last three years" are the same code path with a different date range. So the backfill was one campaign, not a bespoke script-and-pray exercise:

• 856 weekly-training partitions
• 2,530 daily-prediction partitions
• 1,787 daily-forecast partitions
• 525 position partitions
• 1,700 backtest partitions — 100% success

4,912 partitions across 19 assets, 4,909 succeeded, ~4 hours.

The interesting part is the three that failed. They weren't infrastructure flakes — they were a genuine train/serve feature-skew bug, and the campaign caught it instead of silently serving wrong numbers. A set of crypto prediction models had been trained before a feature-enhancement change and expected 280 features; the newer dataset handed them 580. The models couldn't score against a schema they'd never seen. Three partitions, out of nearly five thousand, flagged loudly and were documented rather than masked.

That's the whole point of doing backfill as a first-class feature. It doesn't just reprocess history — it's a test that runs your entire platform against every partition you own, and tells you where the seams are.

Three things that broke (and what fixing them taught me)

Any honest teardown includes the war stories. Here are three, because "it just works" is not a credential — knowing your own failure modes is.

1. A ClickHouse query that silently did nothing for weeks. The risk layer uses correlation-aware, dynamic risk budgets across asset classes instead of static caps. Except the queries computing those budgets nested a window function (lagInFrame) inside an aggregate (avg). ClickHouse rejects that with ILLEGAL_AGGREGATION (error 184) — but the failure was being swallowed, so the "correlation-aware" budgeting had been quietly falling back to static caps since the day it deployed. Nobody noticed, because the system kept producing plausible numbers. It surfaced only when I ran a deliberate post-deploy verification pass and rewrote the queries with CTEs. The lesson has stuck: shipping isn't done when the deploy is green — it's done when you've verified the new behavior is actually happening.

2. A recovery protocol that was a one-way trap. The execution layer has kill switches — tiered drawdown limits that halt trading when things go wrong. It also has a recovery state machine: COOLING → OBSERVATION → NORMAL. Reviewing it before it could ever fire, I found there was no code path that advanced past COOLING. Every kill-switch event wrote recovery_state="COOLING" with resumed_at=None, and nothing would ever move it forward. The safety mechanism, if triggered, would have permanently frozen the system until I intervened by hand. It took a dedicated review pass and 11 new tests to turn the trap back into a protocol. Safety code that's never been exercised is a hypothesis, not a safeguard.

3. Compliance that had to be removed at the source, not hidden. Ansaar isn't a SEBI-registered Research Analyst, so it can't publicly show forward-looking model opinions on Indian equities and ETFs. The obvious fix is to hide the UI cards. The correct fix was to delete the data-fetching calls entirely, so no signal data ever reaches the browser — while leaving the components in the codebase, unreferenced, so re-enabling them after registration is a revert, not a rebuild. Hiding a thing in the UI while still shipping it over the wire isn't compliance; it's a liability with a CSS class.

That last one is worth stating plainly, since I've mentioned the public site: any forward-looking signals for Indian equities and ETFs are gated off entirely, pending SEBI Research Analyst registration. Crypto is handled separately because it isn't a regulated security in India.

What it costs to run

This is the number most architecture posts skip.

• Infrastructure: ~₹0 marginal. ClickHouse, Dagster, PostgreSQL, the API, the frontend, and GPU compute all run on a Docker Swarm cluster I already own. Adding another asset or another year of history costs electricity, not a bigger bill.
• LLM summaries: ~₹0. Plain-language instrument summaries are generated by a small local model (Llama 3.2 3B via Ollama). No token meter.
• The only recurring cash cost: ₹500/month for a broker API subscription — and that's for live equity execution, which isn't switched on yet.

Compare that to the default reflex — a cloud warehouse, a managed orchestrator, a hosted vector DB, an LLM API for every summary — and the same platform is a four- or five-figure monthly commitment before it's produced a single useful number. I wrote about why that Snowflake bill is mostly overhead separately; Ansaar is the same argument, built out end to end.

The trade is real and worth naming: self-hosting means I own the uptime. But for a platform serving a public site through a cached proxy, with a warehouse that re-ingests idempotently, that ownership is a Sunday-afternoon problem, not a 3am one.

What this proves — and what it means for your platform

I'm not claiming a solo homelab replaces a well-run data team. Genuinely large data, heavy write concurrency, and org-wide governance still want real infrastructure and real people.

What it does prove is narrower and, I think, more useful: an enormous amount of what teams reach for cloud warehouses and big platforms to do can be done by one person with boring tools and ruthless discipline. ClickHouse instead of a rented warehouse. Polars and Dagster instead of a sprawling managed stack. A local LLM instead of an API meter. And around all of it: 1,700 tests, 56 written-down decisions, idempotent stages, and the habit of verifying your own deploys instead of trusting a green checkmark.

If your data-platform costs feel heavier than your data — or you're standing up something new and don't want to start with a five-figure monthly commitment — that's usually a fixable architecture problem, and it's exactly the kind I do for a living.

Book a call and we'll look at your setup. If you'd rather start with the playbook, the local-first stack behind a lot of this — DuckDB, Parquet, Arrow, with runnable code — is in my book, and you can grab it free here.