Do you actually need a data warehouse?

By Arshad Ansari

"We need a data warehouse" is one of those decisions that gets made by default. Someone reads that the modern data stack starts with Snowflake or BigQuery, a contract gets signed, and six months later you're maintaining infrastructure nobody can quite justify. Before you do that, it's worth actually asking the question.

Here's the framework I use with clients.

Start with the data, not the tool

The warehouse decision should fall out of your data's properties — not the other way around. Four questions, in order:

1. How large is the data you actively query? Your archive can be huge; that's a storage question. What matters is the hot slice analysts touch. Under ~100GB, a single machine running DuckDB over Parquet will out-run a warehouse and cost you nothing per query. You probably don't need one.

2. How many people (and jobs) query at once? A warehouse earns its keep on concurrency — dozens of users and pipelines hitting shared data simultaneously. A handful of analysts and a nightly job is not that. That's a laptop-shaped problem wearing a warehouse-shaped price tag.

3. Do you need a single governed source of truth? If many teams must share one authoritative, access-controlled dataset with row-level governance, that's a genuine warehouse strength. If it's really one team and a few dashboards, you're buying governance you won't use.

4. How fast is the data growing — really? Plan for the next 18 months, not the hypothetical exabyte future. "We might scale" is how teams end up paying today for problems they never have.

The honest scorecard

  • Mostly "small, few, one team, slow growth"? You don't need a warehouse yet. A files-first stack (Parquet + DuckDB + Arrow) will be faster, cheaper, and simpler — and you can always graduate later.
  • Mostly "large, many, shared, fast"? Buy the warehouse. It's the right tool and you'll be glad you did.
  • A mix? The best answer is usually both, deliberately: keep the bulk of the workload local and cheap, and reserve the warehouse for the slice that genuinely needs shared, governed, high-concurrency access.

The real failure mode

The expensive mistake isn't choosing wrong once. It's choosing by reflex and never revisiting — running (and paying for) a warehouse for years to serve a workload that outgrew the need for it the moment it stopped being warehouse-scale. Architecture should match the weight of the problem, and the problem changes.

If your stack feels heavier than your data, it probably is.

The book: Local-First Analytics walks through the whole files-first approach — DuckDB, Parquet, Arrow — with runnable code and real datasets. Chapter 1 is free to read; the rest is on Amazon.

Want a second opinion before you sign a five-figure contract? Let's talk — a 30-minute architecture sanity-check is cheaper than a year of the wrong warehouse.

Working on this for real? Get a one-week, fixed-price answer on whether DuckDB fits your workload — with the 16-question checklist in full on the page. Or run the 18-check data platform teardown against your own stack, free, no email.

Found something strange in your Snowflake bill?

Send me the cost export. I'll tell you what I'd investigate first — no call required. If it's interesting, I'll ask before publishing an anonymised teardown.

What to send, and what you get back — a written reply within 3 business days.

Does DuckDB fit your system?

The 16-question production-fit checklist I run before putting DuckDB on a critical path — writers, working set, durability, memory, and who talks to it. Each question comes with what a bad answer sounds like.

The whole checklist, sent at once. No confirmation step.

What breaks and what it costs — pipelines, warehouse bills, and the failures that only show up in production. A few a month, never padded to hit a schedule. No sequence, no pitch deck. Reply 'stop' once and you're off — it reaches me, not a queue.

Want the whole playbook?

If this was useful, the long version is my book. Local-First Analytics — 313 pages, runnable code for every chapter — is the full build: DuckDB, Parquet and Arrow, from install to production. On Amazon, and chapter 1 is free to read here.

Get the book

Not ready to buy? Read chapter 1 free — the whole chapter, no email required.

Rather talk it through? Book a free 30-minute call. No slot that suits your time zone? Email info@hikmahtech.in.