What a data pipeline actually costs: build, run, maintain

"How much does it cost to build a data pipeline?" is a fair question with a frustrating answer: it depends. But "it depends" is useless on its own. What it depends on is knowable, and once you see the drivers you can put a real range on your own situation. Here's the honest version — build, run, and the maintenance cost nobody quotes you.

Why there's no single number

Two projects both called "a data pipeline" can differ by 20x in cost, because these things drive it:

• How many sources, and how messy. One clean Postgres database is a different job from twelve APIs, three of which paginate badly, rate-limit you, and change their schema without warning.
• Batch or real-time. A nightly batch job is straightforward. Sub-minute freshness — streaming, change-data-capture, exactly-once handling — is a different discipline and cost.
• How much transformation and modelling. Moving raw data is cheap. Turning it into correct, tested, business-ready models — with the data-quality checks that keep it correct — is most of the work.
• Volume. Gigabytes fit on one machine and stay cheap. Genuine tens-of-terabytes-per-day changes the architecture and the bill.
• Serving and dashboards. Who consumes it, how many of them, how fast it needs to be.
• Compliance and integration. PII, audit requirements, and having to fit into systems you can't change all add real cost.

The three ways to pay for it

1. Build it in-house. You pay in engineer-time. That's a senior data engineer's fully-loaded cost (often $175k–$250k/year all-in) for the months it takes, plus the opportunity cost of what they're not doing. Cheap-looking if you already have the person idle; expensive if you're hiring for it. (More on that trade-off in consultant vs. full-time hire.)

2. Buy tools and glue them. The modern SaaS stack — a managed ingestion tool, a transformation layer, a cloud warehouse, a BI tool — gets you running fast. The catch is recurring cost that scales against you: per-row ingestion pricing, per-second warehouse credits, per-seat BI. It looks cheap at signup and grows quietly. This is where a lot of "our data costs are out of control" stories start.

3. Bring someone in to build it. A consultant or contractor delivers the pipeline as a project. You pay once for the build (plus whatever recurring infrastructure it runs on), and ideally you own it afterward.

What the build costs: honest ranges

Broad strokes, because the drivers above move these a lot:

• A simple pipeline — a handful of clean sources, batch, into one warehouse, with basic dashboards — is typically a low-to-mid five-figure project, or a few weeks of senior time.
• A real platform — many sources, proper modelling and testing, some real-time, ML or serving, governance — runs mid-five to six figures, or a few months.
• The recurring infrastructure to run it can be anywhere from near-zero (self-hosted, right-sized) to thousands a month (a full managed SaaS stack). This is the number people underestimate most.

The build is a one-time cost you can see coming. The recurring bill is the one that surprises you — which is why it's worth estimating your warehouse spend before you commit to an architecture, not after.

(A note on how that build gets priced, rather than what it costs: I don't quote a day rate, for reasons I've written up separately in what data engineering costs and how I price it. This post is about the cost of the thing; that one is about the shape of the deal.)

Typical shapes by stage

The same three ranges, mapped onto where teams usually are when they call. These are the shapes I quote, not a price list — your drivers move them.

Seed. One or two sources, a founder or generalist engineer currently doing it by hand, and the real goal is "stop making decisions from a spreadsheet someone updates on Fridays." Usually the low end of the five-figure build, and it should be. The right architecture here is deliberately small: your production database plus a lightweight analytical engine, batch, no streaming, cheap infrastructure — a running cost that can genuinely be tens of dollars a month. If someone is proposing a full managed stack at this stage, they're selling you your Series B problem early, and you may not need a warehouse at all yet.

Series A. Several sources, a growing team wanting numbers that agree with each other, and the first person whose job is partly "own the data." This is where proper modelling, tests and an orchestrator start to earn their keep, and where the build lands mid-five figures. It's also the point at which the recurring bill becomes a line item someone notices, because usage has grown and nothing was right-sized on the way up.

Series B. Many sources, multiple consuming teams, governance and access control as real requirements, probably something that needs to be fresher than daily. Mid-five to six figures for the platform, plus — and this is the part that gets left out — a permanent ownership cost. At this stage the pipeline isn't a project that finishes; it's a system somebody is responsible for from now on.

What a pipeline costs to run and maintain

The build price is what gets negotiated. The running cost is what you live with, and it has three parts.

Engineer time — the big one. Pipelines are not appliances. Upstream APIs change their schema, a vendor deprecates an endpoint, a source starts sending nulls in a column that was never null, someone asks for a new field. None of that is a failure of the build; it's the cost of being connected to systems you don't control. The honest planning number is a recurring share of someone's attention: a stable batch pipeline over two clean sources might need a few hours a month, while a platform pulling from a dozen third-party APIs is a permanent part of somebody's job. Estimate it deliberately — for most teams it dwarfs the infrastructure bill, and it's the line that never appears in a proposal.

The recurring bill. Warehouse compute and storage, ingestion tooling, orchestration, BI seats, observability, egress. The shapes matter more than the totals — see the next section.

Monitoring and the cost of failure. Somebody has to know when it breaks, and "our dashboards were stale for four days" is a real cost even though it never lands on an invoice: decisions made on stale data, plus the scramble to work out what else was wrong. It's why I argue for pipelines that don't page you — alerting and idempotent retries are a small build-time cost that removes an ongoing operational one. Recovery isn't free in cash either: a re-run costs you the compute twice, and post-outage backfills are frequently the most expensive queries a team runs all quarter.

The low end of that recurring bill is achievable, and I'm not describing it hypothetically — I run a full production platform at near-zero marginal infrastructure cost, and the teardown shows the choices that make it possible.

Managed ETL pricing vs building your own

The comparison people want is "tool X versus writing it myself." The more useful comparison is between cost shapes, because the shape determines what happens to you at 10x your current volume.

What managed tools charge for, by category:

• Per row, or per monthly active row — the common model for managed ingestion connectors. It scales with your data volume and with churn: sources where records update constantly cost far more than their row count suggests.
• Per credit or per second of compute — cloud warehouses. Elastic and fair for spiky work, punishing for steady, constant querying.
• Per seat — BI tools, transformation platforms, catalogs. This one scales with your headcount, which is the metric least connected to the value the tool provides.
• Per connector or per source tier — you're fine until the twelfth source moves you up a plan.
• Per run or per task — orchestration platforms. Usually small, until someone schedules something every minute.

What building it yourself charges for: engineer time up front, cheap infrastructure afterwards, and a maintenance tail you own forever. The trade is genuinely opex versus capex — managed tools convert your problem into a bill that grows with your success; building converts it into a fixed asset with a running human cost.

The rule I use: buy the commodity connectors, build the ones that are core, high-volume, or weird. A managed connector to a standard SaaS API is excellent value — that integration is undifferentiated work you'd hate maintaining. A per-row bill on your highest-volume event stream is the opposite: you're paying a scaling premium on the pipeline that's most central to your product and most predictable to build. Most out-of-control data bills I see are that second case, unnoticed for a year.

How to cut cost without cutting corners

There is a version of "cheaper" that just means "worse later." These four aren't that.

Batch until streaming is earned. Real-time is a different discipline at a different price, in both build and operating cost, and most "real-time" requirements dissolve under the question "what decision changes if this number is an hour old?" Sometimes the answer is real. Usually it's that a dashboard felt more impressive updating live.

Right-size the warehouse, or don't buy one yet. You're billed on compute you provision and queries you run, and most teams over-provision both and never revisit it. A smaller warehouse, fewer scheduled refreshes, and materialising what gets queried repeatedly instead of recomputing it will take a real bite out of the bill without changing a single answer.

Self-host the boring parts. Orchestration, BI and metadata tooling are mostly open-source software that runs fine on a modest box. It's a trade — you own the upgrades — but per-seat and per-run models are where bills grow for reasons unrelated to the value you're getting.

Keep the surface small. Every source, table and dashboard carries a maintenance cost forever. The cheapest pipeline is the one you didn't build for a report nobody opens. Audit what's actually used once a year and delete the rest.

The cheapest build is often the most expensive to run

The instinct is to minimise the build cost. The mistake is ignoring what the choices you make during the build do to the monthly bill for years afterward. A pipeline built on per-row and per-second pricing can cost more in eighteen months of running than it did to build. A leaner architecture — right-sized infrastructure, engines that don't meter every query — can cost a little more to design well and a lot less to live with.

How to get a real number for your situation

Sketch your drivers: sources, freshness, volume, how much modelling, who consumes it. That alone turns "it depends" into a range you can reason about. To turn it into an actual number — build cost and the recurring bill — the work is a proper scoping of your specific stack. That's exactly what a Data Platform Audit produces: a week, a written roadmap with effort and cost estimates, yours to keep. The scoping call below is free and there's no pitch.