A data platform you won't outgrow
The platforms that last are not the ones with the most tools — they are the ones with clean seams. A few patterns for building data infrastructure that scales with you instead of being rebuilt every two years.
Almost every data platform gets rebuilt. Not because the technology was wrong, but because the first version hard-wired assumptions that stopped being true — a single source system, a reporting-only workload, one team. The goal is not to predict the future; it is to build seams so the future can be absorbed without a rewrite.
Start from questions, not tools
The fastest way to over-build is to pick a stack first. Start from the decisions the business needs to make and the questions behind them, then work backwards to the data and the latency those questions actually require. Most "real-time" requirements are hourly; most "big data" is a few hundred gigabytes. Right-sizing here saves more than any tool choice.
The layers that scale
A durable platform separates concerns so each can change independently:
- Ingestion — decoupled from sources, so adding or changing a source does not ripple downstream.
- Storage — an open lakehouse format (open table formats over object storage) so your data is not locked inside one engine.
- Transformation — versioned, tested and reproducible, so logic is auditable and safe to change.
- Serving — warehouses, BI and APIs read from the same governed layer, not from private copies.
- Governance — cataloguing, lineage, quality and access built in from the start, not bolted on later.
The value is in the seams between these layers. When storage is in an open format, you can swap query engines without migrating data. When transformation is versioned, you can change business logic without fear. When serving reads from one governed layer, you avoid the sprawl of conflicting copies that eventually forces a rebuild.
Cloud and on-prem are not either/or
Plenty of organisations have good reasons to keep some data on-prem — regulation, latency, sunk cost in existing systems. An open storage layer and portable transformation logic let you run the same patterns in both places and move workloads when it makes sense, rather than committing everything to one location on day one.
Govern early, lightly
Governance added after the fact is a migration; governance designed in is a habit. You do not need a heavy programme — you need lineage so people trust the numbers, a catalogue so they can find data, quality checks that fail loudly, and access controls that match your obligations. Trust is what makes a platform actually get used.
Avoid the rewrite
You will not get every decision right, and you should not try to. Build the seams, keep your data in open formats, version your logic, and govern from the start. Then when the business changes — and it will — you extend the platform instead of replacing it.