Why data platform costs drift
Data platform costs have a particular quality: they rise gradually, nobody owns the increase, and by the time anyone examines the costs in detail, it is a large enough number that the instinct is to propose replacing the platform.
That instinct is often premature. In cloud computing estates the waste is measurable. Flexera’s 2026 State of the Cloud report, based on a survey of 753 cloud decision makers, estimated wasted IaaS and PaaS spend at 29 per cent, up from 27 per cent the year before, the first increase in five years. The same survey found 85 per cent of respondents named managing cloud costs as their top challenge.
On-premises estates drift differently. Database versions fall out of support, licences are sized for hardware that has since been replaced, and servers accumulate because consolidating them was nobody’s project. SQL Server 2016 left extended support on 14 July 2026, which means an estate still running that version now receives no security fixes without paying for Extended Security Updates, which are expensive. Out-of-support software is a serious risk to any organisation and should be addressed immediately.
Where does your data platform spend go?
The below items appear obvious, but are missed or ignored remarkably often. Before any modernisation decisions are made, we suggest a review of the following:
- Cost by workload, rather than total platform spend alone. Which workloads, which queries, which consumers.
- The most expensive queries. In most platforms a small number of queries account for a disproportionate share of compute, and some of them feed information that is no longer required.
- Storage by tier and by age. How much hot storage is holding data nobody has queried in two years? Do you have redundant, obsolete or trivial data that can be archived or deleted?
- Idle and over-provisioned compute. Clusters running overnight on oversized VMs with nothing scheduled, warehouses sized for a peak that occurs only monthly or annually, servers at single-digit utilisation.
- Full reloads that could be incremental. Pipelines reprocessing all history instead of processing only what has changed, because that is how they were first configured.
- Versions and support status. Which SQL Server instances only require Standard edition licences rather than Enterprise, and can they be consolidated to reduce licensing and management costs? Which servers are out of mainstream support, and what does that cost in extended support or unpatched risk?
That exercise takes days rather than months to complete, and in Northdoor’s experience it routinely identifies enough waste to fund the rest of the programme.
Which changes give the highest return?
Remove redundant processing. The most common finding. The same transformation implemented in multiple places because multiple teams required it. Full table reloads where incremental processing would do. Pipelines still running for decommissioned services or reports. Intermediate tables materialised and never read. Each item is small; together they are often the largest single component of waste.
Tier storage by access pattern. Access patterns change and tiering is usually configured once and forgotten. Cloud object storage offers tiers priced by access frequency, Azure Blob Storage for example has hot, cool, cold and archive tiers with lifecycle policies to move data between them automatically. Moving cold data down a tier is close to free to implement and has no performance consequence for data nobody is querying. The same principle applies on-premises through storage classes and archiving.
Size compute to real concurrency. Data warehouses, VM clusters and servers are typically sized for the worst case scenario and left there. Separating heavy analytical workloads from interactive dashboards, sizing each to its actual requirement and suspending anything intermittent when idle generally produces a step change.
Stop reprocessing history. Converting a full-reload pipeline to incremental is real engineering work, and on a large table running regularly it is frequently the single largest cost reduction available. It also improves latency, which is the performance benefit users notice.
How does the same work improve performance?
Consumers experience performance three ways: how long a query takes to complete, how fresh the data is, and whether it is reliable. Each maps onto the work above.
Query latency improves through partitioning and indexing aligned to how data is actually filtered, appropriate materialisation of expensive aggregations, and removing contention by separating workloads. On an out-of-support database, simply moving to a current version often brings query engine improvements that have accumulated from the vendor over several releases.
Freshness improves through incremental processing, because a pipeline that handles yesterday’s changes finishes far sooner than one reprocessing three years’ worth of history.
Reliability improves through the DataOps practices of automated testing and monitoring, which stop failures propagating silently into reports.
When is migration the right answer?
Sometimes the platform is the problem. The signals worth taking seriously:
- The platform cannot scale to the volume you now have, and tuning has been exhausted.
- It cannot support the workloads you need, for example semi-structured data or streaming, without significant workarounds.
- Licensing or support costs are rising faster than the value delivered, or the version is out of support and the upgrade path is itself a migration.
- Operating it requires skills you cannot recruit anymore.
- The architecture forces choices you would not make today, such as storage and compute that cannot be scaled independently.
If several of those items apply to your environment, then we would suggest that a migration is warranted. If only one does, assess, tune and reassess. The FinOps Foundation’s 2026 State of FinOps survey found that mature practitioners now report diminishing returns from optimisation alone, having already removed the large, obvious waste. The implication for a team that has not yet undertaken that work is the reverse: the first pass is where the big savings sit.
A sensible sequence
Measure: cost and performance by workload, plus licence edition, version and support status. Do not skip this.
Remove: redundant processing, unused pipelines and tables, retention applied. The cheapest phase and often the largest saving.
Tune: storage tiering, partitioning and indexing, compute sizing, incremental conversion for the biggest offenders, version upgrades where the workload is remaining.
Reassess: with the waste removed, the case for migration either becomes much clearer or falls away.
Migrate: if warranted, incrementally, workload by workload, running in parallel and comparing outputs rather than switching wholesale.
Instrument permanently: cost and performance monitoring per workload, with a named owner, so drift is visible and can be managed as it happens.
Northdoor’s data platform modernisation services start with a readiness assessment covering steps one to four, and are deliberately open to the answer being ‘keep the platform, fix the estate’.
What this means for your organisation
Organisations that work through the first three steps typically find meaningful cost reduction available without any architectural change, alongside better performance. The exact figure depends on how long the platform has run without a cost or optimisation owner, which is why the assessment should come first.
The more durable outcome is the last step. A platform with cost and performance monitored per workload, and a named owner, does not drift back to old ways. One without them will, whichever platform it is. If a migration is on the table, the measurement is also the only reliable way to size the business case; if it is not, it is the shortest route to a smaller platform bill.