According to IDC, over 527 zettabytes of data will be generated annually by 2029 – up from just 39 ZBs in 2018. Yes, that’s zettabytes with a ‘z’ – a number that consists of 21 zeros and is so large that humans have difficulty even conceiving how much that actually is. If 1 TB were equal to a kilometer, 527 ZBs would allow you to take about 685,000 round trips to the moon and back, taking about 20 days at the speed of light.
Regardless of how you try to conceptualise a number this massive, it’s clear that most organisations and cloud service providers (CSPs) are looking to expand and optimise their storage infrastructure to keep up. The challenge is not just about adding more storage, it’s about doing so in a way that maximises value, business continuity and lowers total cost of ownership (TCO).
Traditionally, expanding storage capacity often meant adding more servers, networking, and compute alongside storage devices. However, this approach can lead to increased power consumption, cooling requirements, and unnecessary data centre sprawl. To avoid these challenges, many organisations are looking for ways to scale storage more effectively by prioritising density over footprint growth.
One way to achieve this is through higher-capacity HDDs, which provide greater storage within the same physical footprint. Another approach is disaggregated storage, which allows storage capacity to grow independently from compute and networking resources. This model enables businesses to right-size their storage expansion, reducing overprovisioning and optimising TCO.
As data demands continue to accelerate, organisations will need a smarter approach to storage growth: One that maximises capacity, enables long-term scalability, and ensures that storage infrastructure can keep pace with growing workloads.
Simply adding drives isn’t sustainable
The adoption of AI, ML and large language models (LLMs) into every aspect of our lives is growing exponentially. People are using these models to perform genomic sequencing, create intelligent chatbots, provide real-time fraud detection and more. Each of these applications are continuously trained on petabytes of information while producing outputs throughout the AI data cycle.
Australia is emerging as one of the most dynamic data centre markets in the world, with AI adoption driving unprecedented demand.
The current investable universe for data centres in Australia is valued at approximately AUD 30 billion and projected to grow by around 50% over the next four years, reaching AUD 46 billion – driven by the rising demand for AI. The country already ranks among the top ten for live built-out capacity, and with a robust pipeline based on existing, under-construction, committed, and early planning stages, Australia is looking at becoming the third largest globally in terms of data center capacity.
However, increasing capacity by scaling out can add immense costs. In addition to CPU resources and networking, scaling out requires physical real estate in a data centre, cooling, climate control, wiring and cabling. Humans need to staff these data centres, adding human resource costs. These sprawling data centres have an environmental impact as well, consuming massive amounts of energy.
As data needs continue to grow, organisations and CSPs will need to think beyond just adding more resources or components and start thinking about incorporating high-capacity HDDs and storage disaggregation for better TCO benefits.
The death of HDDs has been exaggerated
Solid state drives (SSDs) have been slowly gaining market share over the past several years due to the need for ultra-high performance, low-latency storage solutions. However, HDDs are the backbone of the AI-driven data economy and are still the best solution for storing massive amounts of data effectively at scale. According to Western Digital calculations, the acquisition cost of HDDs can be six times less expensive per terabyte than SSD solutions. And today, nearly 80% of cloud data is stored on HDDs as they provide the best option for low TCO at scale. . With that said, every storage technology, from HDDs to SSDs and tape, has a role in the stack. It’s not an either-or option. It’s selecting what works best for the business and the workload.
Improving storage density can have the same effect as adding drives but without all the side effects. You still get more storage space for expanding needs but don’t have to increase the data centre footprint. Think of a hotel in a highly trafficked part of town. Developers can save on expansion costs by increasing the number of rooms per floor through better space utilisation and by repurposing unused common space. In contrast, adding capacity by building additional floors up or by adding another wing to the building is much less efficient due to the additional overhead the increased floor plan would create.
This “smarter” way of increasing capacity can be applied to storage architectures. Increasing storage density through higher-capacity HDDs and optimising storage architecture allows organizations to grow their capacity without excessive infrastructure sprawl. By maximising how much storage fits within existing resources, organisations can better align storage and data growth while maintaining power, cooling, and real estate constraints.
Disaggregation of storage and compute resources can also help to meet this demand. For instance, organisations have typically had to scale storage and computing resources in concert with each other. By disaggregating storage from compute, and sharing storage resources across applications via NVMe-oF, storage can be increased independently of compute. This ensures that workloads, including AI applications, have fast access to the information they need without forcing organisations to over-provision resources across the entire data environment. Improving data center utilisation in this way is more sustainable while meeting the needs of data-hungry AI applications.
Together, higher capacity HDDs and disaggregated storage innovations are not only powering the next wave of AI-driven workloads but also supporting sustainable, scalable data centre growth – ensuring Australia can keep pace with rising global demand for high-capacity infrastructure.
Rise of the yottabytes
Data needs are expected to explode over the next year and continue to grow as AI, high-performance computing and other data-intensive applications become more ubiquitous in data centres. Instead of expanding infrastructure to meet the storage demands, organisations should first look for ways to maximise storage density by leveraging new innovations for high-capacity HDDs in addition to disaggregating and scaling storage independently from compute resources. There will be a day when global data generation will grow too large to be measured in zettabytes, and organisations need to have the density in place to be ready for the era of yottabytes (24 zeros).
By Stefan Mandl, VP of Sales and Marketing APJC at Western Digital
This article was first published by In AI Today