Data management A Decade in Review: Challenges in Data Storage 2014 and Today.

While a decade brings little change in other areas, the technology of data storage and management experiences a revolution in this time. How have the challenges and approaches to solutions changed since 2014, and what do experts expect for the future?

Federica Monsone is the founder and CEO of A3 Communications.

Einstein knew that the perception of how quickly time passes depends on the perspective of the individual observer. And for a dog, a year might feel just about as long as the span of seven human years. In the data storage industry, however, changes occur much faster than in many other areas of human activity.

How can the challenges of data storage and management that companies were faced with ten years ago be compared with those of today? What do experts say about the ways in which the current storage landscape and its increasingly complex challenges are influencing technology developments?

What was then, what is now: Changing storage requirements

More than one expert is on record as saying that the challenges of data storage that IT organizations faced in 2014 are very similar to those of today—at least at a high level. "The challenges haven't changed much, even though the technology has. The biggest challenge was probably dealing with the constantly increasing storage capacity requirements. The second challenge was protecting the data. Even though the intensity of ransomware attacks wasn't as high as it is today, data protection was a major issue. The third challenge was that there weren't enough staff available to handle the volume of storage. This personnel problem has only gotten worse since then," says Randy Kerns, Senior Strategist and Analyst at analyst firm Futurum Group.

Brock Mowry, CTO & VP of Products at storage system provider Tintri, agrees but adds an important caveat. "The challenges are essentially the same as ten years ago, but the scale and magnitude of these challenges have dramatically changed," he says.

Pain Point Storage Capacity: Data Growth with No End in Sight

Erfane Arwani, CEO of Biomemory, a start-up company specializing in DNA storage and synthesis, underscores the difficulties in keeping up with data growth in 2014: "Companies were struggling to handle exponential data growth with technology solutions that were not yet optimized for large volumes of data." Arwani points out that company hard drive capacities ten years ago only ranged between 1 TB and 4 TB. In the decade since, hard drive capacities have dramatically increased. Today's highest-capacity hard drives can handle 30 TB. At the same time, the use of flash storage in data centers has greatly increased, and the largest enterprise flash drives now have a capacity of over 60 TB.

In 2014, companies were still focused on on-premises storage and used public cloud storage services to a lesser extent than today. "The choice was between NAS and SAN, and cloud solutions were comparable to ice baths - beneficial, but not suitable for everyone," says Ferhat Kaddour, Vice President of Sales and Alliances at Atempo, a provider of data protection and management software. Ensuring sufficient total capacity for a company was a complex task. "The challenge of scalability lay in predicting future storage needs, optimizing storage utilization, and implementing effective strategies for storage allocation," says Drew Wanstall, Vice President of Business Development at Scale Logic, a provider of storage and workflow infrastructures for media production.

Today, the volume of data is still growing rapidly. "It's interesting to see how data is growing at a mad pace," says Enrico Signoretti, Vice President of Products and Partnerships at Cubbit, a provider of geodistributed cloud storage systems. Valéry Guilleaume, CEO of Nodeum, a provider of data management software, named some of the new data sources that are driving this growth and have already ushered in the so-called era of Big Data. "Today, it's not just users who generate data, but also the systems that are being developed in various industries, such as data generating cars, electronic microscopes, blade scanners, or even seismic sensors. These new sources generate data at a speed that is incomparable to the data generating sources from ten to fifteen years ago."

The difficulty of increasing physical storage capacity to keep pace with data growth has been somewhat mitigated by the increasing use of public cloud storage and by improvements in data storage technology. Among the technological developments of the past ten years, the enormous price drop in flash memory, which has led to widespread use of flash memory in corporate data centers, stands out in particular. "The demand for capacity continues, but the size and performance of flash allow for greater consolidation and fewer physical systems, less energy, cooling, and space requirements, and simpler means to increase performance," says Kerns. "The technology to solve problems is available and more effective than ten years ago."

While other experts believe that the scalability of storage remains a major problem, Kerns' view is shared by further industry analysts. "More data makes management more complex, but less complex than in the past. Storage solutions are much more scalable today than they used to be. The data explosion, especially in the field of artificial intelligence, poses the difficulty of finding the right data, putting it in the right, clean format, and making it readily available. The challenge today lies not so much in storing data, but rather in using it," says Scott Sinclair, Practice Director at the analyst firm Enterprise Storage Group (ESG).

Data Security, Data Protection, Data Mobility – Modern Requirements

David Norfolk, Practice Leader at analyst firm Bloor Research, says: “The technical problems of ten years ago are largely gone. Storage today is cheap, reliable, and easy to scale. But storage management - including threat management - is now a cost factor."

The threats Norfolk mentioned also include cyber attacks, which, according to several experts on our panel, have significantly increased in number and intensity over the last ten years. "Security is clearly the biggest challenge for data storage today. While there have always been security threats from malicious actors and users, today's issues are indeed harder and more costly to deal with, which is due to the well-organized and financed ransomware actors who often belong to state-supported groups," says Paul Speciale, Chief Marketing Officer at object storage specialist Scality.

"Given the ongoing ransomware boom and the emergence of malicious AI tools and as-a-service cybercrime models, data protection is now at the top of the challenges in the storage area. Not only is the number of security breaches increasing, but also their impact with a view to improved tactics (multiple extortions), or the dual-strain attacks observed of late," says Sergei Serdyuk, Vice President of Product Management at Nakivo, a provider of backup, ransomware protection, and disaster recovery solutions.

That's not the only change in the IT landscape which has driven up the costs of storage management. Ten years ago, data growth was driven by the general digitization of the economy and the increasing use of analytics. Now, it is also driven by the need to collect data to train AI and machine learning systems and, as Guilleaume describes, by the growth of the internet of things as a data source. Although the term IoT was coined in the 90s, it has only become a commonplace reality in the last ten years. At the same time, companies are also generating more unstructured data. These now make up the majority of the data stored by businesses. Unlike structured data, unstructured data are not organized according to a predefined database schema, which significantly complicates their management.

High data quality - the be-all and end-all for a future-proof data concept.

"Today, you have to navigate a vast ocean of big data. From customer interactions to collected sensor data - even smaller companies process petabytes, larger ones even exabytes. The difficulties lie not only in the sheer amount of data, but also in the strategic tactics that are needed to extract, categorize, and protect them," says Kaddour. Norfolk of Bloor Research mentions a critical goal which is difficult to achieve when using unstructured data: "Quality, because the data comes from a swamp, not a proper database."

"Efficient management of data at the edge has become critical. Ensuring data availability as well as fault-tolerance in distributed environments pose new challenges," says Johan Pellicaan, Vice President and Managing Director at Scale Computing, a provider of edge computing, virtualization, and hyperconverged solutions.

Companies not only have to secure their data at the edge, but also be able to move data between different locations. "Today's challenges all relate to moving data in multi- and hybrid cloud environments. About 50 percent of companies report that they are constantly or regularly moving data between on- and off-premises environments. These issues are harder to address because the environments are so different when the data spans AWS, Azure, GCP, the data center, or the edge," says Sinclair of ESG.