Unstructured Data Management Predictions for 2024

If you don't trust or admire AI, the year 2023 was not a fun year for you. Almost daily, we are hearing about another new product or service powered by AI, especially generative AI, that will change our lives. In the B2B world, AI technology is not yet everywhere but, according to a recent Gartner poll, 55% of organizations are in piloting or production mode with generative AI. In 2024, we will begin to see a sensible rationalization of AI technologies. Innovation will continue, but there will be more focus on how to incorporate AI technologies safely, ethically, and logically into the workplace and into customer-centric processes. There's a lot to figure out here. 

The average enterprise IT leader, however, is not thinking about AI all day long. They need to continue being cost-effective by automating and streamlining operations and being vigilant in the face of cybersecurity threats as they modernize hybrid, multi-cloud infrastructure. They need to balance resiliency and agility: Business counterparts want real-time analytics, data services, and digital intelligence to get ahead. The elephant in the room is that unstructured data, which started out as some user data on file shares, has rapidly grown in the past decade: Today it is roughly 90% of all data, and this growth is continuing. AI relies on unstructured data, which increases the significance of proper data management, data mobility, and data governance of unstructured data.

Related:How Sustainable Data Storage Can Reduce Corporate Emissions

Here are a few top trends we see for next year from the data storage, data management, and IT infrastructure perspective.

IT leaders focus on AI foundations, efficient storage, unstructured data management & FinOps. 

The year 2023 started off with much talk about cloud cost control and even repatriation, amid layoffs and a rollercoaster economy. In 2024, enterprise IT organizations should be in a better position to make new strategic investments, if inflation continues to decline and the labor market remains strong. As generative AI adoption grows, IT will need to implement proper data governance strategies to minimize risks. According to research conducted by Komprise and others in 2023, preparing for AI is a top priority. This requires the highest-performing data storage, backup, and data protection technologies; new security and privacy tactics; use-case specific AI and ML tools; and automated unstructured data management solutions that enable custom data workflows and governance.

Subscription-based storage as a service (StaaS) is another growing area that improves efficiencies with the vendor managing the hardware and software. Cloud optimization will trump cloud repatriation in 2024 after a year of caution. There will be an emerging array of new services and products available, such as FinOps tools integrated into the enterprise IT infrastructure stack to make this easier.  

Unstructured data migrations become more intelligent and automated.  

Data is increasingly in motion as IT needs to leverage new storage technologies and satisfy new business requirements. Enterprise data migrations of unstructured file and object data have long been complex and too manual and often require professional services. Automation and AI tools will change this, enabling intelligent, efficient data migrations that no longer need IT managers to babysit them and they will also be adaptive. Modern tools will know how to solve problems on the fly and self-remediate and will be able to recommend optimal storage tiers for different unstructured data workloads and use cases.

This is a timely development, as data migrations are becoming more varied all the time and dependent upon the customer's changing environment — from firewall to network connections to security configurations. Intelligent migration software will provide an order of magnitude faster cloud migrations with better long-term outcomes and fewer instances of data loss, errors, and security risks.

Organizations must plan for more frequent disasters and outages with a nuanced approach as one-size-fits-all backups are no longer economically feasible for unstructured data.  

With two major global conflicts now underway, a pandemic that has become endemic, ongoing pressures on the global economy and supply chain, an accelerating frequency of climate calamities, and growth in ransomware attacks, uncertainty is the new normal. Organizations need to be better prepared for a variety of disasters and distractions. That means more than ever, IT needs to understand data needs and threats and invest in the right infrastructure, storage, and security products and services to be resilient. This requires a different approach than traditional backup and data protection, since those solutions are too expensive and therefore not tenable on all data.

Unstructured data management will deliver affordable resiliency at a fraction of the cost, by creating cheap copies in durable object storage in the cloud for non-critical data — which is the bulk of all data in storage. This "poor man's data resiliency" approach will complement the 3x backup method for mission-critical data to create a cost-effective and holistic disaster recovery strategy. 

AI data governance will require a multi-layered approach, and storage leaders need to play their part. 

Whereas data governance is a mature enterprise IT practice, AI has introduced new challenges and uncertainties. Generative AI has created a multitude of risks from privacy and security to data leakage, transparency, accuracy, ethics, and more. Rather than procuring one system to manage these different issues, IT will deploy layers of AI security tools and practices, starting at the network layer to prevent the access of blocked data by an AI tool or prevent users from sending corporate data to unauthorized AI services. A second level of protection sits at the data layer to audit which data was moved, where, when, and by whom and alerts if PII or other sensitive data is being shared. Finally, a security mechanism could exist at the user layer to warn users when they are engineering prompts with corporate or sensitive data or provide feedback when prompts may be giving away too much corporate context. 

These are indeed fascinating and high-stakes times for people working in enterprise IT infrastructure and operations. With so much in the business depending upon the cost-effective protection of data through its lifecycle, easy access to data for users and applications, and the ability to leverage data for new value with AI and analytics, it's imperative that IT leaders get more strategic with the data storage and unstructured data management tools and strategies they deploy.

Darren Cunningham is the Vice President of Marketing at Komprise.