Data Analytics and Data Management Predictions 2024 From Industry Insiders

As we look ahead at the biggest trends that will be driving the tech industry this year, AI is near if not at the top of the list (although not everyone believes generative AI will disrupt IT). Since data is the fuel that powers AI, it's small wonder that AI plays a role in many of the data analytics and data management predictions for 2024.

Below are the trends IT leaders and industry insiders are predicting will have a major impact in the data analytics and data management space in 2024. Check out their data predictions below, as well as the following 2024 predictions:

What the Tech Industry Expects From Data Analytics and Data Management in 2024

Data Is the Most Important Asset

In 2024, we'll continue to see a push into all aspects of data. Why? Because data (trusted, complete, secure, and timely) is the foundation that drives the generative AI and prescriptive analytics that accelerate growth and drive measurable, real business impact. — Eric Johnson, Chief Information Officer, PagerDuty

The Role of Chief Data Officer Will Become a Prerequisite for CIO Hopefuls

In 2024, there will be a new, surefire career path carved out for CIO hopefuls — becoming, and excelling as, a chief data officer. Over the last couple of years, the CDO has evolved from a low-budget advisory role to a critical asset helping businesses get the most out of their data. As more organizations invest in AI and the cloud to democratize their data and spur innovation, CDOs are in the driver's seat — and closer to the CIO, and the success of the business, than ever. Organizations looking for great CIOs will choose the ones who truly understand how data moves, flows through, and influences organizations, meaning that CDOs will have a natural advantage in pursuing that career path and continue to exert tremendous influence in the enterprise. — Heath Thompson, President & GM, Quest Software

Increased Adoption of GenAI Will Drive Need for Clean Data

The foundation of generative AI is data. That is, to function as desired, data is what provides the basis for this new technology. However, that data also needs to be clean. Regardless of where you're pulling the data from — whether you're using something like modeling or a warehouse of your choice — quality data will be essential. Bad data can lead to bad recommendations, inaccuracies, bias, etc. Having a strong data governance strategy will become more important as more organizations seek to leverage the power of generative AI in their organization. Ensuring your data stewards can access and control this data will also be key. — Rex Ahlstrom, CTO and VP of Innovation and Growth, Syniti

Open Formats Are Poised to Deal the Final Blow to the Data Warehouse Model

While many anticipate the data lakehouse model supplanting warehouses, the true disruptors are open formats and data stacks. They free companies from vendor lock-in, a constraint that affects both lakehouse and warehouse architectures. — Justin Borgman, co-founder and CEO, Starburst

Multi-modal Interfaces for Analytics — the Winning Formula for Enterprise Collaboration

Analytics is a team sport. And like every sport, we'll see organizations seek platforms that allow each player to harness their strengths by giving flexibility to various professionals in various roles across the business. This will give rise to multi-modal interfaces that empower every user in the analytics value chain to engage with data intuitively and successfully based on their role — all while increasing collaboration across the decision-making process. The result? A winning formula for every organization. — Suresh Vittal, Chief Product Officer, Alteryx

CIO Perspective of Top Data and Analytics Trends for 2024

Two Hot Topics, Data Products and Data Sharing, Will Converge in 2024

Data sharing was already on the rise as companies sought to uncover monetization opportunities, but a refined method to curate the shared experience was still missing. As the lasting legacy of data mesh hype, data products will emerge as that method. Incorporating GenAI features to streamline data product creation and enable seamless sharing of these products marks the pivotal trifecta moving data value realization forward. — Justin Borgman, co-founder and CEO, Starburst

Security Is About the Data

We'll continue to see a fast evolution in strategies to holistically secure data. The network is gone, the endpoint is remote and commoditized, but the data is where security risk will need to be managed across the all-cloud, all-remote world we now live in. — Eric Johnson, Chief Information Officer, PagerDuty

Overcoming Data Silo Challenges

Data silos remain a challenge for organizations — many analytics and AI systems spread across regions, clouds, and platforms, resulting in a vast amount of data duplication and separate governance models. In 2024, to accelerate time-to-insights and scale analytics and AI initiatives, organizations will increasingly need to manage distributed data. More will develop data strategies for unified management of scattered data through flexible orchestration, abstraction, and virtualization. — Haoyuan Li, Founder and CEO, Alluxio

Adapting to Regulatory Changes

The regulatory changes in privacy and consumer data will act as a forcing function for companies to deploy new technologies that allow more personalized marketing at scale. To start this process, companies will replace their current tech stack with new data aggregation systems that enable real, deep data for better personalization and efficiency — Phill Rosen, Global CTO, MoneyLion

A New Class of Aggregators

A wave of innovation will redefine data aggregation in fintech, ushering in a new class of aggregators that will surpass the current industry leaders. These emerging aggregators will revolutionize data updating processes by offering more comprehensive solutions, boasting better bank buy-in and outpacing their predecessors. — Phill Rosen, Global CTO, MoneyLion

Generative AI Turns Its Focus Toward Structured, Enterprise Data

Businesses will embrace the use of generative AI for extracting insights from structured numeric data, enhancing generative AI's conventional applications in producing original content from images, video, text, and audio. Generative AI will persist in automating data analysis, streamlining the rapid identification of patterns, anomalies, and trends, particularly in sensor and machine data use cases. This automation will bolster predictive analytics, enabling businesses to proactively respond to changing conditions, optimize operations, and improve customer experiences. — Nima Negahban, CEO and co-founder, Kinetica

Managing Telemetry Data in 2024

As organizations scale, telemetry data complexity, volume, and cost of data management will continue to increase. Observability, security, and business teams will demand access to a wider variety of data. More functional groups will consider incorporating AI/ML into their observability workflows, but data quality will constrain business outcomes. As organizations realize that it's not the ML models but underlying data that will separate them from their competitors, they will have to take a newer enterprise approach to manage their telemetry data. In 2024, we will see organizations taking steps to manage telemetry data as an enterprise asset and put mechanisms and governance in place, first to understand the data and then to provision the right data in suitable formats, with appropriate quality and context to observability and security teams. — Ajay Khanna, chief marketing officer, Mezmo

Ability to Stay Competitive and Innovate Will Come Down to Enterprise Data Strategy

Over the past year and a half, there's been a significant explosion of "ready for prime time" generative AI, opening opportunities for enterprises to benefit from intelligent automation. There's no denying that AI will continue to increase efficiency, accuracy, and overall business agility in 2024. With this, we'll start to see an increased need for a robust foundation of reliable and well-governed enterprise data. Utilizing the power of this data is paramount for training precise machine learning models, deriving insightful analytics, and enabling intelligent decision-making. As AI technologies continue to evolve, the quality and accessibility of enterprise data could significantly impact an organization's ability to assess large datasets in real time, stay competitive, eliminate bias, and free up more time for innovation. — Stephen Franchetti, CIO, Samsara

Sync or Sink: How Companies Are Using Data to Drive Success

The ability to connect past results and current realities with probability of success for future objectives through effective data utilization will emerge as a defining factor in 2024. This will separate the successful innovators from those struggling to keep pace with evolving market demands. Often, data resides not in one or two or three systems, but in hundreds, sometimes thousands, of different systems. Looking forward, we can anticipate a growing divide between organizations that harness the power of data for strategic alignment and predictive capabilities and those that neglect this critical aspect. Those adept at synchronizing data to align with their goals and OKRs will continue to thrive, leveraging their insights for rapid adaptation and maintaining a competitive advantage. In contrast, organizations failing to prioritize data-driven decision-making risk lagging behind, facing challenges in navigating the dynamic business landscape. — Razat Gaurav, CEO, Planview

Embrace Selective Replication for Global Data Harmony Despite Data Residency Challenges

In 2024, more organizations will realize the only sensible approach to data residency and data sovereignty issues is to use selective replication to keep local data local while sharing data globally as permitted. Organizations that have to deal with data residency requirements mandated by regulation, customers, or both have until recently had to address this problem by standing up entirely separate application stacks. With the selective replication capabilities of a distributed database — able to replicate just certain tables, or parts of tables — it is possible to avoid this and still have a global database that supports (for instance) global aggregation and reporting. — Phillip Merrick, co-founder and CEO, pgEdge

Big Data Insights Won't Be Just for Data Scientists Anymore

The ability to extract meaningful business insights from big data has largely been the domain of the highly specialized data scientist. But, as in cybersecurity, these experts are rather few and far between, and more and more teams are placing demands on this finite resource. In 2024, we'll see this change exponentially. Data fabric platforms and data science and machine language (DSML) platforms are changing the game, unifying and simplifying access to enterprise data. The more user-friendly interfaces of these platforms give more people on more teams the ability to see and act on threats or other challenges to the business. The democratization of data comes none too soon, as advancements in AI are making it easier for bad actors to infiltrate. With more eyes watching and able to take protective action, enterprises have a real shot at staying ahead of threats. — Nicole Bucala, VP & GM, Comcast Technology Solutions

AI Will Impact Data Observability Vendors

In 2024, large enterprises will rapidly increase investment in AI and large language model (LLM) technologies. This will create a greater need for data observability to validate that data feeding AI initiatives is accurate and complete. As a result, data observability vendors will be expected to expand support from predominantly cloud-native data environments to larger, more traditional enterprise data stacks and provide native solutions for LLM data pipeline monitoring and validation. — Kyle Kirwan, co-founder and CEO, Bigeye

The Shift to Data Fabric Will Accelerate Thanks to AI

When I surveyed the landscape at the end of 2022, I anticipated that more organizations would move from a data mesh approach to a data fabric to help break down information silos and make data available to business users more quickly. We haven't quite seen as fast a transition, but this trend is certainly accelerating and, in 2024, this trajectory will be driven largely by increased adoption of AI and other self-discovering technologies. While there's been a lot of discussion around data fabric in recent years, it will become a bigger goal for organizations thanks to the emergence of more advanced AI. — Rex Ahlstrom, CTO and VP of Innovation and Growth, Syniti

Data Quality Will Become an Executive-Level Topic

Ownership of data and data quality are core to business success but are still too often ignored or overlooked by the executive and board levels of most organizations. We can see this in the disconnects around perception versus reality. More than 80% of executives we surveyed think they trust their data, but the reality is that there are still a lot of people who are doing a lot of work to get data quality to a level where the data can be consumed and used. As data quality takes on greater importance, it will escalate to an executive-level conversation. — Rex Ahlstrom, CTO and VP of Innovation and Growth, Syniti 

The Rising Value of Network & Application Data

Companies increasingly realize that data is one of their most valuable assets. Of course, not all data is created equally or effectively utilized, and some data classes are overlooked entirely. For example, many organizations are not taking full advantage of data regarding the health of their networks and applications, even though it can help predict performance problems and cybersecurity attacks. More and more organizations are looking for solutions that share and integrate data across platforms for more real-time insights to address this challenge. We can expect this trend to continue in 2024, especially as organizations grow their data lakes by incorporating new sources like packet and application performance management (APM) data. — Paul Barrett, CTO, NETSCOUT

English Will Replace SQL as the Lingua-Franca of Business Analysts

We can anticipate significant mainstream adoption of language-to-SQL technology, following successful efforts to address its accuracy, performance, and security concerns. Moreover, LLMs for language-to-SQL will move in-database to protect sensitive data when utilizing these LLMs, addressing one of the primary concerns surrounding data privacy and security. The maturation of language-to-SQL technology will open doors to a broader audience, democratizing access to data and database management tools, and furthering the integration of natural language processing into everyday data-related tasks. — Nima Negahban, CEO and co-founder, Kinetica

Location-Enriched Time-Series Data Will Outpace Conventional Time-Series Data

Location-enriched time-series data will outpace conventional time-series data in automotive, telco, and logistics industries. The widespread deployment of GPS devices, location-aware chips, and advanced satellite constellations will fuel this shift. Organizations will begin to adapt their data architectures to harness the full potential of location-based sensor data, marking a pivotal transformation in these sectors. — Nima Negahban, CEO and co-founder, Kinetica

Vector Databases Will Become the Most Sought-After Technology

In 2024, vector databases will become the most sought-after technology to acquire. In an era where data-driven insights fuel innovation, vector databases have swiftly gained prominence due to their prowess in handling high-dimensional data and facilitating complex similarity searches. Whether for recommendation systems, image recognition, natural language processing, financial forecasting, or other AI-driven ventures, understanding the top vector databases will be critical for software development across industries. — Ratnesh Singh Parihar, Principal Architect, Talentica Software

Distributed Databases Will Reduce Complexity

We will see distributed databases simplify a lot of messy, complicated, and costly architectures that rely on middleware and spaghetti API connections to just move data around. Over the past two decades, middleware architectures employing enterprise service busses, message queuing, and web services have been used to move data bidirectionally between enterprise applications.  This has literally created mounds of expensive and hard to maintain "spaghetti code." With distributed databases that can safely and efficiently manage bidirectional replication, this complexity can be significantly reduced and be managed within the database schemas. — Phillip Merrick, co-founder and CEO, pgEdge

There Will Be a Spike in Interest in Vector Databases, but It Won't Last

Vector databases will be the hot new area for discussion by many but will eventually be absorbed by relational databases after a few years. Every 10 or so years, a "new" database technology is proclaimed to be the end of relational databases, and developers jump on that bandwagon only to rediscover that the relational model is extremely flexible and relational database vendors can easily adapt new technologies into their products. Look at PostgreSQL's pgVector as an example of how a relational database can process vector data today and why you will be able to ignore the hype around specialized vector databases. The community around pgVector and PostgreSQL was able to support this use case around vector data quickly — the project started in 2021, but it has developed quickly with all the interest in generative AI and vector data. For those thinking about this area and looking at implementing open source components in their projects, pgVector makes PostgreSQL an obvious choice. — Dave Stokes, Technology Evangelist, Percona

SQL Is Here to Stay

Structured Query Language, or SQL, is proclaimed too old-fashioned every few years, and in 2024 proposals to use LLM AI tools to generate database queries will get a lot of attention. But one of the reasons SQL is the only programming language from the 1970s that still gets used so widely is its power in querying data. You may not like the syntax. You may find its rules somewhat arbitrary. You may have gripes about learning such an old language. But for decades, SQL has proved itself again and again as the premier tool to manipulate data. It won't be going out of fashion any time soon. — Dave Stokes, Technology Evangelist, Percona

Quick and Easy Cloud Solutions and Cyber Breaches Will Cause Creeping Chaos

The demand for databases that are easy to spin up and use will only accelerate. Object relational mappers will be popular with developers who do not write SQL, and this will produce badly performing queries that will need to be rewritten by specialists who do write SQL. Cloud bills will consistently creep toward the stratosphere, spurring sales of antacids for managers and accountants. There will again be embarrassing data breaches, but the public is getting numb to the revelations that yet another set of their private data is in the wind. What will change in 2024? Developers will have to take more responsibility for solving these problems before they affect operations more generally. A well-managed database instance reduces costs and improves security, as well as makes developers more productive. Without these steps, developers will see more problems creep in and affect their ability to be productive. An ounce of prevention is worth a pound of cure, and with more focus on getting applications and data ready for business, any distractions around security and performance should be dealt with ahead of time. — Dave Stokes, Technology Evangelist, Percona

Cost Management Will Hit Cloud and Database Deployments

Organizations have always been concerned about managing the costs associated with databases. Cloud was heavily marketed as a solution to reduce IT expenses. However, as organizations started to migrate their applications to the cloud, they discovered that although it offered several benefits in comparison to on-premises systems, cost reduction wasn't always one of them. These days, with the economic downturn, teams are becoming more cost-conscious, and this is likely to have an impact on developers' plans. There's nothing like a lack of resources to force you to be more innovative. Demand for open source software will continue to go up as developers have to do more with less, while existing cloud environments will continue being audited to check on spending and what can be done to reduce costs. Approaches like "scaling through credit card" will get curtailed, and those existing deployments will be updated or changed to reduce their consumption levels. This work will then free up budget for other vital projects, so there will be impetus to carry it out fast. — Aleksandra Mitroshkina, Senior Manager, Product Marketing, Percona

Organizations Will Continue Looking for Public Cloud DBaaS Alternatives

What we hear from our users is they want public cloud DBaaS alternatives. There are multiple reasons for this — for example, they may want more independence from their vendor, they may want to optimize costs, or they may want to get more flexibility around their database configurations. Right now, the market provides a limited number of alternatives to those willing to make a change. Rather than looking at DBaaS from a specific provider, there is a gap in the market for an open source private database platform that gives organizations and IT teams greater control over data access, configuration flexibility, and costs associated with cloud-based databases. The growth of Kubernetes and Kubernetes operators has made it easier to implement this kind of approach, but there are still multiple gaps around this that make it harder to deploy and run in production. Closing those gaps and making fully open source DBaaS options available will be something that comes to fruition in 2024. — Aleksandra Mitroshkina, Senior Manager, Product Marketing, Percona

For more 2024 trends stories, check out the list below:

Do you agree or disagree with these predictions about data analytics and data management, or do you have some of your own that didn't make this list? Let us know in the Comments section below!