Pandemic Exposes Data Deficiencies

Too much critical business information hasn’t been accessible during the pandemic

By Thomas Davenport

The highly successful investor Warren Buffet is fond of saying, “Only when the tide goes out do you discover who’s been swimming naked.” The COVID-19 pandemic is the most recent example of a major low tide event for unprepared organizations. We see its effects in tangled supply chains, confusing work arrangements, transportation systems in disarray, and now unresponsive information environments. This latter issue is an important one-so much so that I plan to devote several posts to it.

But a new survey sponsored by IBM suggests that there is a bigger issue unfolding:

(disclosure: IBM contracted with me to help design and analyze the survey). It may even be that the lack of good, timely information for decision-making is a major factor in all the other problems we’ve faced during the pandemic.

The global survey of 750 IT executives across ten countries had multiple questions about the impact of the pandemic on companies’ information environments. Fully 90% of respondents agreed that “the pandemic put pressure on our IT department to deliver information quickly about our fast-changing business environment.” Similarly, 92% agreed that “due to drastic changes in customer behavior or other factors during the pandemic, reliable real-time data became even more important for business decisions.” No surprise thus far, I suppose.

But many companies didn’t fare well under the pressure of the pandemic. More than a third (36%) agreed that, “during the pandemic, there was information that my IT department was unable to provide to business users for effective decision-making.” And when asked, “to what extent was there information that existed within your organization that was not usable by the business?” 84% answered either “to a great extent” or “to some extent.” This is big news;

More generally, 95% of respondents agreed that, “a business or societal crisis highlights the need for a flexible and responsive reporting and analytics environment.” Whether the crisis is a war, a recession or depression, or a pandemic, virtually every IT executive agreed that such environments make data and analytics more critical to a successful response. Turbulent environments require better information, and something is wrong if IT organizations can’t provide it.

The Underlying Problems

You are probably interested in why the information couldn’t be provided or was unusable. Here are the top 6 reasons provided, and the percentages of survey respondents who selected them:

These are surprising results. They suggest that companies for the most part had the necessary data, but many couldn’t make it available to the business users who needed it. Either it took too long, required too much integration or programming-either by internal or outsourced resources, or access wasn’t possible. Some firms had several of these problems.

This is not an academic issue; it has real implications. When asked what the implications of data not being available when necessary, only 3% said there were no consequences, and other respondents cited a large number of concerns — including impact on customers, products, services, and finance:

These seem to be dire consequences. And when asked to estimate the cost to their organizations of missed opportunities involving data, the estimates were--not surprisingly--quite high.

We Need to Do Better

It’s clear that the current situation is unsatisfactory and very costly, and that Improvement is needed. Many organizations obviously need to take a different approach to how they capture, store, and provision information. One major change, according to survey respondents, is that we need to place additional emphasis on providing analytics more quickly. Nearly all (95%) of respondents, for example, agreed that “making decisions based on accurate, timely analytics and insights has become as important as having accurate transactional data in my organization.”

However, in practice, transaction data often takes precedence. When asked, 87% of respondents agreed that “information from business transactions tends to be available more quickly than information for analytics and insights in my organization.”

It’s not surprising that transactional data is a high priority; in the past that was pretty much all we had, and analytics were largely based upon it. To some degree that is still the case. In the survey, 93% agreed that “information in operational, transactional systems is often also needed for analytics and insight-based decision-making in my organization.” But now, analytics and AI come from a variety of data sources, and they increasingly can provide a picture of what’s happening in the outside world as well as inside companies. That, of course, is particularly important during pandemics or other business crises.

The importance of real-time analytics for decision-making is strongly supported by survey participants. When asked, “how important to your organization is it that data used for analytic insights is available in real time or close to real time?” Virtually all (97%) agreed that it was either extremely or somewhat important. However, only 28% say they can deliver the real-time insights the business requires, though virtually all (99%) can deliver real-time insights for at least some business issues.

So let me review the situation and the survey results. The pandemic, like other business crises, put pressure on IT organizations to deliver business information quickly. Reliable real-time data became more critical. But IT organizations couldn’t deliver; many said that needed information wasn’t available during the pandemic. And it’s timely analytics and insights about the business that were sorely lacking — for a variety of reasons, but timeliness, integration, and the need for programming were critical. The IT executives interviewed say that real-time analytics for decision-making are particularly critical during business crises, but many can’t provide them in real time. Instead, their systems and reports are more focused on transaction data, which tends to be provided more quickly than analytics.

In my next post, I’ll try to shed some light-from the survey and my own experience of how we got into this situation. In addition to the historical factors I’ve mentioned, IT architectures, I believe, are part of the problem. So are different perspectives among IT executives. But more on all that later…

Related posts:

I wonder if there is some kind of psychological study exploring the effects of frequently saying goodbye in one’s life. Yes, much of life is filled with beginnings and endings of various types. But I…

In my Communications in Writing and Digital Media class that I am taking at Nazareth college, I have made several posts on Medium about different topics that we learned about throughout the course…

Unshielded electrets attract neutralizing ions from the air and lose their charge within hours or days. Fig 5 — Jefimenko’s Slot-Effect Electret Motor. Both inherent deficiencies of Gubkin’s motor…