Artificial Intelligence & Data Analytics Management-QLab Digital Marketers
Data Has A Better Idea
Big data isn’t quite the term de rigueur that it was a few years ago, but that doesn’t mean it went anywhere. If anything, big data has just been getting bigger.
That once might have been considered a significant challenge. But now, it’s increasingly viewed as a desired state, specifically in organizations that are experimenting with and implementing machine learning and other AI disciplines.
“AI and ML are now giving us new opportunities to use the big data that we already had, as well as unleash a whole lot of new use cases with new data types.“We now have much more usable data in the form of pictures, video, and voice [for example]. In the past, we may have tried to minimize the amount of this type of data that we captured because we couldn’t do quite so much with it, yet [it] would incur great costs to store it.”
Could AI solve that problem? How AI fits with big data
“The more data we put through the machine learning models, the better they get. It’s a virtuous cycle.”
There’s a reciprocal relationship between big data and AI: The latter depends heavily on the former for success, while also helping organizations unlock the potential in their data stores in ways that were previously cumbersome or impossible.
“Today, we want as much [data] as we can get – not only to drive better insight into business problems we’re trying to solve, but because the more data we put through the machine learning models, the better they get. “It’s a virtuous cycle in that way.”
How AI uses big data
It’s not as if storage and other issues with big data and analytics have gone bye-bye. for one, notes that the pairing of big data and AI creates new needs (or underscores existing ones) around infrastructure, data preparation, and governance, for example. But in some cases, AI and ML technologies might be a key part of how organizations address those operational complexities.
About that “better insight” thing: How is AI – and ML as its most prominent discipline in the business world at the moment – helping IT leaders deliver that, whether now or in the future? Let us count some ways.
ways AI fuels better insights
1. AI is creating new methods for analyzing data
One of the fundamental business problems of big data could sometimes be summarized with a simple question: Now what? As in: We’ve got all this stuff (that’s the technical term for it) and plenty more of it coming – so what do we do with it? In the once-deafening buzz around big data, it wasn’t always easy to hear the answers to that question.
Moreover, answering that question – or deriving insights from your data – usually required a lot of manual effort. AI is creating new methods for doing so. In a sense, AI and ML are the new methods, broadly speaking.
“Historically, when it comes to analyzing data, engineers have had to use a query or SQL (a list of queries). But as the importance of data continues to grow, a multitude of ways to get insights have emerged. AI is the next step to query/SQL,What used to be statistical models now has converged with computer science and has become AI and machine learning.”
2. Data analytics is becoming less labor-intensive
As a result, managing and analyzing data depends less on time-consuming manual effort than in the past. People still play a vital role in data management and analytics, but processes that might have taken days or weeks (or longer) are picking up speed thanks to AI.
“AI and ML are tools that help a company analyze their data more quickly and efficiently than what could be done [solely] by employees,”
observed a trend to a two-tier strategy when it comes to big data, as organizations contend with the massive scope of the information they must manage if they’re going to get any value from it: The storage layer and an operational analytics layer that sits on top of it. News flash: the operational analytics layer is the one the CEO cares about, even if it can’t function without the storage layer. “For specific use cases, it revolutionizes the way you get rules, decisions, and predictions done.”
“That’s where insights are extracted out of data and data-driven decisions take place, “AI is enhancing this analytics world with totally new capabilities to take semi-automatic decisions based on training data. It’s not applicable for all questions you have for data, but for specific use cases, it revolutionizes the way you get rules, decisions, and predictions done without complex human know-how.”
(In an upcoming post, we’ll look at some use cases that illuminate how AI and big data combine forces, such as in predictive maintenance – essentially predicting when a machine might fail, for example – and other practical applications.)
In other words, insights and decisions can happen faster. Moreover, IT can apply similar principles – using AI technologies to reduce manual, labor-intensive burdens and increase speed – to the back-end stuff that, let’s face it, few outside of IT want to hear about.
“The real-time nature of data insights, coupled with the fact that it exists everywhere now – siloed across different racks, regions, and clouds – means that companies are having to evolve from the traditional methods of managing and analyzing [data],” Mih from Alluxio says. That’s where AI comes in. “Gone are the days of data engineers manually copying data around again and again, delivering datasets weeks after a data scientist requests it.”
3. Humans still matter plenty
Like others, Elif Tutuk, associate VP of Qlik Research, sees AI and ML as powerful levers when it comes to big data.
“AI and machine learning, among other emerging technologies, are critical to helping businesses have a more holistic view of all of that data, providing them with a way to make connections between key data sets,” Tutuk says. But, she adds, it’s not a matter of cutting out human intelligence and insight.
“Businesses need to combine the power of human intuition with machine intelligence to augment these technologies – or augmented intelligence. More specifically, an AI system needs to learn from data, as well as from humans, in order to be able to fulfill its function,” Tutuk says.
“Businesses that successfully combined the power of human and technology are able to expand who has access to key insights from analytics beyond data scientists and business analysts while saving time and reducing potential bias that may result from business users interpreting data. This results in more efficient business operations, quicker insights gleaned from data and ultimately increased enterprise productivity.”
4. AI/ML can be used to alleviate common data problems
Here’s something that hasn’t changed: The value of your data is inextricably linked to its quality. Poor quality means low (or no) value. This is something that so-called big data has in common with AI. “The ‘dirty’ secret of ML projects is that 80 percent of the time is spent cleansing and preparing the data.”
“Every conversation about machine learning always comes back to the quality of the company’s data. If the data is dirty, any insights derived from it cannot be trusted,” says Moshe Kranc, CTO at Ness Digital Engineering. “The ‘dirty’ secret of ML projects is that 80 percent of the time is spent cleansing and preparing the data.”
Everything old is new again, it would seem. But the solution to this problem (and potentially others like it) might already be staring you in the face.
“Fortunately, machine learning data can be cleansed using… machine learning!” Kranc says. “ML algorithms can detect outlier values and missing values, find duplicate records that describe the same entity with slightly different terminology, normalize data to a common terminology, etc.”
5. Analytics become more predictive and prescriptive
An ML algorithm can be taught to make a decision or take an action based on a forward-looking insight.
In the past, data analytics was more postmortem than anything else: “Here’s what happened.” Future predictions were still essentially historical analyses. AI and ML are helping open a new front: “Here’s what’s going to happen.” (Or at least “here’s what likely going to happen.”) Moreover, an ML algorithm can also be taught to make a decision or take an action based on that forward-looking insight.
“Today, AI is moving big data decisions to points further down the timeline, in more accurate ways, by using predictive analytics,” says Sean Werick, managing director of analytics at Sparkhound. “Traditionally, big data decisions were based on past and present data points, generally resulting in linear ROI. With AI, this has grown to epic and exponential proportions. Prescriptive analytics, leveraging AI, has the potential to provide company-wide, forward-looking strategic insights helping to advance the business.”
Werick notes that this is a “learn to crawl before you walk” progression. Using AI to make predictive or prescriptive business decisions based on inaccurate or inadequate data could have “catastrophic” outcomes, according to Werick. But this is the progression AI is enabling.
“The value to the business increases with each progression through the analytics maturity model: beginning with process and data mapping, to descriptive analytics, to predictive analytics, and finally, to prescriptive analytics,” Werick says.
6. What’s next for AI and big data? We’ve merely scratched the surface
If most teams are still learning to crawl (or walk), that might be OK because the combination of AI and big data is just beginning to reveal its possibilities.
Andy Vitus, a partner at Scale Venture Partners, sees a big future in more intelligent enterprise software, for example. Many business applications still show their analog DNA, in his view. “Users are still spending inordinate amounts of time slogging through endless reports.”
“Most business apps are still built using the design language of paper forms and ledgers. This means that for all of the data being captured and stored by enterprises, users are still spending inordinate amounts of time slogging through endless reports to find useful information,” Vitus says.
“The future is intelligent software that leverages all of that data to solve problems and do work for us – providing context and answers rather than just nicer-looking reports. From an engineering perspective, intelligent enterprise applications will require that we connect individual AI/ML systems to other systems so that they can communicate with and learn from each other. Enterprises will finally see significant ROI from all of that data they’ve been storing.”
That’s the essential promise: AI as an evolving means of answering that basic question about big data: Now what?
“This is just the beginning – in the future, there will be new techniques that emerge on how to analyze data for real-time insights,The data is still the data, but the ways of getting insights on it will improve.”
