EP 326: How Data Will Be AI’s Bottleneck

This is the Everyday AI show, the Everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business and everyday life.

Is data the next big bottleneck when it comes to generative AI? I mean, we've heard about a lot of these things that slow AI down when it comes to companies actually using it, right? First it was should we use generative AI and large language models? And then it was how do we use it? Well, now that most companies, especially here in the us, understand that they need to be using generative AI, is data that next big bottleneck? We're going to be talking about that today and answering those big questions and your questions. So thank you for tuning in. And this is Everyday AI. What's going on y'?

All?

My name is Jordan and I'm the host of Everyday AI and this is for you. This is your daily Livestream podcast, free daily newsletter helping us all learn and leverage generative AI to grow our companies and to grow our careers. So I'm excited today to talk about the elephant in the room and how data will be AI's next big bottleneck before we get started. As a reminder, if you're joining us on the podcast, thank you. As always, check out your show notes. You can always read our recap for today's show. We always share more insights as well as what's going on in the world of AI news before we get to the AI news. As a reminder, also in the newsletter, check out our Thanks A Million giveaway to celebrate everyday AI hitting a million downloads. You don't want to miss that giveaway. All right, let's get into the AI news for today. So OpenAI has launched advanced Voice Mode for a select few chat GPT plus users. So OpenAI has finally begun rolling out their Advanced Voice mode, initially available to only a small group of ChatGPT+ users. So this new feature, known as GPT4, it's under the GPT4O kind of umbrella. It delivers hyper realistic audio responses and is expected to be available to all plus users by fall 2024. When first demonstrated in May, GPT4O's voice named sky, caused quite a stir during due to its striking resemblance to actress Scarlett Johansson, who subsequently took legal action. OpenAI has since removed the sky voice and delayed the release to enhance safety features. Advanced Voice mode allows ChatGPT to talk and listen using a single multimodal model, reducing latency and improving conversation quality. This alpha version that's rolling out will not include though the video and screen sharing capabilities shown in the OpenAI's spring update, which will be released later. OpenAI's claims that this new advanced voice mode can detect emotional, you know, emotion in your voice, such as sadness or excitement. And it is wild, right? If you've seen the demos, it is, I think, a whole next step in looking at generative AI more as an assistant than just a large language model. Speaking of models, Meta has released a new multimodal AI that can segment video in its SAM2, so segment anything. Model 2, SAM2 for short, has been released by Meta, extending the capabilities of the original SAM to video. So the new SAM 2 can segment any object in an image or video and track it consistently in real time across all video frames. Pretty again, pretty wild breakthrough technology. So for that we'll have more in the newsletter. Speaking of Meta, also they revealed AI Studio for personalized chatbot creation. So Mata, the parent company of Facebook, Instagram and WhatsApp, has just released a new suite or a new tool called AI Studio aimed at enabling users to create and share personalized AI chatbots. So Meta introduced this tool that really just allows anyone you can go on their platform and we'll have the link, you know, in our newsletter drag and drop, you don't have to know how to code and also you can use these in Instagram. So Instagram creators can use these AI characters to handle common direct message questions and story replies, acting as an extension of themselves. And Obviously this new AI studio is powered by Meta's Llama 3.1, their latest in pretty pretty impressive open source model. So pretty big news there from Meta and Chat GPT. There's always going to be more, you know, mid journey 6.1 dropped AMDs, big stock jump. We'll have all that in the newsletter. All right, so enough about AI news. Let's go ahead and get to the big topic for today, which is how is data going to be slowing companies down and what we can ultimately do about it. So please help me welcome on the show today. I'm excited to have. There we go, Matt DeFries, who is the founder of Nuke Nuclei. Matt, thank you so much for joining the Everyday AI show.

Super excited to be here. Thank you so much for having me and I'm really looking forward to just have fun talking about data today.

All right. Yeah, same and hey for our livestream audience. Appreciate y' all tuning in from from Tara to Rolando and Fred and Douglas and Daniel and everyone in between. If you have a question about data and AI, please get it In. Now, before we dive into the topic, Matt, tell us a little bit about what you all do at Nuclei.

Yeah, so we are building infrastructure for everything around data. That means that we identify a couple of problems in many different industries where you want to innovate on data. Imagine you want to do something with AI, even if it's generative AI, you need data for that. And that data is locked in silos. It's in many different data sources. You have to write a lot of adapters. So we provide the layer to have generic access to all the data without having to deploy new infrastructure, having to deploy another ETL or whatever. And additionally, we make it very easy to add context to the data so that the data is understood at the same level across all data sources or data sets so that you can innovate very easily. On top of that, create many different kinds of data pipelines that can eventually lead to the creation of novel AI models or to train existing models, including LLMs.

And, you know, you kind of brought up a very good point there. And that's kind of the crux of this show is data is so important for large language models. Why do you think, Right, let's just kind of skip to the end. Why has data become such a big bottleneck for AI implementation? Because, you know, before this big generative AI wave, right, we all heard, oh, data is, is the next oil or, or whatever it is. So, you know, we've known about the importance of data for, for decades. So why is it still this. This big bottleneck for companies to implement generative AI around their data?

Yeah, I think, you know, first of all, I think we already passed the point of calling data is the new oil. I think really data is already our oxygen. Because frankly, with many of the things in our life, we can't live without data anymore. We can't function as a society normally anymore without data. Everything that we do using our phone, taking public transport, we use data. It's really the new oxygen. And I think that introduces a new problem because the data landscape is so extremely fragmented that there is not a lot of access to it. Like, it serves a purpose, and that's it. And that leads to a lot of silo data and that siloed data, that boring, structured data that's actually very necessary for the next generation of generative AI to hallucinate less. Because if you look at the generative AI landscape today, you know, it tries to come up with really proper answers based on a conversation that makes sense, right? Based on the next word, prediction or Word part prediction, but without taking the really factual data into account. And having more access to structured data sets will definitely help improve on that level.

And we talk about this a lot here on the show, Matt. But maybe for people who are tuning in for the first time or don't know the difference, can you explain simply what's the difference between structured data and unstructured data?

Absolutely. So structured data is best to be compared with what you have in Excel. So if you have a boring Excel where you have the first row, you have just column names and below that you have data numbers, yes and no states, you know, it's predictable, the next row is the same as the row before that it's structured. Unstructured data is typically what you would find on an Internet forum. A blog article. Like it's, it's text but it's not formatted in a specific structure. So you have to understand the context or you have to take the whole text as a text, which makes LLMs perfect conversationalists. But the lack of structured data make them very lacking at proper fact based conversation. So that's the difference.

Yeah, and yeah, thank you for that explanation. And I think that companies that you know have been using AI because you know, AI is not new, machine learning is not new. It's been used for many decades. But this concept of using unstructured data I would say is newer.

Right.

With generative AI and large language models, that's where they truly shine, being able to work with this unstructured data. With that in mind, a great question to start off the bat here from Douglas. So thanks for this. He's asking how do you recommend people who do not know where to start leveraging their data? That's great. Yeah. Companies, if they're maybe smaller and they haven't had a huge data team for a long time, where do they start?

I mean it's, I mean you have to start with a goal in mind. Right. If you want to leverage your data, where do you want to leverage for? So let's assume that in most cases the data will have to be used for large language model integration. I think there are a lot of use cases for support chatbots and stuff like that is where companies start nowadays. And having a predictable access to the data across multiple data sets of data sources is good start and adding context to that data, like going back to an example of the spreadsheet, like the Excel spreadsheet, where you have just a bunch of columns and rows. That first column name, that's your data point. The column name and that tells you something about that data, but not everything. And this is why we think that metadata is extremely important that further contextualizes the data. In fact, we did a proof concept with integrating an LLM on top of our data infrastructure so that you could prompt the LLM with a question and it would see what kind of data source or data set would it need to answer this question. Or a combination of different data sets that will give back that fact based answer. But it wouldn't do that properly without having that additional context on top of that data. So back to that Excel spreadsheet. You have the column names. Now imagine that you will add descriptions to that column names. You can explain what kind of limitations that data has. Like if it's just a column full of numbers, what do these numbers mean? Is there a limit? Is there a range? Are there exceptions? If you start deeply explaining that, that kind of context will lead to better answers. So it's having generic access to the different data sets that you already have and adding context to, to that data.

Yeah, that's a great point. Because all data, it does have a story, it has a meaning, it has a reason why it can help or hurt your business. So I love what you said there, Matt, about essentially start explaining your data, which oddly enough is something large language models can help with. Right. But even before we get into that, I want to hit reverse here and tackle this problem even a little bit more. Right. So I love what you said about data isn't oil, it is oxygen. Right. Businesses need data in order to breathe and survive. But why are companies not able to, you know, kind of grab their data out of silos? Right, because yeah, companies will, you know, we consult with companies all the time. They're like, oh, here's our salesforce data, or you know, here's data from this source, but they just live there. So why is that a problem? Data being in silos? And how can companies start to kind of fix that issue?

Yeah, actually, so companies can already collaborate and innovate on data. A lot of bigger companies, they already leverage the data and smaller companies do too. I don't think that there are a lot of technical limitations. I think the biggest limitation is cost. Like it's very expensive to do something with data, very expensive to collaborate with others, especially enterprises that have really complicated data infrastructure infrastructure in very big data ecosystems with a lot of rules and exceptions and standards and frameworks. And it's very hard to be interoperable with these standards as a smaller business. So that's where cost comes. And even for the bigger enterprises, like I've talked to a lot of enterprises about data consortiums where you have a group of enterprises or even with small medium business, even cities, governments that are inside of this data consortium then want to work together on a use case where platform creation on top of all the collective data data that they put in. The biggest problem why most of these data consortiums fail is funding. They stop being funded at some point because it's getting too expensive. And why is that? Because they have to bring in one of the biggest consultancy firms like Capgemini or Cognizant inside of this data consortium to build all the adapters and build other expensive data pipelines. So most of this money is going into the hours that are written by the consultancy firms and we're speaking about hundreds of millions before something tangibles coming from the ground. So we need something that is really low effort, a very low barrier for both enterprises, small medium businesses to start leveraging the data without having big investment. That also allows trying new things without investing a lot of things, without investing a lot of money so that you can get started and test out things early.

How should, you know, let's say someone here works at a, you know, medium sized company, you know, here in the U.S. so you know, they're, they're not working with a Capgemini and maybe, you know, they have a decent hold on their data, but it is still a little bit siloed. What questions should they be asking themselves or what steps should they be taking specifically when it comes to better preparing and better organizing their data for large language models?

Yeah, I suggest to find a way to first have the same type of data across all the data sources and data sets. What I mean is that if you have CSV files there, have a MongoDB there and you have a MySQL DB there, you all have to access that data in a different way. First try to get a layer and this is what we from nuclei specializing focus on is have this layer where all this data is onboarded very easily and you can just have one SQL query to access a CSV file, join it with data from a MongoDB without discriminating what that context of the data engine is. That way you have, if that first step is done and it doesn't cost you a lot, you're already halfway for the innovation because you can tap into all that data and start playing around with it. A good data scientist is crucial to have on board though.

Are you still running in circles trying to figure out how to actually grow.

Your business with AI.

Maybe your company has been tinkering with large language models for a year or more, but can't really get traction to find ROI on Gen AI. Hey, this is Jordan Wilson, host of this very podcast. Companies like Adobe, Microsoft and Nvidia have partnered with us because they trust our expertise in educating the masses around generative AI to get ahead. And some of the most innovative companies in the country hire us to help with their AI strategy and to train hundreds of their employees on how to use Gen AI. So whether you're looking for ChatGPT training for thousands of or just need help building your front end AI strategy, you can partner with us too.

Just like some of the biggest companies in the world do.

Go to your everydayai.com partner to get in contact with our team or you can just click on the partner section of our website. We'll help you stop running in those AI circles and help get your team ahead and build a straight path to ROI on Genai.

You know, a good question here from, from Tara. This one's a little specific, Matt, but I think it begs a good question. So she's asking what strategies can be employed to improve memory efficiency when working with data sets containing millions of rows? Yeah, that's a great question. Especially when these data sets become so big. What steps can companies be taking, if any, to kind of counteract this issue?

So to be honest, that's not really my specialty, so I don't know if I will be answering this question properly. But there is a lot of free or very cheap technology out there that can work with this data very efficiently. I don't think that data sets containing millions of rows should be of any limitation anymore.

That's a good point. All right, let's get to something a little bit more of your specialty, which I want to get a little bit more into the data that makes up models themselves. Right. You know, if you follow large language models or read our newsletter, you know, you saw that there's been this kind of divide recently about the quality of the data in the models themselves. Right. Because when companies are trying to really implement, you know, generative AI across different sectors of their organization, you know, they're not only trying to bring, you know, maybe fine tune or bring in their own data with rag, but ultimately they are relying heavily on the actual data set of these large language models. But we've heard recently with synthetic data, is it, you know, we thought it was bad, but you know, Meta and others have come out and said no, you know, when you use synthetic data, it's actually can help your models improve better. What's, what's your thoughts on this, Matt? And what should business owners and decision makers be aware of when it comes to synthetic data in large language models?

So actually, in fact I've always been a believer in the value of synthetic data. But I also have to honestly admit that it kind of shocked me when I read last week that the quality of the LLM actually improved by bringing in more synthetic data than without. It was actually, for me it was counterintuitive. I, I would not, you know, guess this. So for me, you know, it, the question immediately came like this is largely about unstructured data and I'm very curious because we actually work together with a partner that converts existing structured data into synthetic structured data sets in order to scramble the data so that you cannot, you know, get personal identifiable information out there and the statistical qualities remain the same in that synthetic data. So that was always the goal, that was a very clear use case for synthetic data. But now I'm very curious and we haven't, we haven't got there with testing it yet, but now I'm very curious if that's structured synthetic data, bringing more of that in into AI model training, if that will also yield the same results.

Yeah, same, right. I've always had this thought in my head, not just with synthetic data and you know, if you are new here, that's just essentially, you know, artificially generated data that mimics real world data that is then used in models, right? So you can make models a little bit, you know, technically cheaper, bring down inference costs, etc. But I've always said it's also going to be a problem, right. If so much of the data and the content in these large language models is ultimately just kind of regurgitated, right? Studies say that more than 90% of new information posted online by 2026 is going to be coming from large language models. Even on, on that end, right? Just the, the data quality of what goes in outside of, you know, synthetic data. Matt, is, is, is there problems there, is there problems with, you know, hey, yeah, all these large language models scrape the Internet and is the Internet just getting a little worse because people are over reliant on large language models.

So my, my expectation would be that because that's also a kind of a type of synthetic data or this generative AI created data, right? So having more and more of this data might lead in the short term to better models at first, but we will hit this, hit the Ceiling for sure and by then more high quality data. And I think that's also immediately the elephant in the room that we want to discuss today. Like the bottleneck of next generation AI is going to be around data because it's going to be more difficult to obtain high quality and true data that has been generated by human. Right? And it's going to be, and I'm also going to be very philosophical in, in a moment because we, when, when we have all of these structured, sorry, unstructured data out there that's human written is it starts to become more valuable. And a problem that we've seen so far is that the, the data that has been scraped hasn't been paid for. So you saw New York Times suing OpenAI for using the data without paying for that. So it's going to be increasingly difficult for creating budget, next generation budget generative AI models that, that are going to be able to go beyond that ceiling that, that I just introduced. So it's going to be a problem that, I mean, I find it a problem that only the big companies will be able to buy their way, way out of there. So you will have the large organization like meta, like OpenAI that will have the funding in order to pay for that data, but the smaller ones, the open initiatives, community driven initiatives will hit that ceiling and will be very hard for them to pass. And now for a little bit of philosophy. You could argue, and this is about synthetic data again, you could argue that everything that we have ever come up with as humans and that we will ever come up with as humans is in one way or another already out there. But it just had because a lot of innovation is just bringing existing things together and to come up with something new. So in that sense it makes sense to say like, it just, it doesn't matter if there is too many LLM generated data or too many synthetic data out there. It's just how you're gonna mix that together in order to find new creative ideas, new innovations. So this is a little bit of philosophy for me.

Hey, let's, let's go down this, this, this philosophy road. I like that. You know, everything that you can come up with as humans has already been out there. You know, that's, that's a good point. But I think maybe, you know, especially if we're talking about smaller, smaller companies, smaller to medium sized companies, when they look at data, right, they think data science, they think business intelligence. And you know, oftentimes, you know, smaller companies don't have big resources to devote to that piece, right? When they think of data, they think of big data and they say, oh, you know, that's not necessarily for us. So maybe however we use large language models is going to not be as impactful as those companies that have their own, you know, first party data. How could even smaller or medium sized companies still kind of, you know, break down this elephant in the room and break through this bottleneck and still create valuable, you know, first party or first company data? How can they do that if, you know, going back to your philosophy question, there is essentially saying, hey, all this data that big companies have can kind of already exist in a way, shape or form.

Exactly, exactly. And I think the crucial key component here is collaboration. I don't think, because I really believe in that ceiling and that the smaller initiatives will hit that ceiling much quicker than the companies like Meta and OpenAI and stuff like that, but through collaboration it will be easier to break through that ceiling. And also from nuclear, we chose collaboration over competition. We have around 40, 45 partners that we collaborate with, each playing a unique role in what we try to solve and what we offer.

You know, I have a hot take, Matt, and you know, normally I don't do this, but you know, we're talking data and we're talking about how important it is for, you know, companies and for large language models. One of my hot takes is this, is that we're going to start looking for data in places that we generally wouldn't look for, especially when it comes for our company or, you know, large language models. That could be, I think, large scale university studies. It could be, I think, talking to employees, right, and capturing their knowledge and turning that into unique first party or first company data. Is that a crazy thought to say that that could be a, not a future, but, or the future of data when it comes to large language models and generative AI. But is that a path forward that's worth exploring, just bringing in this more human, human level data?

Actually, I would argue that this isn't hot take, but it's just simply common sense, man. I mean, look, I think crowdsourcing data, which is kind of, you know, really broadly describes what, what you describe when you ask something from your employee. It's like kind of crowdsourcing data. And I think crowdsourcing data is going to be much more important in, in the, in the short term future and it's going to be a much bigger thing, whether it's about polling or stuff like that. And when it comes to trying to find data out there, that's not looking into places that you normally would. I think there are some cool examples out there where, you know, once I talked to organization, they had a factory where they were baking bread and there was data generated by the machines and that data was used in order to monitor machines and see when they had to have maintenance. And then some software company was hired that was really specialized in innovation, typically innovation on data. And they said, let's look at that data and see if we can find patterns that would indicate a better quality of bread that is being baked. So then they use that data for the purpose of improving the quality of the food. Food instead of the initial purpose which is always served. And that the data was like thrown away. And that's really cool. I mean that's how you look at data where you normally wouldn't look. And I think that's kind of multiple purpose data. It's out there everywhere already. And we will start scrambling more and more to get access to that data. And the funny thing is is that it's not necessarily super valuable data. It's not like you will have to, you won't have to sell it for big bucks. So access to it should hopefully be very easily.

So, Matt, we've covered a lot in today's episode. We've talked about how data is not oil, it's oxygen. How companies should be giving a story to their data pros and cons of synthetic data and even you know, this concept of crowdsourcing data. So you know, as we wrap up today's show, how what's kind of your one biggest takeaway for business leaders, decision makers out there in order for data to not become a bottleneck in their AI strategy.

Again, like don't be scared to put your data out there and focus on getting this data as early on as possible in a generic way and immediately put in context to that. If you're a small business, you're just starting to work with data. Start doing this already, like today. Even if you don't have an AI or data strategy yet, like start doing this and collaboration, collaboration is key.

I love it. So, so many great insights in there. And yeah, like I love this. Data is not oil, it is oxygen. So if you want your business to grow, you got to know your data and you also have to keep tuning in to great episodes like this. So thank you so much, Matt for joining the everyday AI show. We appreciate your time.

The pleasure is all mine. Was super cool, man. Thank you so much. And everyone who's watching right now have an amazing rest of the day.

All right. And hey, as a reminder, a lot of great information in there. We're going to be recapping it all in our newsletter. So make sure you go to your everyday AI.com sign up for that free daily newsletter. And also while you're there, make sure, if you haven't already, join our Thanks a Million giveaway. Appreciate y' all tuning in. So we'll see you back tomorrow and every day for more Everyday AI. Thanks, y'.

All.

And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going for a little more AI magic. Visit your everydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.