Unlocking Enterprise Data with Knowledge Graphs and AI, with Juan Sequeda, Principal Scientist and Head of AI Lab at data.world

This is the essence of what my organization is. It's the brain, right? Everything here is accurate. I can use this to explain things. The LLMs, these foundational models, don't have that accuracy, don't have that explainability, don't know my organization. Do we expect these foundational models to know every single organization? No, because these things is private. I don't want them to know, but I want to go use them. So that's why this combination of these foundational models, large language models, with your internal brain of your organization, which is your knowledge graph. That's what I see where there's a future. And I think what we need to work on is understanding best that integration point between the knowledge graph, the brain of your organization, and these foundational models.

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to The Cognitive Revolution. Today we're talking data and how generative AI interacts with enterprise data with my guest, Juan Sequeda, principal scientist and head of the AI lab at Data.World. Data.World, like many established companies that we've featured on the show over time, built much of its platform, products, team, and business before the current generative AI moment, and is now working to make sure that it's taking full advantage of this new technology paradigm for its customers. This is part of an economy-wide trend. Frontier generative AI models are now making their way into all sorts of high-value, often very challenging contexts. From process chemistry labs to novel protein design to US federal courtrooms to the practice of medical diagnosis to enterprise data lakes, GPT-4 demonstrated enough raw power that we now have experts in literally every field working day in and day out to figure out how to make generative AI work for their organizations. Now, it's not all instant success. From Juan's work, for example, it's clear that early benchmarks understate the complexity of real-world enterprise data. And that naive chat with your data type implementations are not up to enterprise challenges. Even the more advanced work that Juan and team are doing with knowledge graphs, while it does deliver major improvement, is still at best a partial solution. So while Data.World, again, like most established technology businesses I've talked to, is not particularly worried about competition from fast-moving AI-first startups, they do see the transformative potential and they are realizing enough practical utility today that they are rolling up their sleeves and settling into the process of AI implementation and optimisation. Interestingly, zooming out, I think this creates the potential for another major phase change in the history of AI. While GPT-4 takes meaningful effort to implement and often still falls short of our dreams, we are nevertheless building foundational capacity for both last-mile distribution and customisation. such that the next big model release will have the opportunity to almost immediately plug into many millions of live business processes and systems. The electrification of America took 60 years and included significant public works projects, and all the appliances were designed with a clear understanding of exactly how they'd be supplied with electrical power. Today, in contrast, we live in an internet-mediated, software-enabled world in which updates can quickly be pushed to everyone everywhere all at once. Today's software application developers are building somewhat ahead of AI capability, both so that they can deliver frontier features to their users today, and more importantly, so that they're ready to flip the switch when the next big advance comes online. How many months we'll need to wait before GPT-4.5 or GPT-5, I don't know. And it might be just long enough that folks start to wonder whether AI generally is underpowered and overhyped. But my expectation remains very clearly that we will continue to see additional jumps in capability and that with each future leap, considering the foundation now being laid, the deployment cycles will naturally get shorter and more disruptive. One note for listeners, enterprise data is a complicated space and we spend some time in the first half of this conversation discussing the general state of enterprise data and data science teams today. If you're already well-versed in enterprise data science, you might find some of this a bit basic and introductory. But for most, I think the additional context will be very useful. If you have any suggestions for how I can better handle the introduction of such advanced topics where listeners inevitably will have very different levels of prior knowledge, and I do expect this to be more and more common throughout 2024, I would love to hear your suggestions. As always, you can email us at tcr@turpentine.co or you can DM me on the social media platform of your choice. And of course, we always appreciate it when you share the show with friends, starting in this case with the data scientists in your life. Now, I hope you enjoy this conversation about generative AI and enterprise data with Juan Sequeda, principal scientist and head of the AI lab at data.world.

Juan Sequeda, principal scientist and head of the AI lab at data.world. Welcome to the Cognitive Revolution.

Well, thank you so much for having me. Really excited about having this conversation.

Yeah, me too. I have been interested in knowledge graphs recently. I've been kind of feeling from, for a few different projects that I'm working on that more structured data, something between your kind of classic SQL database and the, you know, totally amorphous data that people are just increasingly vectorizing and, you know, hoping for the best from. is probably going to be important. And, you know, I went on a little quest to see like, who's working on this and what's the state of the art and came across a paper that you had published. So I'm excited to get into that with you, but I thought maybe we should just start with kind of an introduction to you and the company, because I don't know how many listeners will be familiar. You want to just kind of give us a quick intro to Data.World? Like, you know, how'd the company get started? Was it an AI company at the beginning? We can take it from there.

Yeah, happy to. So, okay, so Data.World, we're an enterprise data catalog platform. So data catalog platform is essentially, we're an inventory for all the data and the metadata within your organization. So basically like your library card catalog, right? That's kind of the high-level thought about it. Managing metadata has been something kind of since the beginning of just data management, but I would argue that it hasn't really been of focus till the last probably 5, 6 years, which became kind of the focus in the enterprise. If I zoom out and we look at what data, the data management world has been, it's always about moving data. So ETL and ELT, storage and compute of data. So your data warehouses, your data lakes and all that stuff. And then you're using your data. You are now trying to, your analytics, your dashboards, your AML and all that stuff, right? And then there's always been this thing on top that we haven't really focused on it, which is the metadata, the data about the data. And that is what data catalogs are focused on, on being able to understand what's going across all the stuff that's being moved, all the different data sources, tables, and columns that you have, keeping track of all your dashboards and how they're connected across all the different sources, and then also keeping track of your business terminology, your business glossaries, and everything. So that's That's what a data catalog is. And so Data.World started off in, I think, 2016, so very early on. And the focus was to be— the first phase of the company was to be a GitHub for data. So it is, we wanted to be a catalog for open datasets. And that was the first phase of the company for a couple of years. And we continue to be the world's largest open data catalog. We have over 2 million users. I think like 2/3 of Fortune 500 folks are on Data.World, the open data platform. There's half a million datasets. Like during COVID all the COVID data was housed on Data.World. It was completely open. And it is continuing to be open. And we're actually a public benefit corporation. So that means that in addition to being a C-Corp and maximizing shareholder value, we have a public benefit mission, which we're evaluated on. It's to be the world's largest and abundant data resource, advocate for open data and linked data standards, and to be an archive for the world's data. So those are our public benefit mission. And what we did is that also during that first phase of the company, the goal was from a technology perspective is create a platform that can scale at a web scale. And that's what we have over 2 million users. And so we really built a platform that we know that you can have high scale and and then be able to go share data. So around 2019, the next phase of the company was, hey, we built this whole platform that the open data community is using it within organizations. People want to share their data and find data and so forth. So that's how we entered into the whole, what's now being called a data catalog. People are doing data marketplaces and stuff like that's all where that fits in. So that's kind of the phase of the company that we're doing right now. Now, from a technology perspective, from day zero, basically, the entire platform has been architected on top of a knowledge graph architecture, which means that we use, first of all, we're all about open standards. I mean, part of our public benefit corporation. So we use the open web standards of RDF, which is the metadata, the graph standard, OWL, which is for ontologies and schemas, SPARQL, which is for the graph query languages. And that's the architecture we use. So basically everything that we bring into Data.World it's already turned into a graph. And why a graph? Remember what I was saying, you start thinking about, I'm moving data, I'm storing data, I'm using data. Like, there's all these pieces get connected across kind of all those pipelines. So you just really want to keep track of things and how they're being connected and so forth. And that's a graph. I always say that your first application over a knowledge graph is really the management of all your metadata to understand basically what is that infrastructure that I have within my organization? to understand, hey, there is this data that comes from this source and then it gets moved to this thing, it moves to this data warehouse, there's this application, there's people who are using it and so forth. So you're keeping track of all of that. And then now with AI, what happens is that, well, you need data, right? Data is the foundation of this. So I would argue that we've always been a data company and the data is a bloodline for AI. So the data catalog and being able to manage your metadata, manage your semantics, manage your context, That is critical for the generative AI because that doesn't have the semantics, the context, the knowledge of your organization. Has it out of LLMs, whatever it's been trained on, but doesn't have your own internal context. So we've always kind of seen this over the last year and a half when it comes out, LLMs coming out, for this to be applicable in the organization at a scale where you need accuracy, explainability, knowledge graphs are critical for that because it provides all that context that the LLMs don't have. And that's what we're seeing and kind of what we've been talking about it. And the paper that you mentioned originally was kind of just the evidence that we wanted to put together to say, yes, knowledge graphs are critical for this, assuming you want accurate questions, accurate answers for your questions.

Let me just ask you to get a little bit more concrete on a couple of things. The 2 million datasets is awesome. The public benefit aspect of that is really cool. I'd love to hear a little bit more about kind of those datasets. Like, what are the big attractors? What are people coming to you for? I imagine that, again, most people probably just don't know. And then I'm also really curious to get a little bit more concrete on Okay. But like, what is a knowledge graph? Like, how have they— how do they typically get created? And like, what role does Data.World play in helping enterprise customers do that work?

So one thing is that is the open data catalog, which is open for folks, right? And then people want the data catalog architecture for there to create their catalog internally. So I see there's usually kind of 3 main, let's call it applications that people that our customers will use Data Vault for. One is when it comes to like search and discovery. I don't know what data I have, right? It's a typical problem. Like you have data scientists, we're looking for data and they don't know where to go find stuff. Your data lake has turned into a data swamp type of approach. So they need to have a way to go search and go find that data that they have. So that's usually one of the applications around that. So search is something that we're very focused on. And the same way you search things on Google today, and if you search on Google, I don't know, you search for Austin, you get that panel on the side. Oh, Austin is a capital. of Texas and here's all these events going on and here's the weather and here, like that is what Google calls a knowledge panel. And that is results that's coming from their graph that they've built. So we do the exact same thing. So if you think about kind of the knowledge graph inside of Data Roll that we build automatically when we bring your data, think about it from the concepts that show up are, hey, there's a database. A database has tables, right? Tables have columns. There are dashboards, right? There are different types of dashboards. Tableau dashboards, Power BI dashboards. This dashboard was built using data from this table or this query over these tables, right? So you start to see how things get related, connected around that. There's this person who created this dashboard, right? Then there's policies that we are kind of defining. There's PII, there's personal information. This table has PII data, right? So then all these things get connected. So that's kind of the graph, what happens underneath. And when we go off and kind of connect and extract the metadata from all these sources, we build that graph automatically for you for all that technical metadata we're seeing. So that's kind of that, call it the digital landscape of all your data that's connected within your organization. So search is one of those applications over that big graph of your metadata. Another one is around governance. I want to know, again, let's keep track of all the policies around our data and who can use what data for what reasons, who can access this data, right? But where is all the data that has sensitive data and so forth? So that's the governance side of things. And then the third one, we see it a lot, is for data ops, just operations of data, making sure that, hey, data is getting moved from different places. Again, keeping them moving is like what we call the lineage. Oh, well, this dashboard extracts data from this table, this table comes from this stuff and so forth. So you have kind of that whole graph and lineage of where data's coming from, how it's living. And then you wanna make sure that, hey, if I'm gonna make a change somewhere, what could that affect? Or if somebody's saying I have an issue in this particular feed of data, where could it be? And I can kind of follow that. So that's all the operations of data. And then you can be in your dashboard in Tableau and then you can actually get notifications from there saying, hey, there was an issue just reported. So just be very careful with the data that's being shown here and it's gonna be fixed and so forth.

So that's the kind of a third application.

So So to summarize data catalogs and our users, they use it kind of for 3 main things, which is for search and discovery, for data governance, and for data ops. And at the end of the day, all of this is connected and we build that graph of that metadata all automatically once we kind of have our collectors up and going.

Hey, we'll continue our interview in a moment after a word from our sponsors. I'm really struck by how many of those things seem like They may already be undergoing significant transformation in the AI era. 2016, obviously not pre-AI, but pre-transformer, certainly pre-large language models, certainly pre-any kind of general conversational interface with a database. I guess I'd love to hear a little bit more about how you build a knowledge graph automatically. in a pre-AI era and how that might be evolving. And then also kind of the kind and the way the data is stored. And I had Anton, who's the CTO at Chroma, one of the high-flying new vector databases, on as a guest on the show not too long ago. And he made this really interesting point that the vast majority of data that goes into their databases has never been in a database Before, when you were first building knowledge graphs automatically, how did you approach that? How is that changing? And how is the actual nature of the data beginning to change in light of what AI can now do with it?

First of all, we need to think about when it comes to knowledge graphs, there is the notion of the schema, of the ontology. Let's go define the semantics, the meaning of a particular domain. So everything I've been talking up to now, when it comes to data cataloging, the domain there is like this technical metadata. Just think about the map, the— let's draw the bubbles and lines on the whiteboard. You have a table as part of a database. You have a dashboard. A dashboard takes data from a table, right? So that's kind of the schema that we're creating. That's one knowledge graph. We create knowledge graphs about any type of topic. So for example, let's create a knowledge graph around e-commerce. Well, you have customers, you have orders, right? A customer places an order. An order consists of a set of order lines. An order line can have products, right? And the orders are shipped to an address. So you could have a shipping address. You can have a mailing address. So that's basically the schema you start kind of defining. You populate that coming, that data may be coming from so many different types of sources. They can be coming from structured sources, relational databases. Some of that can be coming from unstructured, from text and so forth. They can come in from feeds that come on JSON feeds or right from APIs or whatever. So the knowledge graphs are a means of integrating data coming from so many different types of sources, and you can model anything that you want around that. Now, what takes time always is what I call the knowledge engineering process. It's understanding what do you mean by what is a customer, right? Understanding what you mean by what is an order or what is an active order or so forth, right? And I think this is the human nature of the problem because you go ask multiple people and they'll probably have different answers around that. So what is the correct version? What is the correct answer? So I've always argued that If not even humans will agree, how will the machines even know, right? The machines will definitely then be able to generate suggestions. Oh, it could be this, it could be this, could be this. But at the end, the humans need to go say, no, our decision is going to be this definition. And that's, by the way, where governance comes in because you want to have an agreement on what things mean because otherwise then, I mean, we can, depending on the type of business you are, you can have different types of chaos in a way. How things have been kind of slow to create what I'm calling creating these ontologies, be able to understand what this data actually means. Now, what's interesting with LLMs today, like, they're helping. A lot of the experiments we're doing is using LLMs to help us accelerate that process of like, hey, create candidate ways of modeling these things, right? And how about through a chatbot, go talk to the end users to be able to extract the stuff that's in their head. to say, how do you define what a customer is? How do you define what an order is? So I think that's something that we're seeing right now for that. So then when it comes to like the vector databases, I think one thing that we're seeing today is the low-hanging fruit of being able to kind of show some really cool AI applications, specifically when it comes to like chatting, chatting with your data, it's usually about unstructured. So it's the text, as you said, mentioned, like the stuff that has never been inside of a database. So it's your documents, your PDF files, right? And in other places you have images and so forth. And that's where you want to have— that's where vector databases come in. And you have to do all of this, the chunking and kind of all the bits and pieces for the unstructured side. When it comes to structured data, the data that's in your relational databases and your data lakes that come into— that have thousands of tables, right? And tens of thousands of columns. Like that structure has meaning and it's, and that meaning already, a user, a human being will look into that table and they're like, oh, this is what this means. Like, you want to add these 2 columns together, but not these other ones, right? There's knowledge behind that. And I think that's harder to go do based on all the, what I was talking before. It's like, but what does this mean? You got to go talk to so many people. And that's like not a scalable thing to go do. Again, LLMs today are helping us. They're gonna help us because now we can kind of have these chatbots that can help us acquire knowledge. So the low-hanging fruit has been focusing on the unstructured. But I would argue that the untapped potential here is to be able to go focus on the structured data. Because this is the data that goes into your reports, into your dashboard. Like, that's the stuff that executives and the boards are looking at those graphs to make decisions around that. And that's the data that's coming from your CRMs or ERP systems. Like, this is all very structured data. And those types of questions that people are asking, they expect accurate answers and they expect you to explain where those answers come from because they go trust that. Well, if I have— I'm asking questions over text, I'm like, oh, I mean, you have a little bit more freedom over there, right? And then I can point you to this larger document where a human would go in. So I think that's why I'm just like arguing like the low-hanging fruit is focusing on unstructured and there's just so much of it. So we can be very, very busy for a long time and provide a lot of value. But I think where there is tremendous value, one could argue, and I would argue here that probably even a much more higher value is if I can have a chat with my structured SQL data because that's where your critical decision makings are happening for What you're describing there, I think, connects to a couple of big themes that I notice across a lot of different areas of AI today.

One is just how much knowledge is kind of undocumented. A friend of mine used to call this tribal knowledge. It's the know-how that people share by watching each other do tasks over their shoulder, perhaps, or on a screen share, perhaps more likely today, or in kind of a quick chat where people ask each other questions. And that knowledge is, because it's kind of not localized anywhere, it is very hard to make that available to anyone. But we're starting to really feel that acutely now in the AI moment where People are like, well, why aren't these agents working? And often the answer is because they're kind of very generic and they don't have any of the context or the sort of— what feels familiar to you is just totally alien to them. So it's interesting that this problem, I guess, has been felt acutely enough in the data world even before AI that it's like you guys have built a whole platform about it. Could you kind of sketch a little bit more there? Like typical enterprise customer, I have a little bit of a sense for this. I did a project once. I was a very junior consultant, a firm that was working with Washington Mutual, which was once the country's biggest thrift, thrift it was called. And yeah, it was in that mortgage crisis moment that I was doing this project. And I do remember just the absolute total gnarliness of the data and just how many different columns. And there are all these columns derived from other columns, and it was kind of the data swamp situation. And I only saw a corner of it in my brief engagement before the bank went or the thrift went bust. But maybe you could give us a little bit more kind of color on Typical enterprise, where are they today? They have obviously a lot of data, but are we still in a moment where it's spread across tons of systems and people are super siloed and nobody knows where things are? In my case, I remember vividly completing an analysis and then having somebody look at it and be like, this is totally wrong. And me being like, I would've had no way of knowing this. I literally could not. All these suffixes on these columns, they didn't mean anything to me.

This is exactly, yes. And here's the thing is that the problems that we deal with today are the same problems that people have been describing 10 years ago, 20 years ago, 30 years ago, even more since the beginning of just like modern digital, just digital enterprise since going into warehouses and so forth back in the '90s or even before that. So this is my annoyance with the entire world is like, Can you imagine that the problems that we're complaining about today are literally the same problems people have been complaining about 30 years ago? Like, we're kind of fucked up if we're still trying to solve the same problem. Now, things are getting easier, right? Like, now we, like, the cloud has made things a little bit easier to go do, right? We have self-service, a lot of people to go do more things, but essentially the same problem, the problems are the same. So, let's ask ourselves, like, what have we been doing for the last 30-plus years? And then I'm like, wait, but now AI is coming. So, is this magically to solve this. Like, no, we gotta— this is where it's so important to kind of understand our history. So let me go down a couple examples that kind of I've seen across my careers, right? So at the end of the day, with data, your goal is to go answer some questions, and you want to be able to answer some questions to be able to understand how the business is doing, to make sure that I'm going to make a decision that it's going to help us make more money or save money, right? I think you make money, you save money, you want to mitigate risk. The 3 main things you want to be able to go to an organization, all right? So, I have a question I want to go ask. So, I got asked a question and I need data to answer this question. Where's that data? So, in one, like, call it the spreadsheet approach, you basically go ask somebody who has the data, I need data about X, Y, and Z. And that person will go off and send you a spreadsheet of X, Y, and Z, and then you get it. And then you know Excel pretty well and you'll need lookups or— Heck, maybe people have an Access database on their laptop and then they'll be able to go do munging or whatever that stuff. And then they answer their question. There it is. Okay. So that's how we've always been doing things. Is that a scalable approach? Probably not, because then tomorrow I said, I need data about X again because I need it for the new month. So they send it again and so forth, right? But I'm asking about X, I'm asking about orders, I'm asking about our latest customers. Like, But how did— when I asked about orders and you heard me and you are gonna go do that work, how do we know we're talking about the same thing? Do you make— do you have that same interpretation and you're actually delivering the correct thing that I'm expecting to go do? I don't know, right? But this is how we've been doing— this is how we've been dealing with data for so long. Another approach is what I call the query approach. You're like, look, you're asking me for this spreadsheet of data. And every month you're asking for it. Like, there's no— what I'm gonna give— I'm gonna give you access to the database directly. And you can just query the database. You can connect your dashboard directly to your database. And here's the query that I use. That's it. Right. Perfect. Now I can go ask more in real time and do things, right? But you're probably, I mean, depending how SQL savvy you are, you, you, you can go edit the queries, you can go add things, and then suddenly this query that was 15 lines long, And you join it with something else and then it's 30 and it's 100. And then, damn, I've seen queries that are 10, 15 pages long, takes 20 minutes to run and it runs and executes. And that's— and the result of that query is what I'm going to use in my report, present to the CEO. That query has so much knowledge embedded into it. Do we actually know what's happening in there? And we're making billion-dollar decisions based on the stuff that's in here, right? And then everybody is doing this themselves, right? So if I make a change in my query, How do I know that that's not the change that you should have done too, right? Or somebody else made a change that I should have interpreted too. So that's kind of now the, call it the query approach. And then you can go off and say, well, no, we need to have this one standard database or data warehouse where everything is well aligned and we're all gonna go query this thing. And we go off and like, yes, let's go build the data warehouse. But what happens? That takes millions of dollars and it takes many years to go do. You get requirements and then Suddenly what happens is that you're like, oh, now you can go answer your question on this data warehouse that we've all invested so much time and money in. You go off and you answer that. You now run the question, run your query again, answer that question. But what happens? You compare that answer to the way you've been trying to answer that question before. Do you think it's going to be the same? Probably not. So then you're like, wait, I have control of this way I get one answer and then I get the answer from this other system, which I don't control. What am I gonna trust? You basically trust your process that you control, right? So then this is kind of why you have issues like data warehouses fail. They don't fail for technical reasons, they fail for social reasons because people don't trust them and so forth. And then the problem with that is I'm doing ETL and I gotta structure my data beforehand. And then we went off and we said, no, let's go ELT, let's go dump our data into a lake. And then the lakes get all messy and then it turns swamps. And then like, then Got it. So you can imagine like this is the story that we've been going off over and over again. And a theme throughout all of this is keeping track of what something means. This is the meaning, this is the semantics, this is the knowledge, this is the metadata. And I think what has happened is that throughout the last 30+ years, we've never focused on keeping track of the meaning, of managing that meaning. And now, I'll argue that LLMs and all of this AI is making us realize, oh, if I want to trust that, I need to know what it means. And now, well, my hope, and I'm starting to go see this, is like, there is now a new focus on we need to invest in semantics. We need to invest in meaning and knowledge. And that's where knowledge reference stuff comes through.

Hey, we'll continue our interview in a moment after a word from our sponsors. One kind of extra little detour before we get into the paper and how it all works. I mean, obviously there's a lot of data professionals these days and they fall into different kinds of roles. I wonder if you could characterize in rough terms, who are the people that you work with? What jobs do they have? What are the activities in those jobs? How much of it— you kind of gave a sense of the different approaches, but If you were to actually just observe, do a time study of these folks, how much of their time are they spending on different kinds of activities? Like how much of it is that routine query running and kind of process that they've done month after month? How much of it is data migrations or these kind of mega projects that may or may not ever come to fruition? How much is ad hoc analysis that where leadership or whoever has one-off questions that haven't been asked before. I have honestly, probably varies a lot by context, but I have no idea really how people are kind of spending their time. And I always find that to be an interesting foundational question to then ask, like, okay, well then which parts of that is AI going to impact? But for starters, maybe you could just kind of characterize what do they do? What are they doing? I never know what they're doing.

So there's the, what I call the data producers and the data consumers, right? So there's 2 types of folks in here. So the data producers are gonna be more in the backend, the technical side. They're the ones who are moving the data, migrating the data, or going, creating, managing the data, what, making sure it exists and so forth. Then you have the consumers, okay, who are consuming. They're the ones who are searching for data, who want the data, who have a particular task. that they want to accomplish. They have a question that they need to go answer. And so those are the 2 types. So if we look at, let's focus on consumers. So on the consumer side, you can have folks who are gonna be your data analysts, right? Or your BI analysts. So people who build the reports and dashboards, right? You have your data scientists, folks who are, you know, your machine learning engineers, right? They're saying, I need to go find this data so I can go do some work To go, again, implement— I create a model, I create a dashboard because I'm trying to go answer a question or making a recommendation or something. So I'm consuming data to be able to accomplish that task. Now, on that side, you hear always like the 80/20 rule. 80% of the time I spend cleaning my data and all that stuff. And that's a problem. And I think actually another annoyance I have is when people say, oh, I have to clean my data. It's data janitorial work. I'm like, no. That's understanding the critical meaning of your data and you're just like sweeping it under the rug. It's like, oh, that's annoying pieces. Like, that is true. That's the essence of what the meaning of your business is, is in that data. And you're like, oh, I'll just kind of write some quick code and it's just annoying to go do. Like, that's the problem is that we don't treat it as a first-class citizen. And then on the producer side, so they're the folks who are saying, oh, we got data coming from these sources and we get the requirements. So we hear that you want to go, you need this data. So we're gonna go move the data over here and now we need it to be more scalable. So we're going to the cloud. So we're going to use Snowflake or Databricks. And I know that you guys, consumers, you want to be able to go run these types of stuff. So we're going to have the data set up in this way so you can go do that. But then they're the ones who are actually putting the data together. How do we know that the requirements that they get are actually being fulfilled correctly? Especially because the requirements from the consumer side They're connected to the business, how the business thinks. Oh, they're talking about customers, they're talking about average order values, right? They're talking about all these metrics. Are the producers, the more technical side folks, do they understand that?

Do they—

is it very clear to them, oh, I could present the data this way or that way, how do I know which is the correct way? And I think that's always been a gap. that what I call the producer-consumer gap. And that producer-consumer gap is there's roles that are missing. One of those types of roles is, I'm seeing this evolve now, I'm calling it the knowledge engineer or also call it the knowledge scientist. And it's really the person or role who like can work on both sides of the aisle, right? Hey, I can be a non-technical, People person, go understand what you're trying to go do. I can draw the models on the whiteboard, then I can go talk to the technical folks and say, okay, it's this data for this reason and so forth. So, and we're starting to also see this as called the data product managers, right? You're bringing product management into data and stuff. So I think we have this gap. So there's the producers, the consumers. And I think one of, to kind of try to, I'm trying to answer your question, which I can't give you numbers because I don't know for I don't know these numbers, but I think there's a lot of just repetitiveness happening, right? We're wasting our time a lot. We don't know if this is correct. And a way that we should be able to kind of address that is by having a role which is focused on being that bridge. And it's a role where you're bringing in— It's one of the people, the process side. We're doing product management. It's doing this knowledge acquisition. It's things that people are doing today. But they're not doing it in a first-class way. They're just like, oh, that's the second, that's the annoying thing I have to go do. I'm like, it's annoying to you, but it is critical and somebody should take the ownership accountability for that.

Interesting. So that role, if I understand correctly, that would be essentially the power user of the data.world platform. I'm thinking almost like knowledge librarian, knowledge historian.

So that's a good point. So yeah, it's like a librarian. I would argue that for data.world, you would, Things like Data.World, Data Catalogs, you have all, all 3 of those, right? So for example, you're bringing in all your different sources. All the— I need to, I need to deal with all that technical data stuff that's coming in. You want Data.World to say, hey, I just got the feeds of data, and I just wanna know that I'm keeping track of these 400, 500, 600 tables. These other thousands, I'm gonna ignore them. So that's where I'm gonna keep my inventory of stuff that's going on. We're moving our pipelines of data. All of that gets managed inside of data.world. So you have that for your data engineers, right? For then your technical folks. And then you have your consumers of data who are searching for data, right? Those are the ones who will also come into data.world, but they're looking at it from a more consumer perspective. Oh, I'm looking for data about customers and about— oh, they can now search for that and they find that dataset. And then kind of what you would want is the data, these kind of people in the middle saying, hey, I want to expose to you, I want to expose to the consumers this beautiful data product, something that just like your shopping experience on Amazon, right? So if you go to Amazon and you're buying a water bottle, for example, well, there's not only one water bottle, right? There's probably hundreds of water bottles. And you go as a consumer, you go in, you search for something and you're like, oh, these are ranked. I have reviews, I have pictures, there's metadata, there's a lot of descriptions. And which one are you gonna go trust? The ones that have great reviews, that has most stars, that have great pictures, that has more metadata, more descriptions. There's other ones that are like, wow, that has, that doesn't have any pictures. I'm not gonna trust that, right? So that's the experience of a consumer. Now, somebody had to put all that inside of Amazon, in this case, put inside of Data.World, and that's kind of on the producer side and they want, and how to decide who goes into what. I think that's where this data product management roles are coming in. And at the end, I think a lot, you talk about the lifecycle of data and stuff, like we're spending money to keep this running, keep this data. It's like, you should tell me where is, what is the ROI on this? I'm like, look, we've invested all this stuff, we've created all these datasets. Look at all the people who are coming to consume this data. I can actually go survey the people and say, if I turn this off, what do you, How is this going to affect you? And they're— I should be able to know these types of stuff. And you want to be able to keep track of all these metrics and lots of how people are using the data and how they're accessing and searching also inside of your catalog in data.world. So I think data.world is something that just connects the entire kind of landscape of data within an organization from the technical all the way to the consumer and everything in the middle.

That's kind of another angle on one of these big themes that I keep finding in the, as, as AI impacts, you know, all sorts of different knowledge work. I have a thesis that the platforms in which people do their work are extremely well positioned to start to kind of create long-form narratives of like what people are actually doing. And from that, perhaps begin to train future AI systems on this data, this sort of data. In this case, it's like the data of how we work on data, but in other platforms, it's the data of how we work on other things, right? It could be creative. It could be, you name it, right? Any of a million things.

Are you capturing these arcs?

Because I could imagine that in a catalog, you could imagine different modes of interaction where somebody maybe comes and says, oh, I have a question about what does this particular thing mean? They come, they get it answered, and then they go somewhere else and execute a query. And so you only see sort of a glimpse of their work process. But I'm sure, you know, you have all these like data connectors and things too. So I wonder how often people are actually like going on a journey within the platform versus kind of flipping in and out of the platform at different parts of their journey.

This is a great observation you're making. And so I think when you look at tools like in general, the data catalog market, it is very separated from you manage all the metadata, the search and the store to find things, and then I go to another place to go actually access the data. One of the things that makes Data.World different from the entire market is that we have both of those things. So you can actually search and find the data, then you can actually access the data within Data.World because we have virtualization federation. I mean, your data continues to stay where it is, right? So you got your data in Snowflake or Databricks or whatever, your data stays there, but you can actually find it and it accesses it through Data.World. So we actually have kind of, again, that's what I was talking like, we have a full view of everything. Now, you brought up something really interesting, which is something that I'm pushing people to go do, is we need to track and catalog not just all this technical metadata, but it's also keep track of all the questions people are asking, who's asking those questions, What are the answers that they're getting back? The actual, not just, oh, I look for this, I'm looking for this data and here's this dataset, go do something with it. Because what we're seeing, we're actually working on at Data.World, it's like, I want to ask a question, show me all the amount of orders that have happened in the Eastern region, right? I should be able to get, oh, here's your numbers by region, by date over Like, you get the actual answers to your question coming from DataRabbit too. So those are— that's kind of where the whole chatting with your data comes from. That's why we're working on this stuff. Now, what happens is that we should be keeping track of the questions and answers because we should govern also. We should keep track of these questions and what is the actual kind of stamped answer for some of these questions that we can. And what happens is that if you're asking for a question, You should be able to say, hey, this other group of people who have been asking very similar questions, and here's the answers that have already kind of been stamped because these are like official things that we have to know for, I don't know, regulatory purposes, whatever. So you should know that type of information. So I think that's something that we're going to start evolving. And then because we'll be able to go learn from patterns that people are having. That's why I, by the way, all of that just continues to get connected inside of your big graph. So remember, I was like, my original graph was like, you have databases and tables and columns and dashboards. Well, then I can extend that with, I have questions, I have answers, I have people. This person asked this question. This question was executed. This question used an LLM to generate this query that was executed over this database. And here's the answer. And Nathan is the steward who said, this is the official answer for this type of question. And so next time somebody asks a question like that, they can get a really a very similar suggested question and they have a governed answer for that. Like, I think that's kind of where we're going, especially for enterprise scenarios. We're going to have accuracy and explainability. And then obviously you're going to have questions where just like, I'm just discovering things, right? I'm trying to be creative around that stuff. And then I think that's kind of also when it comes to question answering, we should also think about what types of questions. I think this is where a lot of the agents come in. So if you think about it, question answering perspective. It's not just give me a question, I can answer to you. It's like, okay, you're asking a question. What type of question is this? Oh, is this a factual question? Is this a subjective question? All right, 2 different approaches, right? Is this a factual question about the knowledge? Is this a factual question about the data that I have? Do I even have the data to be able to answer this question? Or do you even have access authorization to be able to get answers to that? Oh, maybe not for all of it, but for this subset you can. So then like the agent should be able to go understand what you're trying to go do, understand your context, where you're coming from and saying, hey, I can't answer that question, but here's a subset of that question I can answer and I can't answer it because of the other reasons. I think that's the type of stuff that we're heading towards. That's what I get really, really excited about AI right now.

Yeah. Accuracy and explainability. Those are two It's funny how many— I mean, it's not surprising in the sense that, you know, obviously AI systems, AI models are all trained on data and they're very, um, you know, they're very much like directly derived from their data. Accuracy and explainability come up in, you know, all the AI systems work that I see. It's interesting that that exact same, you know, framing, you know, clearly predates it in the, in the data management work that kind of underlies now the AI that's getting built on top. When it comes to accuracy, do we have any good benchmarks, rules of thumb, reference points for how accurate humans are? This is one thing I find everywhere I go, that people have no idea how accurate the humans are.

I love that you're asking this question, and this is really how we How, how we need to have fierce conversations with each other and push back. So like all the research that we've been doing about, oh, the accuracy. And so the, the, but the first set of critics, what do they say? Oh, but 60% isn't accurate enough. Yeah, of course. I know it's not accurate. So what is accurate enough? 100%? 99%? What is it? Do you actually know? Okay. You, you got data today to go answer a question that you may believe is a barred decision that basically bars. Do you know how accurate that was? Are you going to bet your life on the accuracy of that? Go ask that question. I don't think people— I don't know. Do you know? You got that dashboard or that spreadsheet from Alice and Bob. Do you know how that work was done? You trust exactly? Really?

Do you? Do you?

Do you?

People aren't asking those questions, right? So they're like, they make the assumption that the machine needs to be accurate in everything, but then they don't even ask. They're not being critical about their own process to go do that. I think This is an opportunity right now to be very critical about kind of how do we evaluate ourselves? Like, how are we doing this? And going back to like how AI is impacting all the tasks that we're doing, it's like, let's go figure out how we're solving that problem today and saying, well, wow, this is a very, I mean, what is that process and can we improve that process? And we figure out, oh wow, there's a big bottleneck or that's something that is pretty sketchy, it's not working well. And you want to be able to bring in the machines, the AI to go help for those things. So I think it's really, really important to ask ourselves, how are we solving these problems today? And be honest about it. It's like, oh crap, that got— but that's something that may not be doing it. I mean, I've done this. I remember once working with a customer who were like years getting this report and making decisions about it. And I went off and go analyzed how they got that report because they hadn't done it. And I'm like, well, do you know that to get this value over here, you've, there's some hard-coded values or like multiplying 0.02 or whatever, like that's happening. And they're like, why is that? And then somebody said, oh, I remember like 2 years ago, we were like doing some special discount and like, well, guess what? The person who wrote that query hard-coded that discount, whatever. And it's been there for the last 2 years. So all the decisions you've been making is, has that embedded in there. All your numbers have been wrong for the last 2 years and you've made decisions. Now, but the honest OBS here is like, but the company still made money and stuff. Like, how big of a deal is that? So I think that's also, it's like, how critical are these things? And like, yeah, sometimes they're like, that's good enough, right? So I think it's also part of the culture that you have within your organization.

I've experienced, you know, some very similar things. And in a non, you know, at the company that I started, we did a lot of digital marketing for a while. And we both helped our clients do it and then we did it. And we found just so many instances over time where you'd come to some initial conclusion and you really have to challenge yourself, especially digital marketing will punish you for being wrong on these things. We really had to challenge ourselves. Is this really right? Does it check out across everything else we know? Does it violate our world model? And dig in, dig in, dig in as many layers as we could to finally often find that there was some problem that happened at the Washington Mutual thing too. I remember one time where I was summing something and it was greater than the limit that it was supposed to be governed by. And it was like, well, what's happening here? Your query must be wrong. And it turned out actually, no, like some other process in the bank had changed some underlying data and all of a sudden everything had kind of, the sands had shifted under us. But yeah, broadly I find that, I don't know. I mean, if I had to guess, I would estimate that like a significant fraction of The data analysis that people are basing business decisions on is just fundamentally flawed to the point where they're kind of detached from reality. I don't know if we could get any more rigorous in our estimation of that, but for me, it has been a recurring theme that just somebody gives you the results of a query or a spreadsheet or whatever and tells you it's X and it's It's like, I do not take that stuff at face value anymore. We were just talking a little bit too about podcast metrics. Oh my God. We had one, we were looking at the success of the podcast and it was like, oh, we did really well in this time and we did really poorly in this time by comparison. Look, there's a spike and there's a trough. And it turned out that it was like the way that the month fell with respect to the weeks and the days on which we published the podcast. And it was like, well, we actually had 9 episodes released in this month and 7 in the next month. And that's why we got way more downloads in the first month than the next month. It really had nothing to do with that. The per download, the per episode downloads were the same. But even at that simple of a level, I've seen people just get so confused by kind of aggregate measures. Do you have any, I mean, this is a little bit of a digression from our core topic, but do you have any tips or sort of habits of mind that you encourage people to— mantras maybe even— that you encourage people to adopt? I always say there's no substitute for reading the raw data. If you have not looked at actual raw records at some point in the process, obviously you can't read all the raw records, but if you have not familiarized yourself with at least some raw records, my guess is there's a pretty good chance you're going to be wrong in the aggregate Any things like that that you preach?

I mean, ask the why, right? Ask why 5 times, right? What is the goal? What is the objective? What is the objective above the objective that you're trying to go do? Because this goes back into like, oh, we have all these ad hoc questions that we're trying to go deal with. I'm like, why are you doing that? How is that valuable for the business? Like the person, you give that to that person, what are they going to go do with that? And so for me, it's always like, Are you tying the work that you're doing to specific business value? And the way to tie that is that you need to know what are the objectives, corporate strategies of the quarter, of the year, whatever that needs. Like, you know what those are because you need to be able to push back saying, I'm not going to go spend time on that because that is not related at all to what our quarterly goals are unless you explain it to me, right? So I think That's something how— so that's kind of more from where I am going to go ahead and top, right? Get connected to the business to make sure that we're not wasting time to go do that. Then another kind of another approach technique that I do is what I call the iron thread, which is like, look, how do I know if something is working correctly? Like, don't boil the ocean. That's another mantra. Don't boil the ocean, right? Love to get it. So start from something very specific, One small thing and figure out what is the path all the way to the end goal and do it really, really small. So you have to do it just, you basically get that one thread through and then you're like, okay, I understand all the things I need to go through. And then you say, let's do it again. And then you add like, basically you're adding another thread to it. Then that thread gets bigger, bigger. And then at some point then you're like, well, how about let's do it for another area, something different. And then you, so you can have somebody else in a distributed way independently go off and do another thread somewhere else. Kind of like, okay, now we start getting these kind of threads all straight. I think that's kind of an approach I always do because otherwise we kind of like get focused too much on the bottom stuff and then we're not connecting it to the value that people are seeing because we don't even unders— like, I need to understand what is the, not just the output, but what is the outcome that needs to happen? And so you want to be able to kind of just be able to drive all that through from top to bottom, from the executive all the way down to get into the nitty-gritty, roll up your sleeves. So all that needs to be connected, but you don't have to boil the ocean, do it for everything. Just do it for one very specific thing, go through it, then you understand how things flow and what's working, what's not working, where's the struggles and so forth.

Yeah, interesting. Software development sometimes can be like that more generally too, where you just, it's a vertical spike through all the layers of the stack to get everything connected and, um, you know, make, make one successful round trip is like a good, uh, early milestone in a lot of things.

And tie this to AI, it's like, well, that's like all the steps they need to go do it. Then you figure out, well, where can I put AI in here to automate things and make it faster, better, more productive?

So perfect transition then to the paper. We've got a lot of preamble before the actual paper that you've published, but high level, it seems like the big demonstration of the paper is that you take GPT-4 and you say, hey, I want you to write some queries for me to answer some questions. And you give it either the pure schema, like the table definitions, and there's a lot of Meaning often implicit there in the, or to some degree explicit in like the names of the tables and the names of the columns and the way that database keys and whatnot relate to each other, right? You can sort of, there's an implicit graph, again, somewhat explicit just in the schema structure and you get a certain level of performance. And then you compare that to, okay, now let's also give GPT-4 a higher-level semantic knowledge graph representation of what does this data actually mean. So that, perhaps not surprisingly given all this preamble, yields quite a bit better results. What additional details about that experimental setup or about the findings do you think are most important for people to understand?

Everybody was excited about chatting with their data. Chatting, also call it text-to-SQL, has been an area in computer science that people have been looking into for, again, also probably 30-plus years. So there's been all these techniques happening. And now with LLMs, people are like, well, this should be much more easier to go create, to do text-to-SQL. And from an academic standpoint, there's these benchmarks and techniques people have been doing out there, and they show 95% accuracy. And I'm like, I'm skeptical about this stuff. And when you look into these benchmarks, they're all very simple. They're like, oh, there's a couple of tables and the semantics are very explicit in there. But I'm like, that's really disconnected from the enterprise because the enterprise doesn't always have that clear and it's very clear and small. And then the other part is that the questions people were asking were like, oh, here's like a laundry list of questions that they're like, How, why these questions? How are these, how do you come up with these? So the main research question, so there's 2 questions. One is we want to understand to what extent can these large language models kind of generate SQL queries that could be accurate? To what extent that can be done? And then the follow-up question there is, to what extent can that accuracy improve if you actually put the knowledge graph in the middle? And the hypothesis here is that, hey, if you do the knowledge graph, it's going to improve the accuracy. Now, we didn't know to what extent, we didn't know how much. And the thing is that just talking to folks, everybody's like, yeah, adding semantics, context, knowledge graph like that, that should help. But one, how would we do it and how much better is it? And like, people didn't know. And I'm like, we were at Snowflake Summit in June and we were having this conversation with a bunch of product people and they were the ones who frankly were just challenging us saying, You guys should just do a little bit of benchmark around it. And I'm like, yeah. And literally, I got on the plane, came back, and I started working on this stuff. That's, I mean, that's kudos to them for kind of really challenging, pushing us to that. So that was the whole setup. So what we did was that we have an enterprise schema using, in the insurance domain. So we're using an open standard called, from OMG, they have a property casualty schema. So this is a representative of an enterprise domain. Enterprise schema right there. And then the other thing was the questions we're asking. So you have an enterprise schema, and then we have a set of questions. But these questions, we're actually putting them, we created this quadrant about with 2 kind of spectrums of complexity. One of the spectrum of complexity is on the types of easy questions to harder questions, basically. Easy questions being, give me a list of things. Show me all the claims that are open, right? Those are easy questions. Harder questions are things that are going to be more about strategy, and which involves— maybe questions already, I need to do some aggregations, I need to do maths around that stuff. And then from a technical perspective, you also have easier questions and harder questions. Easier meaning I only need a couple of tables to answer this. And a harder question is I need a lot more tables. 8, 9 tables that need to be joined. So then you put these 2 spectrums together, you get this quadrant of, like, easy questions over easy schema, and then you get harder questions over easy schema and so forth. You get all these different quadrants. So that gave us a perspective of understanding the types of questions that people can ask. And then the 3rd thing that we looked into was, let's make— let's invest in putting and coming up with the semantics. Invest in creating the context. That's what we call the context layer. So it's, here is the ontology semantic layer, and here's how these things map. So very simple. There is a claim, has a claim number. Oh, there is a table called claim. Okay. Perfect. That matches. But there's actually, there's, that table has, I don't know, 15, 20 columns. One column is called claim identifier. That is not the claim number. there is a column called the company claim number. That is the column that has the claim number, right? So that's— so then I make these mappings, and then you would have things like values. You'd be like, well, I want to know where— what are my policyholders? Well, you have to get— it's everything that has the role column equals PH. And your agents, oh, where role equals AG, right? That's the semantics. Like, that's the context that you have in there. So We made that very explicit. So that's what we come up with kind of as inputs kind of into the benchmark. And what we did was evaluate saying very simple setup. On purpose, wanted to do very simple because I wanted to like get the bare minimum of very, the simplest prompt that you can imagine. Here is a SQL DDL schema. You copy and paste the SQL DDL and you say, Write a SQL query for the following question. And then for the knowledge graph, we said, here is the OWL ontology. The ontology, that's a semantic layer, which is an open standard that we gave it, right? We understood that GPT-4 knows these open standards and you copy and paste and you said, write this SPARQL query for this question. And SPARQL is the open standard for knowledge graphs for the queries. And we just, so also the caveat is that We put the schemas and the ontology, the semantic layer, as part of the prompt, right? So it has to fit within the amount of tokens. And you just give that to GPT-4 and it generated the query. And then when we started comparing all the queries that were generated, I think that we get that 3x differentiator. So there's 3 times more accurate queries if you're doing the knowledge graph versus if you only do SQL. That's if you compare all the questions. And if you look inside of the quadrants, effectively when you had questions that required high complex, the tables, basically questions that require joining more than 4 tables, SQL query that was generated by GPT-4 would always fail. And I think that was an interesting point. It's like, oh, so if you have queries that require more than 4 tables, like that's not going to work very well. So, and that was for this, for this very simple prompt that I think up work that we're doing right now is, well, we're going to go test it with other models too. And I think what we want and what the community to go do too, and we're seeing this, is like, well, yeah, let's go improve the prompting. Let's go figure out more context. Is there a way I can pass the context to SQL in some other ways? Like, this is where we need to start working as a community to go figure out. We presented an initial baseline. So going back to our original research question is to what extent we found an extent. We know this. And now this is how we advance science. It's like, let's get other people to build on that work and let's see if we can improve the extent.

It definitely is, again, another major theme, just how many of the benchmarks that largely have come out of academia over the last 10 years have just been kind of blown away by the latest models. I think that's one of the really interesting indicators of just not only how fast things are moving, but how fast they're moving relative to people's expectations is that you see 2019, 2020, even some 2021 benchmarks that are just obsolete because they're totally broken by the latest language models. They were built in the way that they were because people couldn't even really conceive of at the time that an AI would get good enough to solve all this. It was like, well, gee, if it solved all this, we'd have AGI. Turns out there's a little bit more room between solving these benchmarks and AGI, at least. So now you've got this kind of first stake in the ground, basically, right? Here's all these old simple schema benchmarks are increasingly obsolete. Now we got to increase the challenge. Let's bring a real enterprise-grade challenging problem to the to the models here and see how they do. They don't do so well. Here's one initial technique of layering on semantics that makes a huge difference. And of course, we're fully expecting that people are going to continue to improve on those techniques and get better and better performance. Do you have any tips for how people, and especially for enterprises that want to do this for themselves, for actual practical purposes internally, What sort of guidance would you give them on constructing their own internal benchmarks or eval suites?

The contribution of our work, one, is not just the results of the benchmark, but it's the benchmark framework itself. So what I'm telling everybody, and actually working with our own customers doing this too, because they're like, We're working together to build these chat systems, these agent systems and stuff. And they want to evaluate how good this is. So one, they have their data, right? So they got that. First check. Number 2 is on the questions. So I think this is really important to understand what are the questions people are asking and goes back to what we were talking earlier before. It's like you should, we should catalog and keep track of the questions. Right? So what are the questions people are asking today? Who's asking them? Why are they asking them, right? And how are they solving those answers? How are they getting those answers today? Are they getting those answers today through a spreadsheet that somebody gives it to them? Or do they write a query? Do they get it through a dashboard? Like, let's figure that out today. So then what happens is that get those questions, let's then put 'em into those quadrants, right? Is this an easy question? Is this a harder question? Or is this a question that is just a list of things versus this is a question that actually involves aggregating all that stuff? And this is a question that requires small amount of tables or large amounts. So start putting that into those quadrants. So that is my main recommendation for folks. And then in order to avoid boiling the ocean, make sure that you're focusing the questions that are actually going to provide some value, right? And that's why it's important who's asking those questions and why they're asking those questions. So I think that's a very important thing. And then the third one is the context. And I think the whole point of the work that we were presenting is that you should invest in the context. You need to invest in the semantics of what this stuff means because frankly, that's what the LLMs don't have. So you need to start investing in that. So that's where people, that's where at least our customers, they're already investing in the context because they already have data.world. So, but For other customers who are like thinking about this, I'm like, well, if you want to have a chat with your data on your relational SQL databases, and if you don't have a catalog, you've got to solve that. You got to go build, bring in a catalog. But the catalog is for the sake of understanding, keeping track of all that metadata to start building that semantic layer and everything. And then don't boil the ocean. That's where I would say then that have that iron thread approach and tie it, go bring in the data you need to go answer those questions and then keep track of all that metadata. Treat metadata, treat the semantics as a first-class citizen. So again, just summarize. One, you already have your data, so great, check. Second, make a list of all the questions, put them into quadrants and figure out and prioritize them by the why behind them, and then understand how they're answering those questions today. And third, if you don't have a catalog, then that's where you can start from that technology perspective.

So what do you think the prospects Are for this kind of chat with your data paradigm, right? The dream would be like, we, the AIs get really good and the decision makers, you know, whether it's the middle of the night when they have their brainstorm or, you know, whenever they can. I mean, AI has a lot of advantages, right? It's like always instantly available. It, you know, you can kind of pick up where you left off. I also recite these. Here, the big challenge is presumably accuracy. It seems like we're probably not there yet. You said kind of 60%, it's not good enough. I would assume that, yeah, probably not. Do you have a sense for what is good enough and sort of what the steps are likely to be to get there?

In the benchmark, it was like 60% for all the questions. Now, what we really need to start thinking about is, Given a question, I need to analyze, can I even answer this question? And I, so I think the first, the thing, this is what we're working on right now is saying, hey, give me questions. I know that I can answer this question. I know, or give some confidence measure that if I can answer the question or not. So then what you're going to do is reject the questions that you know you can't answer. And then you're going to focus on answering the questions that you know you can answer. And I highly suspect And this is stuff our research has shown us, then from there we can get to the 90% of that stuff. So I want— so if I zoom out and I imagine the user going into your chat, you ask a question, you'll either get an answer with an explanation, and I'll dive into that in a second, or an explanation saying, I'm not going to answer this question for the following reason. That already gives you confidence around that. Now, when it goes into, like, when I do know that I can go answer a question, what happens is, this is where the knowledge graphs come in, and it's tied to accuracy and explainability. If I ask a question, I can extract using the LLMs, extract the concepts, all the things that are being asked in that question, and I can look them up in the graph. And if all the things that I'm asking show up in the graph, It's open in the knowledge graph. I'm like, oh, I know about all these things. I can then formulate a query. But if I'm asking for a question and I'm like, oh, I don't— I can't find this in the graph. I'm like, I don't know. So basically, you know what you know and you know what you don't know. So then that's how I can explain that I don't know this, therefore I'm not gonna go answer that question. And then if I do know, I can then create the question and then I can answer it, execute it. And here's a— here's a big distinction. Is that LLMs are not answering the question. Because if the LLM is answering a question, you're training an LLM, then it's always going to be probabilistic. Like, it can't be for certain. I can't have certainty behind it. LLMs are going to be an assistant to create the code that is going to be executed over the system that's answering the question. And when it's code, That is something I can reason upon. That is something I can govern. That is something I can manage. And it's something I can use for an explanation. So if I match it to the graph and I'm like, okay, here it is, I generate this query. This is a query over the graph. And I can, and then this is where all the technical metadata catalog comes in saying, oh, this exists in the graph for this reason. It comes from this part. It goes to this table. And this person was involved in the creation of this. It was authorized by this. That's why you're getting that information. So I can then not only give you the answer, I can give you an explanation and I can go as deep and granular and nitty-gritty and technical that I want, or I can give it high level. So I think also one of the things that we're working on is to— another kind of a little bit of an annoyance I have is when we talk about explanations, explanations to whom, right? You're giving a non-technical person an explanation. They probably just want an explanation in English and also know who they should go talk to if they have a question. Right? Don't give them code. They're not going to understand it. If you're giving an explanation to a technical person, they probably want to see the code, right? If you give them just a high level of fluffy, they're like, that's— they're going to call BS on that, right? So you also need to understand who are the personas you're giving explanations to. And again, the graph is just basically, I can traverse the graph as much as I can to get as deep as the level of granularity that I need. So I think that's why knowledge graphs are critical for the accuracy And for the explainability.

So I'm kind of mapping, you know, a bunch of familiar AI application building techniques onto this problem. And one is decomposition. It sounds like, you know, you could imagine just taking the semantic layer and the schema and dumping that all into context and saying, hey, have at it. Or you could imagine kind of breaking that up and saying, okay, first thing, your first job is to just look at the semantic layer and determine if you have the information to answer the question. Then you can have a peek at the schema and actually generate the query. That decomposition takes more work. It definitely kind of makes things a little bit more brittle depending on how quick your system is to change or whatever, but it really can drive accuracy. I imagine that's a part of it. Fine-tuning models, obviously also a big trend. GPT-4 fine-tuning is in limited release. I don't know if you've had a chance to fine-tune that, or if you've had experience fine-tuning other models to get better at this. But that seems like another opportunity for improvement. And then you mentioned earlier too, established questions and kind of canonical ways to answer them. I think they're of these sort of skill database type paradigms, like the Eureka paper or the Voyager paper out of Nvidia, where when they find something that works, it's like, okay, cache that basically to a database. And next time we get a similar question or a similar challenge, and in their case it was like Minecraft type of skill building, but here it's obviously database type work. Let's see if we've done this before and have kind of an established answer. I imagine all 3 of those are kind of directions you are pushing on?

Yeah, the answer is always, it's hybrid, it depends. So yeah, for sure. So I think from the science perspective, I start with one thing to go see how well that stuff is. And this one is like, let's start with the more bare minimum prompt engineering to go do things. And then you have other tools that you want to kind of figure out, like, what can I use for vector databases that I can, as another tool there. It's like, also note that for the first experiment, like everything fit in the context window, right? So maybe I have my ontology, my simulator is going to be so big that I can't always pass it in here. So I need to be able to kind of get parts of it out.

Were you working with the 8K? version in this work or the 32?

The 8K.

Yeah.

Because I mean, the IntelliVision wasn't that big anyways. And then when it comes to fine-tuning, I think there's 2 things to look into. One is, are you fine-tuning with the data or fine-tuning with the metadata? And I will think like fine-tuning with the data is probably not going to be valuable because first of all, you got millions and billions of rows in my Snowflake or Databricks, my data lakehouse. Are you really going to extract that and put that in there? It's so expensive. And then which gets updated all the time. Like that, And then, by the way, then in that case, your LLM is trying to go answer the question, but then it's going to be probabilistic and it's non-deterministic. And you, like, if you're going to go ask something and it's going to give you your balance for your bank account, it better be freaking accurate all the time, right? So, like, I don't want to— so there's particular scenarios where you, I mean, the accuracy is critical, critical. So, but when it comes to, like, fine-tuning on the metadata, I think that's going to be like huge opportunities. And as you said, it's like, oh, here are these patterns of questions, here are these types of questions, and it fits these patterns of queries. Like, that's the stuff where you want to go fine-tune on that metadata there because then it's going to like, I know you're going to generate the right, increase the possibility to generate the right query. Now, another thing is when you generate a query, you're generating code, and I can reason upon that, and I can do that in a deterministic manner. Right now I can actually check, I can do static analysis and I can check if this is the correct, if the code is going to be correct or not, probably with a bunch of heuristics. So then I think there's another set of kind of techniques that you can go use. It's a post-processing. So it's prompt engineering, you're going to do vector databases, you're going to do fine-tuning on the metadata, not on the data, the metadata. You'll do some post-processing. So these are all basically just a bunch of tools that you'll have. And what we'll see is like our agent frame, an agent framework that will, for each kind of state in that agent, will use this tool. For this state, it'll use this tool. For this tool, it just has a lot of tools at its disposal. And I'd argue that a lot of the IP that will come out is on how people are structuring their agent frameworks and how At the end of the day, you want to have an intelligent agent who can really make, understand its environment, perceive its environment, make decisions autonomously based on their environment. And that's really going to be how sophisticated your agent framework, your state model is going to be. The current kind of RAG architecture that we all talk about, I think that's the simplest, most naive type of agent system. You get a question, you send something to them, you send it to the vector database, which then get context back and you send it to the LLM. That's kind of one-stop shop for everything. That's going to get it broken down into so many different pieces. And that's how we're going to start developing kind of these agents, which by the way, this agent stuff is another thing I'm calling out there is we've been working on agents. This is the good old-fashioned AI. People are working on this stuff for like 50+ years. There's so much work on agents, on planning. So hopefully we don't all spend time reinventing the wheel and build on the shoulders of giants so we can advance very quickly here. Yeah.

One thing, uh, a tidbit that I've found, and then I have a project I want to ask for your, uh, advice on is when fine-tuning, this has definitely proven true for GPT-3.5, unclear, you know, everything has to be kind of revalidated when you move up an order of magnitude in scale. So I'm not sure yet how this will apply to GPT-4 fine-tuning, but for 3.5, at least fine-tuning on chain-of-thought type data has proven extremely valuable for me in terms of getting the fine-tuned model to behave how I want it to behave. It has not been good at learning facts that way. So certainly I wouldn't fine-tune on raw data, even fine-tuning on the definitions of tables and stuff. I wouldn't expect that to work very well either. You might do it, but you're still going to need it in context to be accurate from my experience. But where I've seen the most boost is in kind of explaining the way in which we want you to go about going through this problem. And having a fine-tuned, and it doesn't have to be huge. Again, in my experience, as few as like 100 examples can get you a huge boost at the 3.5 level. But it's just really critical to have that kind of explicit step-by-step that the model, because obviously we know that like chain of thought, you know, think step-by-step is really helpful. But you have to kind of demonstrate what kind of, you know, chain of thought do you want or do you need, you know, to solve the problem?

in the particular context.

So here's my challenge, and this is really why I kind of got down this rabbit hole of knowledge graphs a little bit in the first place. So I'm working with this company, Athena. We're in the executive assistant space, and we are trying to deploy AI in a million different ways to enhance our service. One of the things that we know that our clients would really like is if we had some way for the EAs to have always-on access to everything about them, right? All their preferences, all their history, whatever, anything in the background, right? A lot of times right now, the EAs have to ask questions, and it would be better if we could somehow answer those questions without them having to ask. So the idea is, can we create a long-lived client profile that we can sort of maintain and update over time? A huge challenge, though, is Almost all the data is unstructured. We interview clients when they come on board and record that call and transcribe that call and then summarise that call. But it's all text. The whole thing is text. There's no schema whatsoever. So we can then throw that stuff into a vector database and query against it and get kind of what did they answer at the time of the onboarding call. But then of course things change, right? So How do we add more to that database over time? And how do we sort of manage what supersedes what? And I wonder, this is pretty early stuff. I think we're like, I don't know if we're on the absolute edge of tackling these problems, but there's not too much work out there on this kind of thing yet. But I wonder if you have any kind of thoughts or intuitions for us as we think about something where the core challenge here is there is no schema. It's all just text and we're trying to sort of be able to dump new stuff in there, but then have it maintained in a way where we get the right information at any given point in time. And we don't have great solutions for this yet. So any pointers or suggestions would be valuable.

I will argue that there's always a schema, there's always an ontology, there's always semantics, and it's implicit because you're talking about the same concepts over and over again. You're maybe using different words that mean the same thing. So an idea here is like, take all those, all that text that you have, all that unstructured stuff, and just ask GPT to say, generate an ontology or taxonomy or business glossary from these things, right? And I start, and then you start identifying what these concepts are. And I think for your particular space, right? Like, Think about the domain, model the domain. Like, what do people do? Like, what are the roles involved? What are the tasks? Like, those stuff are probably, you're probably going to have really generic ones that are going to go, they're going to be specific, they're going to be generic across all types of companies. And then I'm sure there are going to be things that are very specific, but then kind of figure out what are the high-level ones that are generic. It's there. I mean, you said there's no schema. There is a schema, right? There's a meaning behind everything we're talking about. So just to kind of automate it, just go get all that text and just say literally the prompt. And I do this all the time, like generate a business glossary or generate a semantic layer, generate an ontology based on the following texts or questions. It's going to come up with a bunch of stuff and then you're like, oh yeah, that thing is related to this thing and that. And then now you have like a consistent controlled vocabulary way. And that's something that you can use later on for And your chain of thought, right? Or just even how you're presenting things in the user interface or how people are just chatting or conversing with it. I think in a way, this is anecdotal, but I kind of, when we set up these, when we created these semantic layers ontologies, what I've seen is that people start changing their behavior a little bit and using the words that were defined inside of that ontology, inside of that semantic layer. And then that's a human behavioral thing. It's like, oh, well, if I use these words, I actually get my answers, right? So then they start changing. Then actually, as an organization, we all start using a common vocabulary. We have that lingua franca. And it's like, at the end of the day, like, yeah, we should, we should, it would be great if we started to agree on things. And I think this is also a way to kind of get that. Have that human behavior. So anyways, long story short, there is a schema in there. Use GPT to help you extract what that schema can look like based on all the text that you have. And then, then reuse that in just any other techniques that you're doing.

Yeah. Interesting. So would you recommend right now we are kind of just chunking and dumping stuff into vector database and kind of querying against that? We're using the HIDE technique, the hypothetical document embeddings, which is Basically the translation, when a user comes to our chat and says, what is X about the client? We translate that to something that we think is what the answer is likely to look like. And that's the hypothetical document embeddings. I'm sure you're familiar. And then that seems to improve the accuracy of our retrieval, but we're purely using this vector database. We're not really using much structured data. Do you have a recommendation for what database would you recommend? Should we go with a Postgres with their vector thing, or should we be thinking graph database? Because we do have a lot of— I think a lot about that AI Town paper. I don't know if you saw that one, but it made the rounds. It was like these little AI agents were running around and making plans and interacting with each other. For me, one of the most interesting aspects of that paper was the way they handled memory was through a periodic sweep of all these raw observational memories into these kind of higher-level synthetic, like thematic, you know, or kind of more episodic memories. They would, you know, the agents would write down, I talked to Juan at this time, we said this and this. But then, you know, there's too much there, you know, pretty quickly for the, you know, especially with limited context. That's of course expanding, but to manage that, they had to kind of synthesize it into this summarized layer. And I thought that was really interesting. We haven't implemented anything quite like that. That would seem like a very natural thing perhaps for a graph database of like, this was synthesized from this, or these, if I'm looking at some abstracted summarized memory or description, like what are the raw observations that that came from? Where do you think I should put that data?

I think this is going to be a combination of using whatever vector database, vector similarity indices, whatever. And then I think some sort of graphs. I think this is my perception is that that's where things are going to head. And then, I mean, at the end, just use whatever graph database you want and things can change. I mean, I'm not a big proponent of like use this database or whatever. You are going to have virtual knowledge graphs where things where you have all your enterprise data. Like, you're not gonna move all your data from Snowflake or Databricks into a graph database and keep copies of that. No way. That's not gonna happen. You're gonna have a virtualization layer. So it's gonna, you can query as a graph and it gets translated to SQL for that. I mean, that's kind of what we, that's exactly what we did for our benchmark work. So that's something if you already have existing data in a relational database or your data lake, You're gonna have a virtualization, you're gonna acknowledge that virtualization layer. But if you're starting from scratch, then I think you're gonna have a combination of those 2. Guess what's gonna happen? Every single structured database is going to have vector features. If they didn't have it yesterday, they had it today. If they have it today, they'll have it tomorrow. So all graph databases are having this right now, as so well as so much all the SQL databases are gonna do that today. Are vector databases gonna have SQL interfaces and graph interfaces tomorrow? Yeah, I've been explicit about this. Vectors, databases, that's a feature, not a category. There will only be one winner. Whatever vector database is, there's only one winner. Everything else is just going to be, I'm just going to use this other database that's going to have a vector feature and that's going to be enough. That's what's going to happen in the next year or so. And you will use that. You will need to use the vector database if you have a real scale issue, like a scale need. Otherwise, all databases will be adding vector features.

So, well, folks can juxtapose that against Anton's answer in an earlier episode. His big thing, which I did think was compelling, he basically said the data that comes into Chroma has never been in a database before. And he was basically like, just the overall growth of how much data is going to go into databases is headed for like 10x because of just all the structured, unstructured stuff moving into these, these new structures. And so like everyone can win, you know, because the whole market is growing 10x.

Well, my whole point is that you look at the Snowflakes and the Databricks and the Azures of the world who have already all the SQL stuff. They're all going to say, well, we should be able to go support more unstructured and they will. And they will do that. Now these vector databases, they're like, oh, we do that too. But they're like, oh, but you don't offer SQL. So what, they're going to now re-embed SQL, add SQL to that stuff? No, they're not. And then it's now for scenarios where you need best scalability and do all that stuff for vectors, then you will want that. But that market is not going to be that big. It's going to be, there's going to be one big player and everybody else standing behind. I mean, it's like MongoDB, right? MongoDB is the winner. And there are other kind of NoSQL databases. They're all small behind. Where's one winner there? It's MongoDB, period. They'll look at their stock. Yeah, I think it's what's going to happen.

So what other uses for AI in data are you most excited about? I'm thinking about things like, as an application developer, I find it still pretty tough in a lot of cases to figure out what are people doing as they go through my application. It's easy to log stuff, log clicks and log whatever, but To really synthesize that into a story. You know, a lot of kind of products try to do that, and it's been a big struggle in my experience to get them to work well. But I feel like maybe AI can kind of come along and narrativize. I also think about things like anomaly detection or sort of, you know, determining like what's relevant or not relevant when it's not obvious by definition. But I'm very curious to hear, like, you know, aside from this kind of chat with your data question answering use case, What other roles do you see AI playing in the data world in the near future?

Suggest descriptions about things. People have to go write these descriptions and they're like, well, at least I don't start from a blank slate anymore, right? So, and that's just applicable. I mean, I need to write an email about something, right? I just use it and don't have to start from scratch, right? So I think that you're applying that for that type of metadata, just like any type of documentation. That's a big lift on that thing. You can also use it, like, here's another one. I mean, talking about anomaly detection or PII, like, just pass the schema to GPT and say, out of this schema, which columns do you think may have PII data, personal information, right? Or it will give you some things of like, hey, those are actually pretty good candidates. And guess what? You didn't have to look at the data at all, right? And that's probably, and that's something already, it's a big lift around that stuff, right?

Yeah.

So a lot of the documentations and kind of this augmentation of metadata, I think that, that's one thing. Second, for, for search just in general, I think just more natural language search, that's gonna be another one. Again, that's a productivity lift, right? One that I found really exciting is on, like, explanations, right? So it explains code pretty well, which is another type of, it's another type of, of documentation. So a lot of this documentation you do. And one that I'm really kind of, I find this really interesting working with one of our customers, WPP, who's their, the world's largest ad agencies, they wanted ideations. So I want to kind of use the LLMs as a way to inspire me. So basically it's like, Data.World, you have, you are a catalog of all our data. You know our datasets. Tell me what are the questions that I could be asking with the data I have? And then it's like, oh, so I start generating candidate kind of questions. Hey, you could be asking this question and so forth. And then people are like, oh, that's interesting. I didn't think I thought about that. And you can keep track of all these questions and start connecting them. And then we've been doing these things like, oh, I have this problem I want to go solve. What data do you recommend me to go use around that? So it's like, well, you can give all this context. Here's all the context data you have. Here I have the problem. GPT, come up with— LLM, come up with some interesting solutions and then make suggestions of what they can do. So it's about ideation, but I think that's, again, more of the productivity lift. So those are things that we're seeing here. Another one is around this knowledge acquisition. It's like, let's go talk to people. Wait, so what do you mean? When you mean customer here, what do you mean by active customer? Like right now we have to have those conversations. Well, human to human, like what? And now I can kind of automate and scale that out saying, hey, can you please spend a couple of minutes chatting with my bot here? They're gonna ask you questions. And so we've done this, set up our own GPTs and like be a psychologist, be very Socratic. And ask somebody what are the questions that they're trying to go, what questions do they need to go ask to the data and why is that important? They keep going in, why, why, and try to extract the knowledge. And then in one day I could talk to 20, 30 different people and then I could use GPT again to summarize all these discussions saying, hey, look, when we talk about active customers, here are all these different definitions of what they could be, right? So that's really, Helping me again in the productivity way, faster understand what we mean by things. And that's the one I'm most excited about. And I hope people get excited about that too, because that's the one that has always been that bottleneck. The knowledge, acquiring knowledge from people's head has always been a bottleneck, which is you've got to go talk to somebody. Now we actually can do that at scale with this. And that means that we can learn so much about how the business operates. Map out business operations. We want to have operational excellence and understand basically all our business processes and like, how can we improve these things? Like, well, we don't even know what our business processes are. Well, I can go interview somebody and say, how do you do your job? And I can map that out and say, wow, look at all these manual steps that happen for this stuff. Like, that's a bottleneck. We need to go improve that so it can be more— we can improve our operations.

How far do you think this goes in terms of productivity lift in the short term? Meaning, let's just say we have the technology we have and we apply it well. Are we talking a multiple X increase in productivity? Certainly I personally feel that in the coding use case. And what does that ultimately mean for data analysis? Do we end up— the old bank teller thing is sort of the happy story where We introduced ATMs and then, you know, people said, oh, that's the end of the bank teller, but it's not, you know, there's been a lot more bank branches open. Classic example. Is that where we're headed in data work as well?

In all work everywhere. I mean, I think we, on our podcast yesterday, we had Jeremiah Owueng and he, and we were discussing this like, okay, what, Who is not going to be affected in data? I mean, we came to the conclusion that only somebody who literally lives in the middle of nowhere and doesn't talk to any— who doesn't depend on anybody else, because you're part of some supply chain and that supply chain is going to get affected by AI. So I think everybody is going to get affected by AI, going to be more productive and stuff. And then there's just the honest, no BS here is that yes, we're going to have a We have to deal with the job change that's going to happen. There's going to be less jobs and we're probably going to have to figure out UBI. And what we were kind of just talking about is this is probably going to be on the top of presidential campaigns in 8 years from now. Like 8 years from now, this is going to be a deal that this is going to be top of the conversation when we're electing president. There's going to be premiums for dealing with humans, right? Because at the end, everything's going to be so much automated. We're doing, We were playing around with Perplexity yesterday, like we're doing when we're doing our podcast, or Pi AI. And like, this stuff is, it is replacing a lot of the humans. I mean, a lot of stuff that humans are doing. So what we'll see is I will pay a premium knowing that I'm going to interact with a human and not with an AI. So I think everything is going to get touched little by little. Another thing I can say publicly, and hopefully no people will take this the wrong way, is like, I feel bad for people who are doing coding bootcamps because that's just going to be completely commoditized. I mean, writing code is going to be all fully automated. What we're going to need are people, are true algorithmicists, computer scientists, people who are thinking about solving the problem because you're going to come up with the algorithm The step-by-step to solve a problem and the implementation, because you're going to break down the problem into small pieces, then the implementation, the code for that is going to be the product. It's going to be 80%, 90% correct, right? So knowing how to code, but without knowing how to think about problem solving, like think about problem solving like that. And so I think there are all these jobs are going to go, a lot of them are going to go away. I mean, we've seen this again over a generation. I mean, when the typewriter came out, people were against Because it's like, oh, if I write it by pen, I know who wrote it. I can see their handwriting. And like, no, this is offensive because I don't know that people are going to be right typing these things that we don't know who's going to say these things. People, that was the argument back at the beginning. Yeah, well, I don't think that's an argument anymore. So, so all, all these arguments will come in and then we'll go, and I think it's going to move so fast that we're going to see so many changes in our generation.

So how would you anticipate that playing out for an enterprise data team? Obviously it can be generic, but let's say you're a big enterprise and you've got all these systems and you've got however many people. I don't even really have an intuition for what's a typical enterprise data team size and what are the role breakdowns, but how would you expect that to evolve in terms of headcount, which roles kind of change or go away? How much are people then also spending on AI? Like, are they saving, you know, an order of magnitude on their costs? Or is it, you know, does the AI end up being expensive? What's the before and after picture as this transformation happens?

The roles that are going to be, they're going to stick around or are going to be created are the ones that are very people heavy, are the ones that are going to where you need a lot of thought, right? So it's like doing analysis. I mean, think about like a data analyst. It's like, I mean, if you're doing just the generic analyst that like reporting on metrics that are all very well known in a particular industry, like that's going to go away because these large language models or these AI systems will know all the foundational metrics in every single industry. Like that's thought, right?

So Yeah.

But then it's like people are going to be the creative thinkers and they're like, well, what about this that nobody's thought about? Like not even the AI is able to come up with this. I'm going to use the AI to come up with this. So people are really, really critical thinkers who are, who people who can connect the dots from like so many different places. Like, oh look, this is probably something we should go figure out. Like I think that those are the types of roles that will really, really go value. And then you can see that in the analysis of the data and coming up with it. The other one that I'm very focused on is managing the knowledge of your business, right? That's gonna get managed in your— but being able to keep track of that, understand the— yeah, the LLMs, the AI agents are gonna assist me in kind of collecting all that knowledge, but really then figuring out, okay, we are going to make the— decide that the— an active customer means this for these reasons, right? Like, there's going to be people behind all that stuff. So I think when it's all about critical thinking, those are the roles. And then other than that, I think a lot of that stuff is going to get pretty commoditized.

You kind of alluded to creativity, insight, eureka moments, if you will, being one of the things that the AIs don't clearly deliver that we rely on humans for. I very much focus on that as well because It seems like that is a— I think about AI a lot in terms of just threshold effects. When AIs can do something, we can suddenly be in a very different regime from as long as they can't. And having these insight breakthrough eureka moments, it feels like a huge threshold that we have not yet passed, certainly not with any reliability. I would say it's very rare.

This is on my to-do, my to-read list. I've seen a couple of papers and I was like, whoa, This scientific discovery only happened because we use AI. I'm like, that is freaking amazing if that's actually happening. Does that mean that we're actually going to advance science, advance taking faster, taking unknowns and making them known? Like, this is freaking amazing that this happens, but it's not, it didn't do it by itself. Right.

A lot of scaffolding. Yeah. A lot of kind of framework around them is, is definitely still necessary. But yes, I think we're just starting to scrape at the bottom of that threshold right now. DeepMind recently had an interesting one where they advanced the state of the art in some challenging math problems. It's funny, it's like I don't even really understand the problems and the AI is advancing the state of the art. Fun search. Yeah. I think of Eureka, which is an actual name of a paper out of Nvidia where they used GPT-4 to write reward functions for reinforcement learning for robot control of a robot hand. Gabe Gomez, who did the autonomous chemical reaction optimization with GPT-4. So we're starting to see these, you might call it sparks of AGI if you're so inclined. What else though? Eureka moments is a huge one. Is there anything else that you would say, if I'm an OpenAI or a Google DeepMind or an Anthropic. And I'm like, where are they falling short? What are the, what are the kind of persistent weaknesses that you would love to see us address? Do you have a thought on that?

These foundational models don't know anything about my organization. These foundational models, these large language models have, one of all, as a large language model, they're experts in language. They know general knowledge, right? They, they don't know anything about my organization because I've never, they've never looked at it. They never trained on it. The context brain of my organization. And that's brain, that brain is the knowledge graph. Now the thing about the knowledge graph is that it's not a language thing, right? It's not like I can't do natural language product knowledge, but the, the, this knows everything about my organization because I keep track of it. I govern it and I use AI to govern it. I use AI to create it. But like, I, I, there is this thing that I have and this is my precious, this is the essence of what my organization is. It's the brain, right? And everything here is accurate. I can use this to explain things. The LLMs, these foundational models don't have that accuracy, don't have that explainability, don't know my organization. Do we expect these foundational models to know every single organization? No, because These things is private. I don't want them to know, but I want to go use them. So that's why this combination of these foundational models, large language models with your internal brain of organization, which is your knowledge graph, that's what I see where there's a future. And I think what we need to find, what we need to work on is understanding best that integration point between the knowledge graph, the brain of organization, and these foundational models. The naive one is prompt engineering, right? Use some RAG architecture. We're going to expand this. This is what's going to happen, what we're going to be working on in the next couple of years to figure out what that integration is. And then it's going to be a plug and play. You're going to have all these foundational models saying, oh, I can work with your knowledge graph. I can work with your brain, right? Just plug and play. Plug in your brain here and boom, you power it up, right? And then also, I don't want to be, as my company, I've invested so much in my brain, I don't want to get tied just to one when I can probably move around. I mean, it's just like, it's going to get like another cloud, right? But you maybe could get all bought in, right? We use all— everything is— we're an AWS shop, right? We're an Azure shop. I do everything there. That's probably where they go. But people are still multi-cloud, right? So I want to work with different foundational models. So then that's going to happen. So we're going to have these integration points. So I do think that organizations are going to be investing in creating their brain, which is their knowledge graph. You have all these foundational models. And then it's just going to, what we're going to, we're just going to find better ways of plugging and playing. And that's how we're going to, that's the integration points are going to happen. And that's what we're focusing on. That's how I think about it. I don't know. What do you think?

This is like a project that I'm, you know, maybe starting to embark on is kind of a taxonomy of weaknesses of the current systems. And I do think you're hitting on a big one there. That just, I think that ultimately kind of boils down to the fact that in today's systems, there are the weights, the model itself, and then there's the stuff that you're putting into it at runtime. And that is the input, the context, the prompt, but it's limited and there's nothing between those two. I've been really obsessed recently with state space models as something that may enable a much longer and perhaps more coherent, perhaps more agentic, perhaps more human-like. Although I would heavily caveat the human-like because I always try to remember too that these things are pretty alien in critical ways. You can fine-tune a transformer model for behavior, but you end up kind of narrowing that behavior typically And you lose a lot of the generality when you do that. So you're kind of dialing it into a specific task. That's my finding, usually with the fine-tuning that I've done so far. You can't really do something where you're training a transformer model like you onboard a new employee. And so I'm looking for something where I want to be able to give an AI system a big body of knowledge and maybe a bunch of history and here's a bunch of things that we've tried in the past that have worked and some that have failed. And I kind of want to download that knowledge into a system, not necessarily update the weights, but have that encoded in some way where it can be then used to inform the runtime tasks that I want to give it.

And I think state-space models are shaping up to be a If I abstract what you just said is like, you're giving your body of knowledge, things that you know that works, that it doesn't work right now. How are you representing that? Well, text or whatever. I think all these things should be structured right in the graph and the knowledge graph. But basically, but you're giving your knowledge, something you have, and you're going to pass that on to these other foundation models and you're saying, hey, let's go do something together. Right? So yeah. So there's like, that's the thing is like, you have this Foundational models, you have your knowledge, you want to be able to integrate, combine that. And again, how are we going to go do that? Where are those integration points going to be? That's what we're going to go find out. What is even the best one? What does even the best mean? I don't know. Well, we're figuring this out. And then you'll be able to go combine that. And then that's how you're going to deal with the privacy issues. So that's what I find really exciting is like you're combining external and internal things together. And that's what's going to make it super powerful.

You've said a couple times along the way the no BS take on it. And I know that that's a theme of your own podcast. Do you want to tell us where people can hear more from you if they want to go deeper down the data rabbit hole with you? Yeah.

So I, in addition to the other kind of side work I do at DataWorld is together with my co-host, Tim Gassper, who's our Chief Customer Officer, we host Cataloging Cocktails, the honest, no BS, non-salesy data podcast. We've been at it for, I think we're on our 4th year now. So we started season 7. I don't know, I've lost, I think like episode 160 or something like that. We talk about all things enterprise data management, analytics, governance, all this type of stuff. So honest, no BS, because life is too short for BS and drama.

Cool.

I love it.

Well, I'll check that out. I'm sure others will too. For now, Juan Sequeda, principal scientist, head of the AI lab at data.west. world. Thank you for being part of the cognitive revolution.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.