Webinar - Maximizing AI Impact Through Data Quality and Advanced Strategies, with Stefan Wendin

Will you?

Good. So now we're started. So warm welcome to everyone and especially Stefan van Dil, who I met this spring through a shared contact that recommended that we should speak. And it was a really interesting talk around leadership and technology and AI, which is something that you've been working with and with a lot of big organizations helping them. So I'm really looking forward to this talk where you're going to share your experiences. So Stefan, would you like to get started? And I should maybe say that we. If you want to put your questions in the chat during the talk, we will collect them and then we will have Q A at the end.

Yeah, so put your questions in. We all know they're stupid. Question. I love those. I mean we don't have to pretend they aren't. There are going to be many of them and that's the good point. So we should just rather embrace it. So my name is Stefan. I come most recently from a company called Neo4J. From the beginning a Swedish company, but moved to Silicon Valley 10 years ago. A year ago a little bit more. I decided to. To move on. There was too much happening within the space and I was very interested in looking upon knowledge graphs and combination with language models to kind of secure correct answers. But that's not what we're going to talk about today. We're going to talk about I guess a little bit more of a general kind of thing and so on. And the best way to start talking about something is I guess to start sharing and then start talking. We're not going to talk about Dirac equation for you that notice that one. But we're going to talk about this amazing time to be alive. I mean it's insane what's going on, right? If you think about it, AI is literally moving fast. However, there's two things that moves even faster and I would argue that that's stupidity and greed. They are going on a very much higher pace than. We can see this by counting the large number of fake demos. Hello Google. What the hell are you doing? Stop immediately. We also see this self proclaimed LinkedIn AI specialist but we also see a lot of false dichotomies. One of my favorite one is pinning Jan from Mieta versus Elon Musk. I don't know if you have seen them going on at each other at X. It's ridiculous to be honest. I'm like this supposedly to be like some of the smartest people in the world and they behave like in kindergarten. So I think that's interesting. But this is somewhat a Dangerous soup. So very often when we do this, we talk about these things, right? We talk about when will we reach AGI, will AI take our jobs? And then we have this battle of these two giants. And is it just me? But does this feel like a Groundhog Day, anyone? It's just like it's on repeat. It's just going to. We change a word, we find a new trend and we repeat this. So I mean, let's just break it down and see. So we have the CEO of Nvidia which did this quote, which is overly shared, I would argue. So AI is not going to take your jobs. The person who uses AI is going to take your job. You guys agree with that? I would say again, this is a very oversimplification. Of course AI will take a job. It should. Why shouldn't it? It always happened. So, I mean, the reason for him saying so, I would argue is look at the stock. The only thing that seemed to be able to stop Nvidia now would be actually global regulations. And then on the other side we have this so called expert of everything. And I usually say that claiming expertise in AI by only teaching chatgpting, if that's a word, you start to become one at least is like declaring oneself a chef after mastering the microwave generative. AI and ChatGPT specifically is a very tiny part of what's going on in the AI space. And I think if we then double click and look on this, it's very simple to go back to this classical kind of thing. If it sounds too good to be true, then it probably is, right? So I usually think about these questions a lot and I don't really understand why we should be able to explain AI. Isn't the role of AI to comprehend and identify patterns beyond our capability? Why would we want to build a replica about ourselves? Why should we then compete again against ourselves? Should we not build something that complement things that we cannot do? Is that not the whole point of this adventure? I mean, that's why when I set up my teams, I don't pick 10 persons that are like me, I pick different persons. Because then we can have this kind of idea of kind of combined synergies, right? We all know this. If you have 10 of the same, you're just going to have more of the same. We want to have different perspectives. So I would encourage you, instead of trying to explain AI, try to look upon how we change by looking upon technology. And I usually use Excel here, so this is my first thing. So we have the chat, right? So I will ask you very simply, who of you in the chat? A raise of hands so you can just say yes. If you like Excel, you just type yes now, so we can see that. And then let's see. Do you like it or not? Oh, there's some people there. Yeah, cool. And who dislikes it? You type no. It's horrible. Very good. Was it hard for. Sometimes, yeah. We always have a Swedish version in there. Log on version. This was pretty easy, right? It wasn't super hard. You didn't put a lot of effort into this. And I think this is interesting because this is on the level how we should be interested in these type of things. I usually ask, like, is there anyone now that want to sit on a two hour session of the kernel functions of Excel? There's always one person saying yes to this and that's the person that I've noticed never have any budget. So I will ignore that person from now on. If there was one, they say no, they're good. So I think what is interesting here is to understand what it is, but rather how it rewrites us. Because I would argue when you think about data, you think about tables. Am I wrong? I doubt it. Because that's what we do. So we build a tool and the tool in turn rebuilds us, right? So every piece of technology rewrites us. So instead of focusing on AI, focus on you and your behavior, your leadership, your way of thinking. How does that change while using a certain tool? Because then you're controlling the only thing you can control, yourself. Very good. Right. So when we look on these paradigm shifts, they're kind of easy, right? But we tend to argue again and pinpointing a false dichotomy if we should stay in the old and go to the new, and so on. The problem is that this is a twofolded problem, right? So every paradigm builds upon its predecessor insight and cognitive frameworks, right? Our mental models of the past. Right. However, simultaneously introducing an auxiliary viewpoint, frequently encompassing elements previously deemed unattainable. Meaning all of a sudden everything that was true is not not true. But some of the things are just as normal. And this is where it gets really complicated. So the way to understand this a little bit better is to kind of look upon history over time. So I think we should do a super quick research in how to look upon technology and see what we can from it, because I think this will help you not being so stressed about a lot of things. So if we take new technologies and we split them into two, we're going to have an innovative phase. And we're going to have an accelerated phase of those. The innovative phase is super easy to pick. There's some sort of groundbreaking technology that you most likely don't understand. It has zero business model or value. It's super expensive. But the easiest part, to pick it out, it's where people go like this, hahaha. It will never work, it's stupid. Haha. People have been talking about this forever, idiot. Now we are in the innovative phase. Very good. You don't have to care, it will not impact your life. Ignore it. Then you will start going to the accelerated phase. Here we see mass adoption. Think about ChatGPT from the GPT version. When they put that chat window, you have the mass adoption overnight. Everybody almost know what it was. It was insane, right? So we have a democratization of technology in that sense we have a price drop, it was free. Have a hypergrowth phase coming there. And with that hypergrowth we start to panic. It will take our jobs, we start screaming regulate it. Now I mean look at eu. The only place where we have regulation bigger than the innovation. Which is interesting. But I'm not here to talk about EU politics for now. But now you know how to spot them. And then if we look upon history, it's actually kind of fun because this has happened way many more times than I can count. But here are some of my favorite examples. So if we go back to 3000 B.C. we have the kind of dawn of knowledge preservation. So here we have the innovative phase of typing down things. In this case they typed down beer distribution on a clay tablet. Awesome thing, right? So if you think about this, what's the value of this all of a sudden? I know that Mia now owes me four beers. The tricky part here is that we can ask ourselves what do we need to have to keep that value? We need to have that over time, right? So I need to read hieroglyphs. I don't do it. I don't know if you do it Mia. I don't think so. And also storing a clay tablet next to beer which is moist. Very bad, right? But this is also the same type of problem we have nowadays. If we accelerate to a couple of 3,000 years ahead, what we see is this discovery of yours will create forgetfulness in the learners souls because they will not use their memories. They will trust to the external written characters and not remember of themselves. So we have now fast forward to the pen and the parchment roll. We don't need to use the clay tablet, we can just fold it up. And I mean, imagine how many more parchment rolls you can put on your horse when riding to the next village compared to the clay tablet. They're going to break when you try to transport them, right? All of a sudden, I mean, Plato and Socrates, we can consider them kind of okay thinkers. I think there will be no debate about that. So whenever you freaky freak out about AI will take your job, you're in good company. Don't worry. You don't have to worry. So here we have a one to one scenario. I write something, I give it to you, right? The value of that is that I can remember of things. I have a receipt. For that to be true, we need to be able to read it and it should be consistent over time. We go to the next one. One too many. The rise of the printing press. One of my favorite inventions I use often this kind of crazy monk called Luther and how he used the printing press to distribute his new version of Christianity. What he did was he ignored the normal way, bypassing the normal gatekeepers in the sense which is of course the classical church with the long parchment roll or leaflets. They're standing there in Latin. Domino, spiritu maximus, whatever. This is again fake. But the good part here, I guess you don't speak Latin, so you don't know. As long as I keep a straight face, that would sound correct, right? So he did the exact opposite. Opposite. He wrote in plain folksy German, posted it on every single wall, and all of a sudden it was a new form of Christianity in Germany. So if you overlap this on a heat map, it's a one to one rotjo. If they have a printing press, they have that type of Christianity. So what did we learn from this? If I am to write to unknown people, meaning one to many, I need to adapt the message otherwise it won't work. Right? Then we come to the next phase of this, which is just swapping the other way around, which is the knowledge of many to one. In this example we have Ada Lovelace and her analytical Engine. We have millions of examples of this. And this is also almost in the range of our own time, so we will remember them. But I think this kind of crazy teachers here ranting about the calculator is pretty much the same rant we see nowadays. It's exact, oh, it will be stupid, it will make people stupid and so on. So it's like, oh, maybe it's the usage that we need to talk about, but that's another story, right? So then we go to the next one which is this idea of anyone to anyone, or rather many to many. So the idea of Internet is born, or what I call the graph paradigm, right? When something is connected. So you're somewhat old like me, so you remember when Internet sounded something like, you know, that modem kind of weird thing. Sorry for my bad interpretation of the modem here, but I hope you remember it. So the thing there is like, how was it to search on that Internet? Because you didn't really do it. There was altavista, lecos and so on, but it didn't really work because it was a manual process based on keywords. Imagine that there's a person updating Internet in the morning, so your links works. That's the most ridiculous idea I ever heard. Right? It is a ridiculous idea because it doesn't make sense. So even when this was automated, it was still based on keywords, meaning we search for keywords in that sense. What we get back is I said, for blue sky, I get blue shoes, I get blue man group, I get, I don't know, blueberries. What about my skies? Oh, violet sky, red sky, yellow sky, cloudy sky. And that's the problem. It's not kind of working in the way. So what Google did when they invented PageRank in 1996, or Larry Page and Sergey Brin did, I think they weren't calling it really Google at the time, but the foundation of that is that it says that if in this example, B is the biggest circle, right? So if B is important because it has a lot of incoming links and B links to 1, which is C. B can give a lot of its popularity to C, meaning. The classical example of this is, you know me in school, I was a super introverted nerd that liked Star wars and math. You can imagine how popular I was with girls, right? Hence, no, I wasn't. I was sadly bullied, to be honest. And one day, all of a sudden, this super cool girl Sana asked me, oh, hi, Stefan, do you want to walk home together? And I'm like, okay. So now she will lure me in the forest and the cool gang of the school will beat me up. Now we all know the story, right? So I'm just. I'm so tired by now, so I'm just. Just get it over with. We walk into the forest, just beat me up. It never happened. And I'm like, what's going on now? Then the other day we walked to school together, and then we did that a couple of times. And all of a sudden, I am the next most popular person in school. I Didn't change anything. However, her popularity saying I was cool made me also cool. Not as cool as her, but almost. So this is what they did to Internet, meaning we start to see paths, we start to see clusters, we start to see centralities. So this is an algorithm sitting within the centrality family, meaning it gives us importance of things rather than just keywords. So super interesting to think about that. And then we come today. How many of you in the chat you can type? Yes, I did read at 2017, attention is all you need. I guess most people did not read it. There was very few reading that paper, right? Most have not heard about it until ChatGPT happened because it was just a stupid idea, this will never work. And so on and so on, right? And then we ended the accelerated phase. We all remember what happened, right? It was total shit show what's happened and, and remember how bad ChatGPT was in the beginning. It was lying all the time. It was worse than my drunk uncle, actually. But the thing is, we're living in this right now. So if we take this and zoom out back, what we have learned now is rather a story here about what's valuable within data, right? So all of these are still equally valuable, right? We need to store it in a proper manner. We need to write it in something we can read. It needs to be accessible to many. We need to adapt the data so people can actually read it so it's not totally obscure. We need to enrich it with relationship because relationships between data points is equally important as data points in itself. And then we need to feature engineer this into what we have so we can say on this scale, where are we? And I mean like I work with companies, most of them, they say they have a data lake and it's all in order, but not really. So I think we have problems with something which is 5,000 years old. So we maybe don't need to focus super much on the AI part here. Maybe it's data engineering. We should focus on slightly the thing here though, however, why things feels like they're going fast. It's not that we invent things faster. I think what's actually happened is the exponential decay of mass adoption. We have the mass adoption phase, right? So if you take the parchment roll and the clay tablet, it's about 3,000 years, if you take attention is all you need. The first paper Google sent out with the transformer model which powered your GPT to now it's five years. So we went from 3,000 years to five years for mass adoption. That's what's hurting. It's not that it's the change of innovation, it's the mass adoption that's the problem. Because you don't care about the innovation, I would argue, because most of it we don't hear about. So it's the mass adoption that's the problem. So if you think of this and think about all of those paradigms and then we look upon this, that most companies will unlock value with their language model projects if they can successfully connect it to their own data. You remember this guy? How many have clean accessible data distributed for everyone, have a sandbox up and running? Uh oh, it's not even this guy, it's rather this version. This is fine. Blasting, right? Very good. It can only get better. Since we're total chaos, we can only get better, right? So the idea here is that very often we talk about this, the perception of data is super good, but then we start to look at the actual data and very often it turns into a black dark hole. Maybe we didn't have those things in order. So since the value of your AI initiative is directly proportional to the quality of your data, we need to be mindful of these things. Right? So I think this is one of my favorite releases from one of the employees from OpenAI where he states that trained on the same data set for long enough, pretty much every model with enough weights and training time converges to the same point. This is also saying enough money because you need to pay for this to run. Right? This implies that model behavior is not determined by architecture, hyperparameters or optimizer choices. It's determined by your dataset and nothing else. So there's very little happening on the actual technology invention. Most of what's happening now is on the data side, which is pretty interesting. We have a pretty much the same architecture on most of these big models. So they have all converged into one thing. Right, but what is the secret sauce here is data. So to the point, I would argue that data is your only competitive advantage. So if you don't know what you should be doing with your data, start cleaning it, start structuring, start understanding what are the questions I'm asking, how do I reason when I get to those, and so on. Right? So I would argue that this means that the quality, governance and compliance are going to be the most trendiest jobs going forward. So we're going to actually see more jobs in this domain for sure. So we are in a verge of a change in how data and infrastructure will be perceived. But Then to the good point, cannot a machine do this? Or when will we reach the AGI phase? I don't know. This is one of the most annoying question. I keep getting this. So if anyone planning to put it in the chat, don't put it in the chat. I will log out completely and I will never speak to me again. I'm joking, you can put it in there. But is this really the question, when does a machine have equal intelligence? And what are you? No. And the problem is rather why are you acting like robots? Because take this example, we have a teacher that leverages AI to create engaging and challenging question for the students. The students in turn employ AI to powered tools to research and formulate well structured answers. The teacher then gets very tired and uses AI to grade these. Should we do another example? An individual receives a lengthy email. How many of you have done this? With a very long complex jargon? And then you like put it in there, give me a summary of what this is, what is it? Write the reply, and so on. Sorry. And that's equally true. And we have the HR system, it's the same thing. So the question here really is not when we'll reach AGI, but understand why we acting as an API for AI. Because what we do is we take some thing, we put it into another AI and then we take that thing and put it into another one, right? And if this is already happening, instead of bashing on it, what can we learn from it? Right? So what can we learn from this? So if you think about people, what, what we are extremely good at is that we are very good in communication. So to the fact that there's a couple of species and every single time when we have that capability of being hypersocial, we always take over things, right? This is true for ants, for example. There's a specific ant that invade all the different other ant sorts or species and then have some that they eat and some that they use for working slaves and some that they use for, I don't know, soldiers. They are not the strongest, they are the one that are able to communicate differently. So with that said, we actually have an understanding of modern systems because they are much more alike to people than we think. So imagine you have your office, you have a nice building with a lot of people sitting. Is this value? No, this is a cost. It's a potential value if they start working. But there is nothing happening here, right? People working equals value. People working in synergy equals even more value. The same is for data. If you just store your data it's not going to have anything else than a cost. So data working is equals value and data working in synergy with other data. This is when we get to the real value. And with that said, I was thinking we should look into kind of AI agents and how you can actually think about your different tools as maybe your co workers. Right. Because your coworker doesn't necessarily have to be a person, I would argue. So the reason for this is very simple. So if we take a non agentic workflow or what's going to be called as zero shot in chatgpting, you know when you ask a question and then you get a reply back. So that would be like please type out an essay on topic X from start to finish in one go without using backspace. Okay. Would you write an essay like this? Can I see if anybody that has written an essay approach it like this? I will now write an essay about AI. I start now and then I'm done. Done? Of course not. You do a draft, you talk to people, you do some research, you bounce back and forth with revisions. So this is a workflow that we can mimic. The reason for mimicking this is that we can actually create a lot more accuracy. So here it's some benchmark on kind of zero shots with 3.5 and GPT4, the turbo version. So if we look at this one, we see that the zero shot has 48% accuracy. So every other one is drunk. Every other answer is a lie. Not super impressive. Right? It's super cheap though. And we look at GPT4, it has a 67% accuracy, which is fairly good. However, if we then use a couple of tools, which is like reflection tool, use planning or a multi agent, we quickly see that these combinations then give us almost 100% accuracy. So it means that you can run GPT 3.5, which costs nothing by the way, and still get much better result than you can with the GPT4, which is pretty cool. So I thought I should spend a little bit time on this. There might be some hard parts here. If it's too hard, just let me know and I'll go slower. So we're going to talk about reflection tool use planning and then combining them into one kind of office of random AIs. Right? So for example, for a coding task here we have a left and right sided brain. One is the one writing the code and the other one is the one critiquing the code. So what we're going to do here is so write the code for this and Then the right side here says, there's a bug on line 5. Fix it by XYZ and then write a new version. It failed. Unit test 3, try changing XYZ. This is exactly how you would work, right? So if you write code, this is you going to GitHub and stealing other things or pinging your senior friend or some other person's like, what the hell should I do here? So this is a very interesting way to increasing already your accuracy. I think you should implement this already. When you get a reply, ask it to be a critic of its own work and see for improvements and add those and figure out how to kind of tune that part. What you can also do is you can split them apart into different one. If you, for example, use GPTs in the OpenAI suite, you can have one which is like the writer and one is the reviewer, for example, or the code critic. So split them and give them similar ones. This is actually easier to handle for a lot of people. And if you don't know this, an easy way in the window is that you can actually call on your GPTs by just using the so. And then we'll come up all of the ones that you have bookmarked, so you can call on yourselves and others and then combine them in one flow. The other one is tools. What happens if we ask a language model which is based on similarities to compute 1 +1, which number is most similar to 1 1, 11. Is that good math? No, that's horrible math. Right? So what we often do is then that we build tools using other things and then we use something which is function calling or a tool calling, meaning the language model calls for a specific tool. If I want to do the Dirac equation here, for example, I can set that up and I run that in Python and then I will validate my math that actually the math is mathing, as you say, and so on. So that's also one way of thinking about it, right? Instead of seeing it as a holy grail of something black box, think about small parts. One is doing the ocrs of text, one is searching the Internet. So it's almost like you have Santa's Little Helper there. It's like a version of small things that can do different tasks, right? Because with that we can then start planning. So if we take this fairly stupid example which has this request, generate an image where a girl is reading a book and her pose is the same as the boy in the image, then please describe the new image with your voice. So we have a picture we use open pose to do a pose determination. We get the result, we put that pose determination into the next model where we do a pose to image. So we generate now an image with that pose. Now we use image to text. So now we have a text, a girl sitting on a bed reading a book, and now we use text to speech. So here we use four different models to do something that seems impossible to do. The fun part here is that this is super simple for you to do. So when you were being an API for AI before, when I ranted about this in the teacher and the students example, you're already doing this, so why don't you do it in a planned way? That's what I'm saying, right? So plan for these things, lift it between systems and start to create those, because that will teach you how to think architecturally about these systems. And I think that's the real benefit for you. Which then gives us to this kind of multi agent kind of debate where you literally have some sort of chat dev in the middle and then you have some sort of designing department, coding department, testing department, and then a documenting department. They're all assigned to different type of tasks and they can all use different type of tools and models to do them, but then you plan them accordingly. And this is how you build a system. So this is literally how you build an office with people and competences. There's very little difference here. Right? So with that said, this was all dandy. But there is a caveat here. Very often what we look upon this is the kind of accuracy versus cost. We can fast forward, this is a little bit, this is just describing what we already talked about. But what we see is that you don't get so much bang for the buck for a lot of these things. So some of them are better than others. I mean, they're delivering good results, all of them amazing results. But it's also about how much you pay for it. If you are to do this on an enterprise level, for individual level, use them all, because you're not paying. But if you're building an enterprise system for let's say 40,000 people, if it costs 2 cents or $5 is of a very big importance. Right? And then when all of this was all dandy and short, we saw this cool girl presenting the OMNI model from GPT4. So it's 2 times faster, 50% cheaper, and 5 times higher rate limits compared to GPT4 Turbo. What is interesting here is that when we look upon this from structure perspective and having the cost Classes and the accuracy performance classes, we see that GPT have a super high rating on how good it is. That's a benchmark there, the blue one. And then we look upon the possibility of tokens per second, meaning how much it can output. Also very good, but it's still pretty expensive. Right. So with that said, what we can do is we can actually combine this. So I did it for you to figure out which model you should use and if it was any doubt, GPT4OMNI comes out as a clear winner. It's the most bang for bug compared to accuracy and that. So I think that is one of the things that seemingly good. It's yet not available, I think in Europe, if you're doing this on an enterprise scale and so on. Another thing which we can actually look upon is this. I don't know if you saw this, but this is kind of a benchmarking thing. And all of a sudden there was this kind of new model showing up. I'm also a good GPT chatbot and we're like, what the hell is this one crushing all of the benchmark? And this happened to be the Omni model. Right. So it's a lot better than all the other ones. However, why on earth would OpenAI give out the best model in the world, available free for all users? Why? The same reason that Apple is now putting OpenAI in there, or they want to put it in there. Right. This is two of the funny memes that I found on this topic. Because if it sounds too good to be true, it probably is. Right. So the secret sauce in this omnimodel is that it does post training, meaning that what it does is that it actually alters what it already have been trained on. So this is something that we work a lot with. So instead of trying to add more data, meaning other books, other PDFs, other things, what we look is rather that we look upon these kind of feedback loops of how people use it. So the real value emerges as many people use it and observing their usage over time. And those patterns. Sorry, yeah, the pollen got me there. It. I'm trying to keep away from it, but it was 36 minutes in and that was very hard. So I'm back again. So observing their usage patterns allows for continuous improvement over time. Right. Which then gives us these kind of agentic feedback loops that then can help us organize and restructure data outputs over time. So now we have done this, so now we come to you and how you can act on this. So very often we can see that we have some sort of event, we have this kind of event horizon, not the physical one that we're going to talk about. So in the beginning we have very little information, at the end we have a lot of information, but the room for action because everybody else have more information decreases. Right. So there is a sweet spot in the middle of the art of knowing enough and you need to figure this one out. If you're a leader of today, I would argue that this is what you need to train on. How can we be better of knowing enough and find the speed to act, finding your agile workflows, understanding the feedback loops and so on. Because the interesting part here, I mean I come from a startup scale up world. What we strive under is high uncertainty, meaning the only people that work on a scale up are crazy people that thrives in high uncertainty conditions. However, if you look on any enterprise, it's built on quarterly report, it's built on high predictability. You shouldn't come in too high, you shouldn't come in too low because that's going to make it very wobbly and people not going to like it. So this is actually the biggest challenge for modern enterprise companies of today. What we need to do is go from having a lot of people that strive under high predictability to find ways of working under high uncertainty and combine these. This is the number one challenge. It's not about AI, it's about leadership, it's about building organization, it's about working together, using technology. Right. Because innovation should be about strategy and execution, not just a kindergarten for adults where they sit and play with, you know, post it notes and these kind of stupid things. But with that said, I think I have come to the end of this and I'm exhausted. What an amazing time to be alive. So that's me. I prepare some questions if you want. If we have any questions, we can also do those. But let me know what you want to do. I guess you can unmute now and stop the recording so people dare to speak.

I will stop the recording or yes.