#226 Creating Custom LLMs with Vincent Granville, Founder, CEO & Chief Al Scientist at GenAltechLab.com

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com well, we are turning points where there's a lot of old stuff like Google search that are being revisited and can be done much more efficiently. Search is going to be powered by RAG LLMs. It's the end of PageRank.

Welcome to Data Framed. This is Richie between GPT, Claude, Gemini, Llama and all the rest. There are a lot of powerful large language models available off the shelf, and yet many organizations decide that they need to create their own custom LLM. Today, we're going to look into when you might want to do that, how you go about it, and what the buy versus build tradeoffs are. Our guest is is Vincent Granville. He's the founder and CEO of the research lab Genai Tech Lab, and he's perhaps most famous as the founder of the data news platform Data Science Central. Vincent's a prolific author, having written six books on data and AI in the last three years, most recently the State of the ART in gen AI and LLMs and statistical optimization for generative AI and machine learning. Vincent was previously the founder and chief AI scientist at ML Techniques and he's worked for Visa, Wells Fargo, eBay and Microsoft. In short, he's doing cutting edge AI research. He's a serial builder of businesses and products and he likes to write books like they're going out of fashion. I'm very keen to pick his brains on custom LLMs. Hi Vincent, welcome to the show.

Hi Richie. Thank you very much for the invitation and very happy to share what I've been doing with my LLM technology.

Wonderful. So I'd like to know why you might want to use a custom LLM when there are so many sort of standard LLMs out there.

Yeah, that's a good question. Initially, when I started to design my own technology, the origins is I was looking for answers using Google search, OpenAI, GPT search boxes on various websites, websites like Wikipedia or WolframAlpha, and I noticed that a lot of the results that I was getting were targeted to non expert to what I would call the common man. So it would be difficult to find specialized references or focus on topics that are more technical. It really required either multiple prompts if using a tool like GPT or quite some time by using all the search tools, Google or Bing or these search engines. So that was the what started the idea of creating a LLM from scratch when the other part was that most of these tools they use neural networks. The training can be difficult, takes time, lots of gpu. I've read also that they're using billions of weights or tokens. That needs to be optimized during the training. In particular, there can be latency issues. And I was wondering if it's possible to design something that run much faster and that potentially could offer better results. And by better, I'm not saying here that my LLM is better compared to everything else. It's better in specific contexts like for instance, professional busy users who want concise, precise, advanced answers to their prompts. So that's what I focused on initially, actually I crawled Wolfram corpus. So Wolfram is probably one of the best website that covers all mathematics you can find on the Internet. And so even without neural network, actually. And I was surprised to I was getting answers to my question that were to me much more useful than I would get from any of the sources, even from the Wolfram search box on its website. Or even if I do a Google search and I ask Google, I'm looking for results only on Wolfram. I got much better results from what I've been doing over time I got a few Fortune 100 customers that were interested in what I was doing. And then it became more and more clear about the benefits of my rather simple architecture. So just to put some context, initially I focused on a specialized simple LLM that would focus on one top category, which was in that case statistics and probability was just focus on that, which is one of the 15 top categories on the Wolfram website. The Wolfram website covers all the mathematics on computer sciences as well. It's not that comprehensive on statistics need to be augmented. But that's something we can talk about later. So it started with one I would call sub LLM. And then for instance in the Wolfram website, like I said, It's 15 categories. Calculus is another one, number theory is another one. So you could create 15 different sub LLMs then have a system on top of it that managed all the LLMs. So one of my clients actually invented or coined the word LLM router to manage those LLMs. You might have heard words like mixture of experts, multi agent systems that are related to this idea of managing these different LLMs. Also what I wanted to do is not just return very wordy long English sentences to the user, but concise bullet points, different sections. One section would be URLs, links or references. Another section would be related concepts. Another section could be synonyms or definitions, stuff like that. And for each of them offering the user a relevance score for each of the item being turned so that by allowing the user to play with some of the parameters, they could see how it impacts the relevancy score. What's the order in which things are returned? Up to the point that two different users with the same prompt could get two different results if they use different sets of hyperparameters. But to get back to these clients, ABI actually is one of them. So Inbeth Brewery company in Brazil, which is a very big company, some of the concerns were we want local implementation. We are concerned about security, data leakage and stuff like that. So that was an important thing for them. Another thing was latency then. Although the issue was if there is some hallucinations or stuff like that, or poor results or wrong answers, it can be a liability for them. It could end up in being sued or stuff like that. If decisions are made on wrong answers, I mean it's a liability. And the ease of implementation because there's no neural network, when you test it, it's much faster to enhance the system, to test new features, stuff like that. That's something they were also interested in. We mentioned other things, but that's what the main arguments that remember what's interesting also is. So I've been improving and developing these. I call it XLLM for Extreme LLM very fast. Because of that there's many features that I've added over time and I've seen many other. We call it competitors if you want to. But OpenAI meta with llama and stuff like that. Databricks, they started to kind of copy my idea. I'm not saying that they read my documents and then they just copied soon. Yes, and probably they invented on Vern as well. But I was happy to see that they do what I do like with three months delay or something like that and kind of validate what I'm doing. Was very happy with that. For instance, you might have heard the concept of multi token invented the word like three months before I saw it used by meta mixture of experts. I didn't invent that word, but the concept of a Multi LLM with a number of sub LLM essentially and each of them specialized like an expert on some topic is turn out to be called multi expert LLM by some of the companies. Just to give you an example, it.

Really seems like there are sort of two motivations then for customizing LLMs or creating your own custom LLM. So the first one you mentioned was in terms of changing the style of the output to suit different audiences. So for example, you said that you wanted more technical information and a lot of like the sort of foundational LLMs they will talk to the common user. So having that sort of technical language seems like quite an important use case and secondly getting that domain specific knowledge. So in this case you said you wanted information about the stats and maths.

Yeah, by technical it does not necessarily mean technical. For instance, the corporate clients, what they are interested in is to deal with the corpus a company like inbeth, they have some information that employees are accessing regularly. So they have a lot of information regarding hr, marketing, finance, sales, tech and stuff like that. They're interested in a sub LLM for each of them. And we can assume that the user who is asking for something specific, trying to retrieve document tables, images and stuff like that is not really a beginner. So it's not the common man. The user typically in this case so far would be an employee of inbeth. So he knows what he's looking for and expect to get answers that are not basic, that are going to be really valuable to him or her. The other issue was the cost as well. So because those like GPT4, stuff like that, that are based on deep neural networks, they use a lot of GP billions of tokens. And I believe they've not been designed to serve needs like that. Like for corporate clients, they might have been designed to replace a Google search or something like that. So they also charge by the the token and they have a large number of tokens. There's an ecosystem around this that everybody's happy, you charge by the tokens, tons of weights. Company like Nvidia also making lots of money thanks to that. And I might be one of the few that went into an opposite direction and want to make the things as inexpensive as possible without jeopardizing the quality, actually even delivering better quality on specific topics. One feature in the XLLM architecture is that in addition to the prompt you can have another box where the user can specify which sub LLM he wants to query. And by sub LLM I mean category, top categories. Are you looking for information about hr or are you looking for stuff about legal? And sometimes some stuff that the user is looking in. One sub LLM is going to be found in another sub LLM. Like for instance, I'm talking about the statistics and probability sub LLM, which is a mathematical flavor because Wolfram is a mathematical website. If you look for something like gradient descent in the statistics and probability sub LLM, you're going to find little, maybe nothing. But there's plenty of stuff like that on the calculus LLM. So that's where the LLM router is here to tell the user, even explicitly or not, that he needs to get that stuff from a different sub LLM. And one of the things that the corporate client was also interested in. So you can have different types of LLM based on the content, but you can have also different types of LLM not based on the content, but the content might be identical, but based on the type of information you want to provide to the user. Is it someone who is looking for very highly technical stuff or more like definition? So the flavor, to put it this way, the flavor of the LLM could result in different LLMs serving different goals. You might have heard words like actions in that context. It's being used by some of the folks. One thing that I would like also to talk about quickly at some point, I don't know if we have the time that I want to mention it here. So one of the key feature of XLLM is that it's based on taxonomy's knowledge graph from the very beginning. So it's kind of the bottom layer. While these days you hear knowledge graphs is becoming popular, everybody's starting to, but they are putting it like on a top layer. They're trying to build a knowledge graph. In this case the taxonomy, the knowledge graph could be related concept, it's embedded into the website or the corpus itself. For corporate corpus, you have a breadcrumb. You know where you are in the hierarchy of the categories. You also have related concept on the web pages. You look for something, it's going to point to some of the links. So you have all this taxonomy and knowledge graph actually there on the website. You can just simply crawl it. So you may say it's kind of cheating in this case. I've not built a knowledge graph from scratch. You can do it. I don't have the time to go into the details like that. You can augment the taxonomy. The taxonomy can be poor. Some part of the taxonomy can be poor. You can improve that. But for instance, in the case of Wolfram you had a pretty decent taxonomy that you can really crawl. And one of the reasons other folks are not doing it is that if you use tools like Python Library, Beautiful Soup, for instance, it may not be obvious how you're going to retrieve those taxonomy elements, but they're there. It requires a little bit of one or two hours of work to find how am I going to detect these pieces of information and rebuild the taxonomy. But it can be done. And what's interesting is that all the major corpuses that I've looked at, corporate corpus, Wolfram, Wikipedia, they do have a strong structure. So you always hear this unstructured text. Actually it's pretty well structured and you can recover that structure. In the case of Wikipedia, it's very similar in all these corpuses. In the case of Wikipedia, for instance, machine learning would be a sub LLM. The taxonomy is good and there are some things that you can ignore in a statistics to the contrary. So you can browse Wikipedia. What I'm saying here, you can browse Wikipedia by looking at kind of a top page that tells you what are the categories, subcategories and stuff like that. And then from these top pages as a user, as a human being, you can go down and so that's what I'm talking by the taxonomy that you can recover from the website and on Wikipedia the statistics Statistical science taxonomy is pretty cool. It's better in Wolfram Alpha there benefits on Wikipedia in the sense that even though it is a poor taxonomy for statistical sciences, the content is pretty good. It's more comprehensive than on Wolfram. But even for Wolfram is a better taxonomy. That's where I probably don't have the time to go into the details. But in the end you're going to end up not only with content augmentation, but also taxonomy augmentation or knowledge graph augmentation. If by any chance you're working with a corporate that don't have these features, you can use some external taxonomy.

It sounds like there were two really interesting architectural decisions you made there in terms of creating your own LLM. So the first one was around using this mixture of experts approach where you've got a two stage approach to the LLM. So you've got the first layer just chooses which of the experts are going to give good results. Then you've got these experts which are smaller LLMs which are going to provide the actual response. And then the second one was the use of knowledge graphs. So I'd love to get into both of these in maybe a bit more Detail, but we'll cover the mixture of experts first. So this is the approach that you mentioned was has been championed by Mistral and I believe I've heard rumors that GPT4 from OpenAI is also based on a mixture of experts approach. Can you talk me through how you decide what each of the individual experts should be?

Yeah, so for corporate corpus we're working on that right now. But the idea would be that you have pretty well separated component. I mean they overlap of course, but you know, like hr, legal, marketing, sales, you know, all the things that you find in corporations. And you can have also subcomponents. We're not there yet, but you can imagine an LLM made of multiple sub LLMs and sub LLMs, a sub LLMs, a tree like that. I don't think we'll get to that point. Too much granularity in my opinion. The risk is that those sub LLMs they are too small. It brings this kind of an optimal size too big, you get hallucinations potentially you have billions of weights or tokens. And what I've been I found myself because at some point I created multi tokens which are tokens instead of data science. The word data science is just one token and then what I call contextual token. Like data science, the two tokens may not be adjacent, they could be found in the same paragraph, but not adjacent. But when you start working with these multi token or contextual tokens, you can have an exclusion in the number of tokens. Even in my much smaller system, what I found in the end is that very few, typically I'm going to be having maybe like 5000 tokens for a sub LLM if I don't pay attention with these multi tokens, it can grow to 10 million, something like that. And what I found is that most of them are never fetched from the backend tables when you serve a user query. So they kind of noise useless and even worse than that, in case they are actually fetched, most of it is noise or garbage, it can corrupt a little bit the answer that you're going to provide to the user. So having a big token or wait tables can cause problems. Anything. Tables that are too small are also going to cause problems. Some kind of an optimum somewhere.

I find that approach really interesting in that it mirrors the way you would ask your colleagues for help. So if you think, okay, I've got a problem, I want to ask one of my colleagues what the answer is. You think what? Well, who in my organization is going to be the best person to answer it and then you ask that person. So in a way this sort of mixture of expert approach is sort of, it's mimicking that. So you decide which LLM shall I or which LLM expert should I call and then that expert gives you the answer. And it sounds like in a corporate situation each of these smaller expert LLMs is going to represent maybe like the head of department for that particular team. She said you had like a marketing expert, a sales expert, HR expert and so forth. So the second idea you mentioned was around the use of knowledge graphs. So my understanding of these is that it's very helpful for letting the AI understand relationships between different objects. Can you talk me through how that knowledge graph interacts with the neural network part of the LLM? How do they fit together?

Knowledge graph? There are two things about them. First, they are made of high quality words, high level keywords that may be made of two or three tokens but it's not like written text or it's high quality. Sometimes it's actually it's been created by humans like all these categories on the world from alpha they've been made by. They might be like 5,000 of them, probably more than 5,000 categories of categories and all from and then related concept which is the knowledge graph. They've been created by human beings. So they're supposed to be on high quality and they consist of word tokens that are very clean of superior quality compared to the words that you're going to find in text in the summary words. So if you use them also in your embeddings and you put higher weights on these knowledge graphics words, you're going to in the end get better results in the output that you show. You're able to also provide to the user in the answer related concepts. So it's much more easy. So that was one of the things is so to use them for in the embeddings and you may put the higher weights on the tokens that are coming from the categories or knowledge graph as opposed to the ones that are coming from straight text. Then the other thing is you have all these relationship that and provide. So some users are going to want references or related concepts and that's where these knowledge graph can be particularly useful.

So I think just to make sure that I've understood this correctly, the idea is that because the knowledge graph shows relationships between different concepts, it's then going to link to different related bits of text. So if you ask a question about one thing it can if there's some text that is in some part of the knowledge graph, then it's also going to return ideas that are in adjacent nodes in the graph. Is that about right?

It is the ability to jump from node to sub nodes to sub sub nodes up or down, higher level, lower level, parallel. The risk is you can return a tremendous amount of information to the user, even for the answers are extremely concise. Like bullet items you can have. I mean if you are asking to go two level deeper, two level above and stuff like that, you can end up with massive output return to the user. It could be of some valuable in my case actually I allow for that and you can save the results as text file because the output can be pretty big. But the challenge is to find some kind of optimum maybe to stick to the closest possible items in the knowledge graph. And everything here comes with a number, not necessarily an integer number, a score which a real number positive number typically or maybe normalized between 0 and 1. And that score is going to tell you the relevancy between some original piece in your prompt. So your prompt might have multiple pieces, there may be multiple embeddings that you can derive from your prompt which by itself also if you have stuff unrelated concept, you're asking for two different concepts in your queries. Be better to separate them. But then you need to be careful about the amount of information you're going to return to the user because of the knowledge graph. Knowledge graph, essentially it's global context. From one query you can return the whole thing. You know, it's like human beings. You are connected to any other human being on earth with a six link connection. You probably have heard about the story. We all six links apart from anybody else on earth. So. And it's a problem.

Okay, that's interesting because I think one of the most common complaints with these large language models is you give them a simple question and they just return paragraphs and paragraphs of text and you think well just answer that in the first sentence. It wasn't. I didn't ask for all that. So I can certainly see how making use of these knowledge graphs, you could end up with way too much text. And then that's going to give those really verbose responses.

Yeah, so with knowledge Graph, one of these few when that can cause to return tons of output is if you focus on individual tokens, you look at all the information attached to data science, you return everything that's related to data, you return everything that's related to science, but you should focus on what's related to data science. Here I'm providing a simple example, but you get the idea. The problem with sub LLMs that are pretty small for some keywords, you are going to have information from one individual token, but not necessary for the join token. It's probably that's one of the reason for the race for the biggest possible LLMs with a massive amount of input, sources and weights is to be able to in your database to have word consists of three or four tokens with enough information about them to return useful information. Another way to do that is to have a specialized LLM like an expert LLM on some topic and then have a bunch of them. Like I like to say the statistics and probabilities of LLM do a 15, something like 15 of them. You cover everything, mathematics and here the focus is knowledge management. I'm not trying to produce code or other stuff like that. It's more like it's actually very useful as a search. And then with 2000 of them you essentially covered the entire.

Okay, wow. So actually you're mentioning that you can consider the phrase data science as a single token. Suddenly something's clicked in my brain because the recent announcements from Messa about Llama 3 and from OpenAI about GPT 4, oh, they both mentioned the idea that they're doing embeddings more efficiently, where the number of words you put in corresponds to fewer tokens than before. And I hadn't really understood why that was the case, but it sounds like if you've got data science being one token rather than two, then that becomes more efficient in the model and then it's going to be cheaper for people to use as well. Because I think particularly with OpenAI they charge like per token of input, per token of output. So you're going to get more relevant results because it's considering the phrase rather than two words separately and it's going to be a bit cheaper for you. That sort of all makes sense now. All right, so maybe we'll move on from some of the technical talk to about how you how and why you might want to get started with the custom LLM. So what sort of organizations are you seeing make use of custom LLMs?

Yeah, so the one that they've been most interested in what I am doing, they're going to be clients as opposed to companies that design AI engine AI. So I mentioned Inbeth is one of them. Morgan Stanley is another one that showed interest and it was actually the same thing. They were interested in managing their corpus and returning meaningful users much better than what they already have. In place because they already have something in place. So two things now that I'm thinking about that you mentioned efficient embeddings. So multi tokens is part of it. Being careful to not make the number of tokens explode by using multi tokens, that's a risk. And one of the features in XL Dynam is variable length embeddings. So I did not invent the concept. Not many people talk about that. It might not be that friendly if you use vector databases which assume a fixed length for your embedding. But when you use variable length embedding, they can be much shorter. If on average they can be shorter. So it's going to be faster as opposed to. Because you know those fixed length embedding in big system they might 8,000 tokens. So yeah, in each embedding is something that's typical. Then another component. I've heard some people talking about this idea that not that many yet is the prompts. So to use the prompt is input to augment data. Put this way, so you have millions of queries, you can analyze them. And so there are going to be multiple interesting things coming from it. You're going to see what the user are looking for. This might help you refine your taxonomy, find stuff. Oh, this is. A lot of users are looking for this step. This step is not in the corpus. So maybe we need to augment the corpus with external sources that answer. Maybe we need to create new pages on the corpus to answer these questions for the users. So that's another interesting component. And then the other component that some people are also talking about it, you might have heard the word is caching mechanism. So you look at the prompts, what are the typical embeddings? What are the top 1000 or top 5000 embeddings that can be derived from the prompts and you just put them in a caching. There's a caching mechanism. So in real time, when you deliver these results in real time to a lot of users, you create embeddings derived from the user prompt and then you need to merge them and to find those that are close to those coming from the user prompts in the backend tables. That can take time, especially if these tables are immense. So caching mechanism can help as well. Variable length embedding can also increase speed. So the way I've been doing this with variable length embedding is nested hash is a dictionary in Python. So it's key value database. But the value is the hash itself can go two or Three level downs. The key can be hash itself. It's very efficient to handle sparse data. What I mean by sparse is if you have a million tokens or a billion tokens, and the association between one token and other token, for the immense majority, there's no association. You pick up one token at random and another token at random, there's no association. This is what I mean by sparse. And tokens that are really connected somehow and associated in this huge list of tokens represent a minority. To put it in a different way, let's say you have 1 billion tokens. You want to create a table of associations between tokens. You'd say 1 billion by 1 billion would be the dimension of the table, but in reality they might be on the 3 billions relationship that exists in the corpus. So those nested hashes, it's an efficient way to deal with it. It's a format that's a little bit similar to JSON, but we can also use graph databases to represent that kind of structure.

Okay, a lot to go there. So let me see if we've got this. So it sounds like the most important use case then, or the most common use case you've seen for the custom LLMs is for corporations wanting to have some sort of Q and A for their own corporate knowledge system. It seems like managing what you know as an organization is incredibly tricky. And I like that you mentioned there's a sort of feedback there where if the AI gives poor results, then it's probably going to tell you you haven't got documentation on whatever it is people want to know about.

Yeah, and with the. With a type of documentation which could be people looking for definitions, people looking for case studies, for examples for code. Not here that my xlm, they're not generating code, but they're going to retrieve code that the user are looking for. And then it's becoming quite odd these days, search in general, especially search for advanced search or search for experts essentially replacing Google. You might have heard that OpenAI, I don't know where they are right now, but they seem to be interested in fighting, going to war against Google for search and even myself. If you look, there's a sample of my output and I can share later. It's everywhere anyway, it's easy to find. But the first section that I return to the user, not for the corporate client, but in the kind of research section is URLs and the title is organic URLs. So what it means, have in mind that this can react. Revenue could come from sponsored URLs and I actually do have a client very interested in this idea. So that's interesting.

Absolutely. There would be an interesting economic model where you have paid or promoted search links within the AI response. I'm not sure people want that, but I suppose the economics of.

Yeah, there would be different layers. There would be one with pretty much everybody who's doing tech these days. You have this free version that comes with advertising and you might have a paid version come with a SDK or you get full access to the backend tables, the embedding and stuff like that, it gets updated more frequently. So if you pay just the way it works right now, but I wouldn't charge by the token, I want to avoid that.

And I suppose thinking about it more like if I was like shopping for new fancy shirts, then if I'm asking the AI what to buy, then having some links to the actual products would also be helpful rather than it just naming some brands and then yeah, I'm.

Not going into that direction. I will leave that kind of stuff to Google and the other folks. It would be more. If you're looking for more like scientific or topics in biology, medicine, computer science, I'm not going to, for instance, focus that's not the plan on looking for the best restaurant around your place and stuff like that. So it's more like knowledge essentially as opposed to anything else.

Now actually, while we're talking about the economics of this, so a lot of the technical innovations that you mentioned, so things like the variable length encoding, caching, more efficient use of sparse data, these all sort of point to the fact that generative AI is still quite an expensive technology like both for training models yourself and also for running these large language models is quite computationally expensive. So can you just maybe give me an overview of where these big costs come in? Like what are the most expensive parts of working with generative AI?

Well, because they are using a billion weights. So what a weight is could be a number that's going to be attached to a token. So there are a lot of updates. These updates, typically what it is is you have a deep neural network. So a weight, when you change a weight it means you may activate a neuron in your neural network. So that's what eats a lot of the time in these implementations. That's what requires a lot of gpu. That's why there is a lot of cost associated with it. The main cost band. Now if you have tons of users in real time and you need to deliver answers very fast, there's another Cost because you need to efficient mechanism. That's why you probably heard words like approximate nearest nipple search, which is one of the techniques they use in a vector database to kind of match an embedding coming from a user prompt to an embedding in backend tables. But when you have that on a very large scale, you have these discourse testing evaluating is also expensive because when you change these neural networks retraining is going to take a lot of time. And in kind of implementation that if you don't have those, those cost. I like to joke and even mention that with my client you can download the whole Internet 99% of what can be useful in term of knowledge on your laptop and not even the most expensive. Or my laptop is, I don't know, maybe six years old. I could do that on my laptop. What's going to take time in this case is going to be the crawling. You can distribute it, you know, you can make it in parallel. But still. But if you crawl the whole Internet is a little bit easier. When I crawled the Wolfram corpus, I think it was like €15,000. It's kind of a small. It can cost real fast to check, check, check, check. It's really fast to extract the whole thing. But you get blocked to realize some ways hitting the whole website. So if instead of you have plenty of websites, you're not going to hit massively one website at a time, it's get distributed. So you avoid these problems. And I know OpenAI, they don't even do any crawling. They use open crawl or something like that. So they you can also use those external tools. But it brings also under the concept should someone or me or anybody else be allowed to crawl those websites? And I mean sure, it's not illegal, but is it ethical? Is it, is it fair? Should I pay Wolfram or you know, these are important questions, ethics behind that and what you're retrieving as well. Me, I'm essentially retrieving knowledge. So I have fewer problems than maybe companies like OpenAI. If you retrieve personal information, information and stuff like that, then it's not a challenge.

Yeah, absolutely. Certainly just trying to scrape the whole Internet or scraping like everything from some particular website. You've got to be very careful about the legalities. And I'm sure most websites have a notice about like what's permissible buried within there. Like there'll be a robots Txt file or something to tell you what you're allowed to do. But yeah, that's very interesting the idea that actually if it's Sort of ethical. And the website owners are okay. You can just get a load of data and on something you're interested in, download all that to your laptop and then go and create an AI. Create your own bot from it.

Yeah. The case of Wolfram was interesting because initially they blocked me, obviously I found some workarounds, but eventually they unblocked me and I believe they must be very happy because it got a lot of publicity. A lot of people have seen this result coming with Wolfram with URLs from. So that's also one big thing. OpenAI is not going to tell you where that content comes from. Me here, this is the URL on Wolfram Alpha. So I believe that in the end they were very happy with that, especially as me mentioning that one of the best website for everything mathematic, if you want to start with.

So yeah, that level of transparency is incredibly important. It has been kind of lacking from a lot of the big players in the generative AI movement. Certainly particularly OpenAI isn't very forthcoming about what sort of data they use. And I agree with you, the bull from site for their glossary of mathematics is absolutely amazing. All right, so just going back to the costs of things, it seems like a lot of organizations, if they want a custom LLM, they've got this choice between did they build something themselves? Do they maybe buy something off the shelf and fine tune it? Do they get someone else to try and create the LLM for them? Can you talk me through how organizations should think about this sort of build versus buy decision?

If you design your own LLMs, it's not necessarily perfect. So there are also some drawbacks that come with it. First, it's not being as thoroughly tested as stuff like OpenAI. If you're an expert who's developing the LLM for your company, what if that guy disappear or the guy leaves and it's not properly documented and stuff like that. So you're facing issues like that. Obviously the benefit is going to be much less, should be much less costly. You have full control over what's being done and there's increased security, but you have a local implementation for these days, many of these big LAM companies are also offering local implementations. Yeah, so it's not, you know, black and white, that one is better than the other or stuff like that. So their drawbacks are protected.

And do you have a sense of who you need to create one of these things? Like, is it going to be a single AI engineer? Is it going to be a whole team of people? What do you need to get started.

Another issue actually with this specialized LLM especially if you want high quality I use a lot of customized tables stop words. If I use autocorrect libraries from Byton unspecialized and then they're causing problems. They autocorrect stuff that should not be autocorrected. So you need do not correct autocorrect stuff tables and building that I don't think it can be fully automated. It's going to be requiring some human interaction by you know, you look at the results you get and when you find out which stop words you need to add in the table. So these human beings, there's a cost there. So I actually made a competition one day that for the whole Internet if I was to cover the whole Internet unique big scalable company I would need like 20 full time people working just on optimizing, looking at the stopwatch, doing that kind of stuff. So a team of 20 people focusing on that that would be the biggest, most expensive component more so than the engineers it and then because someone asked me the question, you need a lot of human beings to fine tune your system. By the way, I also use a lot of human beings. I even myself received offered for home testing prompts and stuff like that. So the cost company like openaise mostly is going to be engineers. While in the end with my 20 employees paid full time, they're not going to be cheap employees, they're going to be experts. It's still less than the number of people a company like OpenAI must have. I don't know how many employees they have. I would assume hundreds. I may be wrong.

Yeah, I think there might be thousands of people at this point in OpenAI but so that's interesting that the biggest cost then is. So I believe this is like the reinforcement learning from human feedback steps. So this is people who are checking the results of the output and then marking it as good or bad.

Especially if you want to do it fast and cover the whole Internet. For instance the case for a corporate client I'm going to be to myself I'm not going to spend many hours looking and but it's a very small part of the whole knowledge. You know doing those kind of test myself, I don't have those expensive GPU and training costs that because besides the engineers of course there's the computer time cost that OpenAI and Mistral in those companies are facing.

So that's interesting. So let's just sort of recap like which skills you need then. So it sounds like you're gonna have to have some people who are involved in a lot of like doing the data processing. You have data engineers, people who are experts in data management up front and then you need some AI engineers to actually build the model and then you need some experts to be able to test the model output. Is there anyone else that needs to be involved if you're wanting to create.

A custom LLM, obviously you need all the traditional people that you would find in any company. In my case I'm pretty lucky it's new people are not seeing what I'm doing. So I don't have the sales team. People are knocking at my door in company like they came to me, Morgan Stanley, they came to me. I'm also self funded, but very well self funded. I saw my previous company, so I don't have this pressure to try to make money asap, that I'm going to run out of money. So I don't have that aspect. So I was talking sales, it sounds.

Like you need all that kind of standard support stuff. So your project managers. Project managers and your.

I do have an engineer which working with me is designing web APIs for my product. The guy was in India, actually was a Oracle principal staff or something like that in India. But I think you need more like ML deaf engineers as opposed to research scientists. Well, I do a lot of research myself, but I would not need to hire another guy doing what I'm already doing. You need a guy like that obviously and if possible a guy like that. For instance, in my example I'm not just a pure academic research guy, a guy who knows how to code, a guy who's going to be very close to the engineers and like a co founder, which you know I am in this case and I think you also interested maybe to know the Python what skills. So there's what I call smart crawling. So it's not just what you typically would learn from a, a simple bootcamp or something like that, to extract the taxonomy, to retrieve the knowledge graph and to do it efficiently. So stuff like simply using a beautiful SOAP in Python is not enough. So you're going to need some training possibly in Python on web crawling. It's certainly very good to be familiarized with all the NLP libraries. So a course on that topic, I faced many issues with those libraries and some of the stuff that I rewrote from scratch better serves my needs in this context than some of these Python libraries. Yet you still need to know what they are doing, what drawbacks come with these libraries and in Some cases these drawbacks are easy to fix like do not stop word or do not autocorrect to create stuff like that. And at the same time you still leverage those libraries.

Okay, so just on that note, so these Python natural language processing libraries, I presume you're talking about things like NLTK and spacy and things like that. So it's interesting that these are still very relevant in the age of generative AI because I think some people sort of thought well okay, now we got these big large language models. I don't need those older NLP tools. So that's interesting that they are still useful in the context of generative AI.

Yeah, any better tools, any new tools that come that are going to replace these libraries are definitely worth investigating. So it certainly would be nice to find out what the new libraries and more specialized libraries that come with it. So another example is I did use LLM for clustering predictions. Everybody's doing that as well. But I use Sklearn Python libraries to do some clustering based on text. So there's a disability matrix there. But I use it's so like I mentioned, I really leverage sparsity. You're really dealing with very sparse data. So sklearn, unless I'm missing something in asking me to build a distance matrix, so if I have a million tokens, it's a million by 1 million token and you declustering on that matrix, it's very simple to create that distant matrix out of your similarity matrix based on hash tables. But the problem, you run out of memory stuff like maybe I'm missing something with the scale in maybe there are some other libraries but the simple libraries like that are not going to be enough in my case I do have some very efficient way to do clustering with connected components with graph related technology that's leverage the sparsity of this system. So what I'm saying here as a summary, if you just use the standard libraries, you're going to run into problems you need to go beyond and at the same time you don't want to reinvent the wheel. That's a risk Also if you. So you need to be knowledgeable of what's out there, what you can use. You can even use OpenAI to create some code, asking some code to solve some problems. I did and in some instances I got some great stuff actually.

So yeah, it seems like an understanding of sparse data is going to be incredibly important for this and then understanding knowledge graphs as well. So there's quite a lot to. All right, so before we wrap up. What are you most excited about in the world of generative AI right now?

Well, we're turning points where there's a lot of old stuff like Google search that are being revisited and can be done much more efficiently. I was just writing this very recently. Search is going to be powered by rag LLMs. It's the end of PageRank.

The end of PageRank. That's a bold statement. I think there's going to be a lot of people at Google are a little bit frightened about that.

Yeah. Then I've been with working on some of the projects as well. Like no Gans. There's plenty of low hanging fruits. People are complaining about how bad many of these tools are. The poor results I found. This is exciting. So much thing to improve. It's going to be improved. There's a lot of work to do. There are some jobs that can be created when I create my own jobs. But I mean people are talking about this technology is going to replace everyone. But for now my opinion is that there's plenty of low hanging fruits. There's a lot of demand. I mean people are talking about billions of dollars associated to LLMs. The market costs are going down, the system are being improved. I mean it is like the first cars they were driving, you know, 40 miles an hour. You need to know how the engine works, the broken road with portholes and stuff like that. Now you, you don't drive a car. It's, you know, I believe it's going eventually in the same direction. Wonderful.

That's a very optimistic take and I like that. The idea that, well, you know, if there are problems now then that just means that there's something to be solved. So there's interesting things to do.

Yeah. And especially when people, some people and even sometimes myself in mathematics I think everybody has been discovered already. We're in the end of it. There's no more, no more thing to discover. But that's not true at all.

That's wonderful. I love that. So, yeah. Thank you so much for your time. Vincent.

Thank you so much Richie, very nice talking to you. Thanks for the invite.