#297 The Past and Future of Language Models with Andriy Burkov, Author of The Hundred-Page Machine Learning Book

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

LLMs they aren't agentic, so they don't have agency on themselves. So the LLM doesn't know at any moment in time that it's in the process of solving any problem. It was fine tuned to generate the next token and then we also fine tuned it to generate the next token so that it replicates some patterns that we find useful. Because the model is not agentic. You just say you are a very capable agent, but it doesn't make it agentic just because you proclaimed it.

Welcome to Data Frames, this is Richie it's widely agreed that machine learning and AI are fairly tricky concepts to get your head around. You can happily spend a whole lifetime learning more about the topics. Unfortunately, life gets in the way and most people don't have time for that. Today's guest has managed to distill these topics into short books that teach you the fundamentals of machine learning and AI in just 100 pages. Our guest is none other than Andre Berkov, the author of the hundred page Machine learning book, the 100 page machine learning Engineering book, and the hundred page Language Models book. Today we're going to distill these 100 pages down even further and find out what are the key concepts about LLMs that everybody needs to know. Beyond the books, Andrej is also working at the cutting edge of AI. Until becoming a full time author, he was machine Learning Lead at Talenteurum, an HR AI platform and he's also a Director of Data Science at Gartner. That means I'm keen to learn about trends in LLM development as well. And based on his social accounts, Andrei is not shy about giving a spicy opinion on AI, so I'm very excited to hear what he has to say. Let's find out. Hi Andrei, welcome to the show.

Hi Richie, thanks for having me.

To begin with, I'd like to know, what do you think is the most common misconception people have about artificial intelligence?

Probably there are More than one. But I think the biggest one is that it's actually intelligent. So it's a scientific term that was invented by scientists to talk to other scientists. And scientists often invent some really funny terms just for the sake of having a common term that everyone is agreeing on. This artificial intelligence was one of those. But then when artificial intelligence has become very popular and available not only to scientists but also to regular normal human being, people starting operating this term as if it was, you know, like I remember news articles or news episodes on tv and they say AI does this and now AI does that. Like if it was like some specific AI and previously it did that and now, now it does that. But it's just different algorithms that people apply in different domains, calling them under the umbrella of AI. But yeah, it's not one AI that does all this. And it's not really an intelligence, it's more like intelligence like outputs that you get when you apply those algorithms to some problems.

So I guess relations in your book, you go back an awful long way in history. So it goes right back to the start of artificial intelligence, like the middle of the 20th century. I say I was very impressed. Like it's a very short book, but somehow you've managed to pack in like 70 years of AI history. So talk me through, why did you decide to start back in the 1950s?

I think this was to show that AI is not language models and language models is not AI. Okay, because again, the book was released at this specific moment in our history where just suddenly, like I call this two years after ChatGPT was released. Suddenly like millions of hundreds of millions people have been exposed to an avalanche of information about artificial intelligence. But 99% of information that people have received in these two years were about language models. And it's okay because, well, currently it's the most powerful algorithm out there. But I didn't want this book to be about large language models like the pre trained transformers, because it would make an impression that this is what AI is like. Okay, there is a book on language model. You open it, it starts with the transformer, then how we pre train it, how we fine tune it, and how we use it. And people would say, okay, Well I know AI, but no AI, it's much bigger than that. And it didn't start two years ago, it started like in 19, I would say 40, 50. And it started small. But over the decades the field has grown quite significantly. So my PhD for example was 15 years ago and it was on agents and multi agent systems. So like when I today see people talking about agents without really understanding what it is. Because there is a scientific definition, not one, but several definitions of what agents are. And if you don't start with definitions, if you just operate words that everyone uses, then it's not science. Because your agent can be different from someone else's agent. Your AI may be different from someone else's AI. So I wanted to show not just the evolution of language models, but at the same time to show that transformers and language models, it's just a continuation of what's going on for many decades. So now people who will read my book, they will say, oh wow, I'm interested in language models, but there are other things. So where do I go next? And you can go next to many directions. And I wanted to cover at least the most important ones in the world.

That's very true that it is helpful to have an understanding of the history so you understand like why these things happen. And it's important that, yeah, AI didn't start with transformers back in whenever it was 2017. I'm curious though, are there any parts of 20th century AI research that are still relevant for people working in data or AI today?

Plenty. For example, logistic regression, it's been around. Well, I might be wrong, but for about 60 years logistic regression was there. And currently it's still one of the most used algorithms in production AI. Why? Because it's fast, because it's implemented very efficiently. So you don't have any troubles training logistic regression whatever problem you want to solve? Well, it's only a classification, we might say only classification algorithm. But classification is one of the most important problems that we usually solve because a classification is a decision. Like for example, when you decide whether a letter is a spam or not a spam, you make a decision spam, not spam. And to make this decision, you train a classifier. And currently, again, we might say only classification algorithm, but classification is one of the most important problems that we usually solve because a classification is a decision. Like for example, when you decide whether a letter is a spam or not a spam, you make a decision spam, not spam. And to make this decision you train a classifier. And currently again, people might not know that there are other algorithms that would allow you to classify things and they would start straight away by taking some pre trained transformer and trying to fine tune it or just instruct it, like classify these things. But this would be an overkill. You know, it's like you want to dig a hole like 50cm deep and instead of taking a shovel, you pay for the excavator. Like it doesn't make sense. So logistic regression is just one, but there is also support vector machines. Support vector machines is very powerful for text classification together with logistic regression. And by the way, in my book I wanted to demonstrate it, and I demonstrated it very clearly, that if you use logistic regression as a classifier and you use scikit Learn, which is like a kind of a basic machine learning library, you can train a very strong classifier that is very hard to beat with the state of the art language model that you will fine tune on the same data that you use for logistic regression. So it's not just faster, it's not just cheaper, but in many cases, especially for text classification, logistic regression and support vector machines are still state of the art and even better. So I've shown that even the baseline logistic regression implementation is very hard to beat with a transformer. Well, of course, depending on the documents in some database, in some text data sets, maybe transformer will be better. But there are data sets and they are realistic where these simpler algorithms are still state of the art.

Excellent. Yeah, so I totally agree about the use of logistic regression and how many of the world's businesses are running on these sort of older techniques. The other thing that surprised me about your book was the use of linear algebra. There's quite a lot of math in there, and this is one thing that we've been going backwards and forwards on at Data Camp for a long time. It's like, well, how much of the math do you teach people? So what parts of linear algebra do you think are important for people working in AI to know about?

Well, people often ask me this question and I would say the only thing that is critical to understand what we call modern AI, which is mostly neural networks. I think the only critical piece is the first derivative of a function. So it's like a part of calculus. Usually you learn this in college or university, like it's the first year, I think. So why it's important is because if you really want to understand how a neural network is trained, you must understand the concept of gradient descent. It's an algorithm that usually 99% of cases, this algorithm is used to optimize the parameters of your model. And this algorithm works on functions. So basically your model is just a function like F of X Y, F of X. And this F of X can be defined very simply. For example, the linear function is, let's say WX plus B. So X is your input, for example, your letter you want to classify as spam, not spam. And those W and B, those can be just numbers. So let's say you, you want to find what those numbers should be for your specific problem so that your classifier is the most accurate. So you use this gradient descent algorithm that will take your data set and it will adjust those W and B iteratively so that in the end those W and B converge to some values that then you put in production and you use it. So just to understand this principle, you should understand the notion of the first derivative. And the first derivative in functions is showing for every point, like for example, if you have a curve like this, for every point on this curve, the first derivative just tells you whether the function grows in this point or it decreases. It's very simple concept. So if you understand this concept, then you understand gradient descent. And if you understand gradient descent, then you understand how the neural network is trained. And then when you understand how the neural network is trained, trained, you understand how transformers are trained. So like, it's all depends on this first derivative in the beginning. And it's not like I don't consider myself like a strong mathematician. I understand many things, but it takes a lot of time for me to understand that. By the way, maybe we will talk about this later, but I really appreciate LLMs as a assistant in understanding things. For example, previously to understand something. So you read the book and you're stuck on the equation. I don't understand why, why the author brought up this equation. Why should I understand how it works? And if I don't understand why, I'm not motivated to continue. Because I'm kind of, okay, now I just continue for the sake of continuing. I don't know where am I going. So with LLMs, you can select a bunch of text, you paste it, you explain the context, and you say why this equation is here. And it will say, ah. It's because it may be wrong, but if it gives you the right intuition or even a wrong intuition, you can compare your intuition with this equation and say, ah, okay, I see now. Or you can say, no, still don't see. You repeat the questions and eventually you understand why and you continue. So for me, it simplified learning from articles, from books quite a bit. So this is all to say that now it's not really difficult for a newcomer in the field to understand the first derivative, even if they didn't understand it in college. It's one of the simplest concepts out there. For example, integrals are much harder for people to understand because their physical meaning is not as obvious as the first derivative. The first derivative, you just say, the function grows, the function decreases. Very simple. But the integral, it's an area under the curve. And why would this area under the curve be important? Why would we measure it? So this is much harder. But fortunately, to understand the modern machine learning, you don't need integrals. But in some cases, yes. But not to understand neural networks.

That's kind of cool that you can get a really long way with some simple linear algebra and understanding derivatives, like essentially basic calculus. And then I also like the idea that you can use AI to explain things that you don't know. In fact, I have to say I have to read a lot of books as part of my work, and often it's like, yep, using AI to summarize them. If it's going too slow, really useful in your case. I feel like you packed like a thousand pages worth of AI research into a hundred pages, so not much summarization needed there. But yeah, I like the idea of using AI to just be able to change the pace of and the difficulty level of whatever you're reading. So related to that, you cover a lot of more recent techniques. So I think there's a whole chapter on recurrent neural networks, which were really, really hyped, I guess, in the early sort of 2010s, and they've been shunted aside a bit with the rise of Transformers. Are some of those more recent techniques still useful?

Well, I didn't want my book to be about some expired tech just for the sake of it. So I really was happy to create a path from the beginning to the end where you not just discover the history, but also you discover the history by learning about stuff that is still state of the art or at least used in some context. So recurrent neural networks, they were replaced in the beginning with the transformers. Why? Because transformers are naturally parallelizable. So in recurrent neural networks, to make the model predict the next token, you first need the model to read previous tokens. And this reading of previous token in neural network is done one by one. You read the first one, you update the neural network internal state, then you read the second one, and then you reach the end of your prompt, and then it starts, generates the continuation. So this was not inherently wrong. It just. It was hard to parallelize because you cannot predict the next token for every time step at once. The way we train transformers, we predict the next token for all previous token, not just for the last one, but for all of them. And in recurrent neural network you would have to kind of this recurrent reading and prediction which is slow. So just because of this transformers they have shown that you can train very fast and the model will become very good and you can train on very large parameter sizes. But it doesn't mean that the recurrent neural networks are worse in any way. And we have seen just in 2024 that several recurrent neural network architectures have been proposed like XLSTM and MIN LSTM that don't have this recurrency dependency like hardwired in the architecture. So now you can train a recurrent neural network which would be as good as a transformer. And in some cases you can win in speed because recurrent neural networks are usually smaller than transformers. So you can run the inference token by tokens much faster. So for this reason I included recurrent neural network. But also it's a very important learning step because if you just start from scratch and you go directly to the transformer, it will, it will feel very overwhelming and many people will be intimidated by this kind of, oh no, no, no. I, I will never get this whole thing just myself. But if you start with count based models that don't use any neural network, you understand principles like language model is how does it work? And these count based language models are not kind of expired either. For example, on your smartphone, when you type something to your partner or your friend, often it's repeated stuff. So you repeat every time. Let's say you have a tradition, go to a bar every Friday and you can type to one of your friends like okay, do we go to X? And this X will eventually start to being suggested as the continuation in your keyboard. But this is not a neural network because no one will retrain the neural network on your smartphone. And even in the cloud it's an overkill. So they still use these count based models that just say they just count how many times this X follows these preceding words. And if it's the most frequent continuation, it will just show it to you and it's okay for you because you don't need on your autocomplete to have a very long context to predict the next one. So count based models still use cool. Recurrent neural networks are still state of the art cool and then transform it. So you kind of start with count neural networks to understand the principle. Then the recurrent neural network shows you what is the most straightforward way to implement a neural network as a language model. And then Once you understand the whole math around the most straightforward way, then you say, okay, now let's consider some less straightforward way. But the principles will be the same, so the reader will not feel as intimidated as they would feel if they started with Transformers from scratch.

Okay, that does make a lot of sense that you build up the concepts gradually rather than just trying to dive straight into the cutting edge stuff and get overwhelmed. And it doesn't really make any sense. And I do like that those simple sort of count based models, I guess these things like bag of words. Is that the sort of idea, like the very simple natural language processing models are still in use?

Well, yeah, so it's not bag of words, it's more like grams. Like for example, you have, let's have an A gram that says this evening we will watch a. And what will be the next step, the next token, the most likely, that you will watch TV probably, or a field. So you just take a, like a data set of different texts. You chunk them into n grams, for example, 5 grams. So it's like five words following one after another. And then you say, okay, I will take the four as my context and the fifth, I will count how many times this fifth word follows this previous four words in my overall data set. And then when it's time to predict, you just take this four gram from a dictionary, you take the continuations, count, and you say, ah, okay, for this four words, this continuation is the most likely and this is what you predict. There is no kind of math other than counting involved, but it's still very, very useful on small devices where you don't need to retrain anything, you just update counts and it follows your usual conversations with your friends.

That's kind of cool because there is a sort of sense that these like big neural networks are just sort of eating everything in terms of technique. So it's nice that some of the older sort of less computational intensive things still have some kind of value. All right, so you mentioned that there were these sort of new recurrent neural net architectures for large language models. What's happening in the world of large language model architecture? Like what are the big developments you're excited about?

Well, the biggest today is of course those, what we call them thinking or reasoning models, also that they are known under the inference, time, compute or test type, compute model. So basically the models that instead of giving you the answer right away, as ChatGPT initially did, these models generate what we call the chain of thought. So it's an internal discussion of the model with itself. That helps it to come to a better response compared to what it would be if this discussion didn't happen. So it was there for probably a year. The ChatGPT, I think, was the first one to release the thinking model. I think it was with chatgpt01 that they released it. OpenAI didn't want to show what's going on in this thinking process. And everyone guessed maybe they invented something complex to make it happen. And it really was the models that are thinking were better in many tasks. So what was the most impressive that happened in the past two months? It's Deep Seq, the Chinese company that released the model that they called R1. And they trained this model to generate this chain of thought. And not just they have shown what this chain of thought looks like, which would already be like a very, very good hint for the scientists to try to reproduce something like this. But they also, they published a paper that explains how their thinking model has been done and they put them up and the algorithm that anyone can reproduce, the algorithm that, that they called grpo and this algorithm, actually they talked about it in previous papers as well, where they explained how they trained a model for mat. They called it Deep seq, Matt. And many people already implemented it and confirmed that it works very well for MAT. But R1 made so much noise because they have released this chain of thought and they made it kind of a demo where anyone can read this chain of thought, confirm that it's actually resembles a thinking, and anyone could reproduce it. So for example, I took a very small, by modern standard, a very small language model, it's Quinn 2.5 with 1.5 billion parameters. It's considered tiny today. And I took a publicly available data set with math problems and answers without the solution, just the problem and the answer. And I fine tuned. So I implemented this GRPO algorithm from their paper and I fine tuned this model to generate this chain of thought and generate the final answer. And just with this small model on this school level math problems, the model was capable of solving 90% of them. And I didn't even use the whole training set that I have. So the model converged to this 90%, it wouldn't improve. But the only reason why it wouldn't improve is because it wasn't large enough. So if I took a model with 3 billion parameters, or 7 or 20 or 72, it would learn higher and higher and eventually it will converge to something maybe around 95, 97%. Like you can say, anyone living with their parents in the basement can rent A quite affordable node of GPUs, like for example, a node with eight GPUs with hundreds like 80, 80 or 100 gigabytes of memory and fine tune whatever model they want to become, the best model in the world to solve this specific kind of problems, like school level math, college level math, some mathematical proofs or whatever, whatever data set you can get, you can use it. It's achievable by a random person. And this is what it was, a wake up call for the entire industry. Because if previously we could at least kind of assume that OpenAI had something, now we don't assume that they have anything. So anyone can do exactly what they do. So it was kind of, this is why, you know, there was this fluctuations in Nvidia stock prices because people were like, okay, if we don't need so much compute to create state of the art models, who would need all those GPUs? But again, this reasoning was a little bit not well thought because like I think now for these reasoning models we need even more GPUs because we do this additional compute during the inference. So yeah, for me this was the most important. And this is why, by the way, I decided to work on a book on reinforcement learning as my next book in the hundred page format. By the way, just recently Satan and Barto received the Turing Award for reinforcement learning. And I'm sure a hundred percent that the very recent result about R1s was one of the reasons why they were selected because it's a very real improvement in AI quality. So it's not like we take a larger data set or we increase the number of parameters. No, it's a smart algorithm that doesn't cost much and results in a very high boost in AI performance quality. Well, only for math, logic, logic and reasoning tasks. But it's not a negligible chunk of what people use them for.

That does seem like quite a huge shift then in terms of the market for large language models. I think there was a sort of trend towards there being a few just huge foundation models. And now you're saying, well actually maybe the trend is going the other direction. So everyone gets their own, I guess, almost disposable model, like just something that's cutting edge for solving a specific problem. Do you think that's going to be the future then having many personalized models?

Well, they will complement each other because like just recently, if you follow the news, maybe yesterday or two years ago, two days ago it was Quen. So it's from Alibaba. They released this qwq 32, which is a reasoning model and they compared it to R1 and they have shown that on those reasoning tasks it's very comparable, but it's much smaller. And many people without really, you know, testing long enough started to speculate that okay, now we can have a R1 grade model, or let's say ChatGPT grade model in 32 billion parameters. Which is not true. Yes, for math, for logic, for reasoning, where you can get the data sets of task solution format. With reinforcement learning you can train a very, very strong model and it will be small, like as I trained 1.5 billion to reach 90%. So of course if you have 30 billion, you will reach higher than 90 and for more complex problems. But it doesn't solve the overall problem of these models with hallucinations. So hallucinations happen for many reasons and on many occasions. But one of the very frequent use cases that people use language models for is to ask questions like for example, what is X? Or who is this person? Or what this person is known for or why this person is so important in the history of this country and so on. So this smaller models, they don't have enough parameters to store factual information. So basically when you do the gradient descent update, the update goes to adjust some parameters that will result in the model capable of predicting the next word in this specific context. So if you have a lot of parameters, somehow and it's still open scientific question how but somehow this gradient descent doesn't erase previous information, it goes to the parameters where it still can't put some new information without erasing the old one. But if you only have, let's say 10 parameters to train well, every new piece of data that you put in it, it will erase everything that was before and replace. So it's always instructional to look at the extremes. So if you have one parameter, you will not remember anything. If you have millions of parameters, you will remember something. Well, your model will remember something. And if you have billions or dozens of billions of hundreds of billions of parameters, then your model can remember quite a lot. So this 30 billion parameters. Qwq is good for math, but if you ask it factual questions, it's as is hallucinate just as much as any model of this size. So currently there is no practical solution to solve hallucinations in small model. The only solution we know is grow model size. So the large models will coexist with small models. People will train small models for very business specific tasks using reinforcement learning or even just supervised fine tuning. But if you want to create a assistant that is knowledgeable in multiple domains, you will still want to work with as large model as possible. This is by the way, why this GPT4.5 that OpenAI released recently climbed quite high in scores. It's not smarter for math or for logic or. But it's much more accurate in fact, because it's much larger than previous model.

Okay, so that's really interesting that you're going to need different types of model for different use cases then. So yeah, you mentioned if you're answering questions with a sort of true or false answer or some kind of factual answer, then you can need that large model because it needs to be wise, has that sort of broad data input.

And then something where you can use a larger model or you can train a smaller model to access some data like kind of knowledge base and pull relevant facts from this knowledge base and then just summarize them, what we call retrieval augmented generation. The problem is to create a high quality knowledge collection like this. It's not easy because knowledge, if you just pull it from the Internet, it can be wrong. Or because people don't always verify what they post online, they say it one of the reasons for hallucinations too. So if you really want to have a very small model that is capable of talking about very wide number of subjects without too much hallucination, you need a high quality knowledge collection, which not many can have.

Since you mentioned reinforcement learning before as being incredibly important, can you tell people like, it always feels like the sort of unloved cousin of supervised learning and unsupervised learning. Why do people need to care about reinforcement learning?

Well, people need to care about reinforcement learning because this is the closest type of learning to how humans or animals in general learn. No one trains, let's say dog by showing it many, many pictures of how to, you know, chase a prey and then the dog, after watching like a million pictures, like, oh, okay, now I know how to pray. No, you actually shoot a prey and you say go get it. And once it brings the prey back, you give the dog a reinforcement. This is by the way, where the name comes from. So you reward the agent and in this case I'm really talking about the real agent and not the language model renamed into an agent. You reward an agent for achieving the the expected result and you punish it. Well, you don't punish the dog, but you punish your robot for not achieving the expected result. So how do you do it? Like, you cannot give a reward to a robot after it makes a very Very tiny movement. Like, okay, it made this tiny movement. I give it 0.1 reward, and then that movement like, and I give it 02, like, you would like to give the reward in the very end of the task. So the reinforcement learning, this is what allows you to propagate this final reward across what we call the states of the environment. So the robot operates in some environment. So if you ask it, bring me an apple. So the robot should find the shortest path in this environment, detect an apple somewhere in a bucket or on a tree, take it, and bring you back. And then you say, good robot, here is the hundred reward. And if it bring you a banana, well, you don't want to punish it entirely by saying, no, I take the hundred back. You will say, well, for banana, it will be zero. But the robot will still learn that at least it brought something and it's not a total disaster. So bringing something is a good thing. So the reinforcement learning allows you to propagate this reward distribution across all the states and all the actions the robot made in all states of the environment in which it was. So the first time it will not bring anything, it will get negative reward. But the first time it brings you something, it will say, reward will be propagated. So the actions that led to this final reward, their probability to be chosen will be higher by the end of this. So this is why it's very important to have simulations in which you can train simulated versions of the robot. Those simulations should be as close to the real environment as possible. So in this case, you can run millions of copies of the same environment and millions of copies of the same robot, and they will all, in parallel, try to, like, execute some tasks. And even if one out of million eventually succeeds, then you propagate its rewards to all the action that it made, and you update what we call. We call this the function which is used to select actions in each state. We call it the policy. So this policy can then be copied to all other replicas of this robot, and they all simultaneously become a little bit smarter. And you continue this way with millions of simulations, and eventually at least one or two or three will start systematically solve the problem exactly as expected. And you can imagine that in supervised learning, showing all possible kind of, here you do this, and here you do that. It's impossible. It will not scale. If you want to have robots that live with us in a typical house, the robot must learn on its own. Like, it will bring you beer on Friday evening, and you will say, wow, good one. And it will learn it will get the reward from this. Wow, good one. And it will learn that bringing you beers on Friday, it's a good thing.

That is seemed like very appealing. Have my own pet robot to bring me beers on a Friday evening. I like that idea. Oh, and since you mentioned policies, I guess this is where we come back to the relative policy optimization from the large language model. So again, I think it's a reinforcement learning technique. It's nice that there are some sort of good examples of use cases for reinforcement learning. I think like a decade ago it was all about playing games. It was like, oh well, this is how you win at chess or go using reinforcement learning. But now it's like, well, it's robots, it's large language models. So lots of like really important business use cases there. Okay, all right. So I don't think we can get away without talking about AI agents. So they've been hyped for months. They're kind of starting to appear in the wild. What do you think the big use cases for AI agents are?

Okay, well, first of all, as I said in the beginning, we operate terms. Well, I say we collectively all people, we operate terms that we didn't define. Okay, so AI, it's operated in a very broad sense like something that can think and solve all sorts of problems. And the same goes about the agents. So there is a very popular. Well, not popular. Well, it's popular but also important book that is used as a textbook in most universities worldwide, from Russell in Norway, called Artificial Intelligence A Modern Approach. And this book is kind of the door that you open in the world of AI when you go to university. So in this book there is a chapter specifically about agents. So the agent is defined as an actor that acts in a certain environment. It perceives the environment states, it can execute actions in this environment, and it receives a feedback from the environment, usually in forms of rewards, and an observation of the next state of the environment in which this previously executed action brought it. So this is how the agent is defined. Now if we define the agent according to we read in the news, it's nothing different from just an LLM that you fine tune to. They call like call functions. So if you say you are a useful assistant and you can help the user to get some information when they need it. So when the user asks for, let's say the information about the best deals to buy a TV online or a car, then you must call the function that connects to, let's say Amazon or Google Marketplace and pulls the search results for it. And Then you summarize it and you explain that this function has these inputs. For example, the topic, it might have another input, minimum and maximum price that the user is ready to pay, and some other inputs. And then once you have this input, you call the function and you will get the result and the result. So the LLM itself cannot call functions. The LLM just says at this point in the conversation, this function with these parameters must be called. So then you parse this, you, I mean the developer, you parse this suggestion by the LLM to call this function with this parameter, you parse it, you actually call the API, you submit the parameters, you get the results back, you put it in the context, and the conversation goes, so this is kind of a assistant agent. There are some other agents that people use to work collectively on something. For example, we write a document and we need multiple pieces of information to write this document. So there is one agent responsible for making sure that the overall structure of the document is as user wants. There are other agents that sees the state of their documents and says, okay, here we need to add additional numbers. So I will go online, I will pull some numbers, I will put them there. It's kind of looks like robots are working for you and you just sit and wait until it's done. The problem with this is that LLMs, they aren't agentic, so they don't have agency on themselves. So the LLM doesn't know at any moment in time that it's in the process of solving any problem. It was fine tuned to generate the next token. And then we also fine tune it to generate the next token so that it replicates some patterns that we find useful because the model is not agentic. You just say you are a very capable agent, but it doesn't make it agentic just because you proclaimed it. Okay, so like where the problem is? The problem is that because none of those agents actually know that they are solving a specific problem, they just follow patterns that we either provided in the prompt or we fine tune them to follow this pattern. So they can be arbitrarily wrong and still continue to follow these patterns. No. One of those agents will say, oh, whoa, whoa, stop, stop, stop, stop. We were supposed to write a document on bananas, but we are now writing on potatoes. Is it something we should probably reconsider? No, if it goes in some direction and it goes in the wrong direction, they will continue to go in the wrong direction. And you don't need agents to test it. You can just ask an LLM to explain to you something. And you just need to make sure that you understand this domain. Like for example, when I work on my book, I ask it like, explain to me this equation and I know how it works. I just want to see an explanation and probably draw inspirations and write something probably similar, probably better. So it explains to me something and I know that it's wrong. So because I know the domain, I say, no, no, no, this is not this, this is that. And it says, oh yeah, right, this is that. And it rewrites it. But it will not happen with the agents, Never. So the agents will take the output of other agents as real stuff and they will take this output as their input. And then the process will continue based on this wrong input going forward. So this is why, for example, you heard about this world first AI software engineer.

Oh, Devin.

Yeah, Devin. So like when they just started to talk about it, they already proclaimed, that is the world's first AI software engineer. It was such a strong claim that I don't know why the rest of the world didn't laugh right away. But the hype was so high that everyone expected miracles. Yeah, well, this will be the world's first AI software engineer. And this one will be the world's first lawyer. And this one will be the world's first doctor. But if you are a doctor, you will ask questions about the human body and you will find so many issues in those answers, you will say, no, I will not trust it. Yes, in some domains AI is good. For example, when you show it this X ray with some cancerous cells and it can circle around, this is cool and I will use it, but I will not talk to AI to decide what treatment I will apply because this is crazy. Like one time out of two it's wrong. So the same about this Devin, which is an agent, because, well, you say, I need this problem to solve, this is my environment, these are my passwords, credentials, and so on. Now go for it and bring me the solution. And like it's been already more than a year, we don't see any, any Devin deployed anywhere in production. Some, some recent tests have shown that up to 25% of tasks have been solved. And knowing that we never reach a hundred for whatever problem it is, if you, after a year of efforts, you managed to read 25%. And even those 25%, I didn't see them, but I think if you really dig deeper, you will see that the code is not good. There are plenty of bugs yet it's compiled and somehow the test cases passed. But it's not manageable. The code is so wrong that no programmer will say I will continue to maintain it or I will continue to ask Devin to maintain it. So like, agents will be good in situations in which the data and the problem itself can be described verbally. And this verbal description can be easily found online because this is where the LLMs get the data for pre training. So if your document or your problem can be described verbally, and there are plenty other descriptions of such problems online, the agent will be good. Why? Because it has seen already plenty of patterns of this problem and it knows what to do with those patterns. Otherwise it will be, I don't know if we can still call it hallucination, but it will kind of actions without meaning in many cases because the pattern itself is not very familiar to the agent. So it will just do something for the sake of doing so. It will kind of correspond to hallucinations. But we need to invent probably a better term to this stupid action. And those stupid actions will accumulate and the more agents are there, they will not cancel problems of one another, they will amplify the problems of one another. So this is my stance on it. So it's not entirely useless, it's just that you must find use cases that are very similar to web, for example, web scraping. Probably, yeah. Like searching online. Probably, yeah. But solving, let's say I say, okay, I have this cold fusion reactors, but plasma. I cannot contain it for more than half a second using my algorithm. This is my algorithm. Hey Devin, fix it. My plasma should now be manageable for 10 seconds. Well, this will never happen. It has never seen any example of plasma controlling code. So no, it will just fail.

This is really interesting then. You make it sound very conceptually easy. It's like, well, it's an LLM. It's doing a lot of, I guess, test time, thinking it's using some tools like APIs. But then in some cases, like you mentioned, you start off with like a shopping example and it seems to work okay there, but it's not going to replace engineers or doctors anytime in the near future, which I think is probably quite reassuring for a lot of people who are worried about their job being taken away.

Yeah, it's a good news for many who don't understand how the thing works. If you understand how the thing works, you are not worried, but if you don't, yeah, it's good news.

Okay, but I guess one of the sort of well hyped things over the last few months has been the idea of these deep Research tools. So it's like AI doing PhD level research. I don't know how much of that is marketing hype and how much is really possible.

Yeah, well, first of all, this PhD level AI, it's a new marketing term that they pull off their head. I say they, I mean the totality of venture capitalists, CEOs and influencers who wants to, you know, this hype to continue forever because they have investments and like, so it's kind of a symbiotic relationship between them. So now we talk about this PhD level AI. First of all, what does it mean, PhD level. So PhDs. I was a PhD, okay. I worked on a very, very tiny direction in the overall AI, which is called game Theoretical Multi Agent systems or computational game theory. So it was very, very narrow direction. And in the world, maybe a hundred other scientists worked on the same problem as myself. And when I read some new paper on the domain, I knew all the people on the author list because we were the same kind of group of people working on this very narrow domain. So this is what the PhD does. PhD doesn't launch spaceships to Mars every day, or, you know, we don't invent new physics every day. We work on a very narrow domain. And on its own, this very narrow domain, it's not particularly difficult. Like, I mean, you just need to know a lot about this specific domain and you will feel comfortable in it. So this is the definition of PhD level work, but it doesn't come from, you know, pre training the LLM on the web data and saying now that you can do this very narrow research. Show me one example where, you know, they, they took an LLM, they said, okay, now you are a specialist in computational game theory. Write me a paper that solved this problem. No, no one can solve for a decade. I doubt very much that it will be done. Okay, because again, an LLM cannot do anything that wasn't in the training data. It can kind of extrapolate a little bit. For example, let's say if in some scientific domain some algorithm was used to solve some problem, and you, you work in a different scientific domain, but your problem is somehow similar to that solved in a different domain. So if the LLM was trained on this former, then for the latter it will be able to say, oh, you can use this algorithm to solve your problem. Why it knows is because your problem is similar to that problem. And this similarity comes from what we call embedding, cosine similarity between embeddings. So if your problem is embedded in the same embedding space as that problem. Of course the suggestions will come from the solution for that problem. But if you work on something very narrow and very only hundred people work on it, the chances that this PhD level AI will suggest something that a PhD can actually use variously or even not. So maybe what they call by PhD level AI is maybe there are exams when you want to become a PhD student, you need to pass some exams. And these exams include the knowledge from different domains. Like for example in computer science it's about compilers, it's about like proofs of theorems, like it's about creating, let's say systems that don't fail. So there are algorithms that introduce redundancy in how you encode information. So there is a lot of knowledge that you should have to be admitted to a PhD problem. And to test you that you have this knowledge, you probably you will pass. Well, I passed some exams, so maybe this is what they say, maybe their LLM is good at solving these problems, like this exam, problems that PhDs take. But again, as I said, if you have today a large enough model and you use reinforcement learning and you have the problems and solutions for these exams, you can fine tune the model to do it and then you can call it PhD level AI, but it will not do the PhD work. And by the way, in my case the PhD work was still kind of, it's a creative work. So I advanced the field by proposing new algorithms, like new solutions to existing problems. But There are many PhDs that they don't produce any new information. They just aggregate a large body of disintegrated pieces of information. Like just recently there was a woman who worked on, I think the use of smell in literature to describe racism. And when they describe people of color, they used like smell references and so on. It's a very important work and it really kind of very fun to read. And for her to be able to write a thesis, she should have read plenty of books and find pieces of where smell was used, how, in what domain, in what stage of our history, like before revolution, after revolution. And so this is a very important PhD domain, but it's so narrow that maybe only 10 people in the world will read it and say, yeah, this is really good work. This is worth like giving the PhD title to the scientist. This is not something you will ask an LLM to do. Well again, if you train it on all books of the world, it might write you something, but it will also hallucinate a lot and you will have to reopen those books. Reread those passages and make sure that it's actually the real deal and not something that it just imagined because you sample tokens from a distribution.

So it sounds like there's a big difference then between like the. The PhD idea of we're doing novel scientific research and the PhD idea of we're just doing something that's deeply technical and requires a lot of domain knowledge. Then. So I guess you need to be very clear on what kind of problem you're trying to solve before you start diving into making use of LLMs for that sort of thing.

Yeah, just recently, Google, I think they released a kind of an assistant for scientists in their scientific work. I think this is where LLMs will really be super helpful, because as I said, there are so many publications right now, especially if you work in computer science, in AI. Like, thousands of new publications are put on Arxiv Weekly. Impossible to read all of them, never. But maybe some of them contain some algorithms that might be useful for your research. But you don't know about this because you're incapable of reading all of it. And even if you could read all of them, you cannot keep all this in your head. Like, it's not like a toolbox where you put different stuff. And eventually, oh, well, some scientists, like the great scientists, this is how they operate. They have so many algorithms and different approaches to solving different problems in their head that when you present them some new problem, they already see what kind of tricks they would try in the first place to solve this problem, because they have a lot of stuff in their head. But for most of us, we remember something that we used frequently. Once we don't use it frequently, it gets lost. So in this case, the LLM, it's a very, very good assistant because it remembers everything. And even if it hallucinates, it doesn't matter because you are the scientist. So you say, okay, this is my problem. I already have this algorithm. But I would like to prove that this algorithm is efficient. How do I start to prove something like this? And the LLM might say, oh, yeah, well, I saw this article and it was a similar algorithm to yours, but apply it in a different domain. And this is how they have proven that the algorithm was efficient for the problems they applied it to. And you will say, show me it. The LLM will show you the algorithm. You will not just copy, paste and publish. This is not how scientific work works. So you will take it, you will implement it, you will see whether it actually applies to your specific case. And if you validated everything and you consulted with colleagues and everyone says real deal, you publish. So this accelerates a lot the research cycle. But again, right now there are plenty of low hanging fruits like this. Eventually everything that was easily transferable from one domain to another will be transferred and then we will confront again a situation where there is nothing new to do. We already transferred all possible algorithms to all possible different domains where they can be applied. Now what do we do? And now we once again have to start to do science from the basic principle of how we did it in the past. So for some time scientists will benefit. Maybe this time will be a decade, we don't know. But at some point we will reach saturation where everything that was already invented by people before us has been used in all domains that we care about. And we now have to start thinking once again for ourselves.

Okay, it certainly just seemed like quite an exciting intermediate time though, just being able to transfer like best practices from one domain into another. Because I'm sure there are a lot of domains where they could do with being like think about social sciences, but they could probably do with being a little bit more quantitative or technical in some cases because it's just not been explored enough yet. So yeah, definitely some exciting times in the near future. But I guess you just gotta be careful about what is the right use case for the large language model.

Yeah, and just remember it about very funny case. A couple of years ago. Well, I say a couple, maybe five years ago, people just on Reddit discovered that some group of scientists in healthcare, they needed an algorithm to actually optimize a function. But because they weren't really, you know, technical, they invented gradient descent and they published it like, okay, we propose this function and we also propose a new algorithm how we can optimize the parameter of this function. And they just described like basic gradient descent and published it as if it was the discovery of our lifetime. So at least with LLMs, the LLM might say, well, no, it's not a discovery, it's already been implemented in plenty of domains.

On the one hand that is very cool that they invented gradient descent because it's obviously an incredibly important algorithm. But yeah, I can see how that might be a little bit embarrassing. The fact that I guess did it get through peer review then and it.

Was, I guess so. Well, it's because it's a domain where they don't usually do any math, like it's maybe some social research, but they needed kind of quantify some results. So they didn't know how to do it and they Invented it.

Nice. That's very funny. All right, so before we finish, I'd like to hear a little bit about what your tech stack is. It seems like there are so many different tools now for building with AI. So just tell me about what your favorite tools are.

I don't consider myself as a software engineer, so I'm. I know how to code, and in my book I put a lot of code and I am quite confident that this code is good. Why? Because I tested it with different language models and asked it to simplify this code and make it as simple as possible, but at the same time keeping the educational aspect. So it shouldn't be just one liner no one can see. It should be something that the reader can read line by line and see the logic. So I can write such good quality code, but I don't use, for example, I know that many people today use Cursor or VS Studio VS code and I use regular text editor, that is sublim text. I've been using it for a decade now and I'm quite satisfied you can run code in sublim text and you can debug in it. So this is my kind of a go to place to write code. And also, for example, if I work with an LLM on coding. So I created a script. For example, let's say I work on a web app, okay? So for example, for my book, I needed to create a website which is interactive, that shows some very interesting animation where I applied a machine learning algorithm to turn elements of the illustrations. Like it looks very fun. So what I do is that I created a script that takes all the files that you work with in your project. For example, if it's a React project, there are some standard files and standard kind of directory structure that you should respect when you work on a React project. So this script, it takes all the scripts, CSS style files and HTML files and just concatenate them in a text document. So next time, if I want to quickly modify something in this app, I take the content of this document, I paste it into an LLM and I say, okay, improve this or replace this with that or add this additional element on the web page or make this button work in this different place. So I just describe it. It says, okay, cool. It writes me the full change. And then I say, if I really don't want to think like do something very fast, I say, no, write me the modifications in the find this replace with that format. And this is like you need to use brain. It is just say, okay, find this replace with that. So you search in your original code, you find it, you replace it, eventually you replace everything. You compile it compiles code. If it doesn't compile you say well this is my error, fix it. And eventually you get to the result or not. Because as I said, if your problem is something out of the distribution of the data that was used for training, you can try very hard and not get to what you want. But again, the good thing is that now we have alternatives. So for example, I today my go to model for coding is openai035. So I have this paid plan for $20 a month and it includes I don't know how many 10, 15, 25 interactions with Mini. I then let's say I exhausted those introductions and I didn't still found a solution. Well, the LLM didn't give me a solution. So I can take this code and go to my second go to model and today it's the GROK on X. GROK is spectacular how fast Elon Musk managed to train such a good quality model. So I don't want, you know, wander from one to another, you know, randomly. My principle is that if after three back and forth, the problem is still not solved, you have two choices. To start from scratch with the same model but explain the problem better or you can go to a different model. And very high chances the different models will just give you the solution right away because they were trained on different data set and especially fine tuned on different problems. So maybe some model was fine tuned on a problem similar to yours and you will just get the output right away. So for me it's like three back and forth. I still see we don't go anywhere. It says okay, this time I fixed this problem. Okay, it will work. You run it doesn't work well, you say okay, no, doesn't work. So if the third time didn't work, you say okay, I restart or I go to a different lm. And now I have plot, I have GROK and I have OpenAI. Like all 3 or 3 mini or even all one is good too. So like a very vanilla case. But I know that many, many developers use cursor or use VS code. And if I were a software engineer, if I did this 24, seven, this would be my bread and butter. Of course I would use these tools. But for my cases I test new algorithms, I build websites for my books myself. The way I use it is simple. And it's also simple for anyone who never use these tools, who never program it at all. Like never wrote code by themselves. If you show these people an id, like a state of the art id, like VS code, it will be hard for them to just to set up the environment for a new project. People who are outside, they would prefer to work this way. Like you just put something in an LLM, you explain what the problem is or what the change you want. It writes you the updated code, you put it where it works and this is how you proceed. But eventually, if you want to be more and more productive as a software engineer, you would gradually move towards this specialized ide.

That's really fascinating that you say you get away with just like, let's use a stripped down ide. I mean, you mentioned sublime text, which is like pretty sort of basic stuff, and then just have this AI assistance. But the trick there is to use lots of different models that are going to give you assistance in different ways.

Yeah, don't spend much time with one model. Test all of them very quickly and there are high chances that one of them will solve it right away. Or you can spend the entire day trying to squeeze something from one model and never get anything that works.

So final question. I'm always in need of new people to follow. Whose work are you most excited about at the moment?

This is a hard question because I rely a lot on my X bubble and my X bubble is like it's created automatically, so I don't follow anyone particularly. So of course there are people that everyone follows. For example, Andrej Karpathy, he's the one who worked with language models or neural networks before everyone else. Even I, I was still in the classical machine learning when he already trained language models from scratch. So Carpathia, of course everyone follows him, but you heard of him and everyone heard of him. So. But for the rest, as I said, like I trust my X bubble and by the for now, it doesn't disappoint. I see all the important news for me in my bubble, so. Yeah. Well, I recommend you to follow me and if I react or repost something, you will see that this something I think is worth spending time at all.

Right, so just following everyone with the variations on the name Andre. Nice. Okay, cool. Wonderful. Thank you so much for that. I feel like I learned a lot in this session. Really great stuff. Thank you, Andre.

Thank you very much, Lucy, for sharing.