What AI Engineering Really Is? Chat with Staff Engineer at CHAI

If you want to learn what it takes to become a successful AI engineer, then this video is for you. In this new series of videos, I'll be talking to experts from industry who are building LLM based applications and serving them to millions of users. In this first video I'll be talking to Dr. Alexander Dawood who is a staff engineer at Chai AI which is a leading platform for Conversational Chat AI. In this episode we talk about what Chai AI does. We also talk about his own journey into generative AI. We discussed skills that are needed to become a successful AI engineer. We also touch upon MLOps evaluation and model monitoring in production. Special thanks to Chai AI for sponsoring this conversation. Enjoy the first episode. Hey Alex, welcome to the Prompt Engineering podcast.

Hey, thanks for having me.

First I would like you to talk about the company mission and then we'll go from there.

I work for a company called Chai. We are in the business of building an AI platform. And what I mean by this is we are trying to bring together community LLM creators and LLM enthusiasts and deploy their models at scale to our millions of users on our Chai app. And so we're trying to create this great ecosystem whereby users of the Chai app can talk with models and bots that are created by open source community members and professionals in the LLM community. So our, at the moment we have our Chiverse product which is available on a website and also via Python package whereby these LLM developers can submit their models. And our Chai app is also available on the on the App Store and the Android Play Store for, for anyone to download for free and chat with these models. So this is our mission at Chai is basically just to bring together these two people who love LLMs on one side, talking to them and who love building them and create this amazing ecosystem.

You basically provide a platform so that people who are training these amazing models can host them and then you also provide a platform for user to interact with them directly, right?

Yes, exactly. So we, our, our expertise is basically in the serving of these models and the recommendation of the models. So ultimately what we're building is some sort of sophisticated recommender system where whenever a user has a conversation on our, on our app, they can get routed to the most appropriate model which is, which is good for that conversation. And so we've, over the course of the past year we've had about 6,000 unique language models submitted to our platform. And over time we've been optimizing our recommender system trying to get the best out of it to serve the content that users actually want to see.

Yeah. Before talking about the platform or going more into details about the tech, I would also want to learn more about your own background. So you have a PhD in mathematics, and how does that transform to both, like training LLMs as well as serving them? So we'd love to learn a bit more about your background.

So, as you say, my training and my academic background is in mathematics, so I did my undergraduate in mathematics, I did my master's at University of Cambridge in mathematics as well. And there I kind of developed an interest in just research and just trying to push forward ideas and things that I found interesting in mathematics. So I did a PhD and a postdoc in London at King's College London. And over there I worked on number theory, which is a topic in pure mathematics. Together with my supervisor and collaborators, we managed to prove a conjecture which was open for about 35 years in our field. And once, once that had been done, we kind of got the papers out and the articles out, I started to think to myself, what do I want to do next? And actually, during this time and for a long time, actually since I've been about 11 or 12 years old, I've been writing software, both in an open source fashion, as a contractor web developer, I built video games. I've hacked around with kernel drivers and things like this. And I've always wanted to just do this at a, at a professional level as well, because as a mathematician, you're mostly just with pen and paper, writing down theorems, proving theorems. And so for me, one thing I always really wanted to do was just, I want to write software at a massive scale. I want to serve millions of users, you know, possibly even billions of users with software that I write. And so I looked, I tried. When I wanted to transition to industry, I, I, I looked into various different roles. I landed my first role at Seamless Capital, which is the sister company of Chai, which is a quant trading firm where we, where we built algorithmic trading platforms to trade different markets. While at Seamless, I got chatting with Will. He said, hey, what do you think of my new company? Like, we're working on language models. What do you think of this? And slowly but surely, I kind of got more interested in these language models. It was brand new at that point in time. This was before ChatGPT released, and so it was kind of still nascent. I know language models have been a thing for a long time, but like this big boom that we're seeing, these Days did not really start until the past couple of years. And so right about that point, I got chatting with Will about it, the founder of Chinese Chai and Seamless, and I, I, I ended up joining Chai and so I felt like I could really contribute with my existing software skills, et cetera, because I used to build websites and backend services and things like this. But also as a mathematician, like a lot of the stuff in, in the realm of AI and machine learning is, can be described in a mathematical fashion. And so this was very appealing to me. Like this, like my background was, seemed very pertinent both as a software engineer, as a mathematician, to come into language models. And so that's, that's how kind of I got into it. But just for me, it's like I know whenever I discuss with people about this, they always feel like it's a big jump from mathematics to software and AI, whereas I really feel they are intrinsically linked. For me, like when I write, when I write software, I feel like I'm just the same as when I'm writing mathematics. It's, it's, in some sense it's the same, it's the same kind of feeling.

In order to become an AI engineer in the era of LLMs, do you think you, you need the traditional machine learning skills or not?

It's a very good question. I think it can do no harm to have those skills. Definitely. If you're an experienced ML engineer or AI engineer in more traditional methods, then it's certainly a big advantage. You're just familiar with ML and AI. In the first instance. For me, as I said, my background was not in ML&AI and I seem to have gotten quite far despite this. I think it's just having a very open mind, being willing to learn about lots of interesting things, be willing just.

To.

Put aside your existing expertise for a moment and just say, look, I'm going to learn something new here. And then when you've learned that particular topic, you can then draw on what you actually knew beforehand. And in some sense you've, you're in a much stronger position in that way. And so for me, like not having done much ML previously at all was never a hindrance to me in this particular thing. I got, I got used to dealing with language models. I learned a lot about statistics and ML in the first instance and then I was able to draw on my skills as a software engineer previously just to be able to then deploy these things at a large scale, if you see what that means.

So based on your experience if you were to start from scratch, what type of skillset you would think are necessary to become a successful AI engineer?

So I think just on practical terms it's very important to know how to program ML engineering, particular LLM engineering. It's heavily Python based. If you know Python you're in a very good position. Like I think the vast majority of people in the industry either use Python or they sometimes delve into C if necessary. I think 90% of roles out there that you would potentially apply for are Python roles. You need to have a good grasp of that language and then just in terms of like things that are very useful to know to get into these sorts of roles. It's just like all the open source packages that exist out there, it's great to be an expert in them, like the Hugging Face Transformers library or other such things. These particular packages, if you get very good at them, they will transfer. It's just a, it's literally the kind of packages we use in, in, in, in production often. So knowing those sorts of things is useful as well. That's kind of on like the very practical perspective. At a slightly higher level it is useful to have a grasp of statistics and, and the, and you know, a more theoretical basis for understanding why these models behave the way they do, why tuning certain sampling parameters might lead to like a model being one way or another, how you can compare to different models, metrics for models and things like this. It's all very useful to have at least a good understanding of, of statistics for example.

Yeah, what I have seen a lot is that people learn these packages or libraries and try to either fine tune a model and as you said they don't really have a deeper understanding of what these hyper parameters are going to affect the training. Right. So, so usually like people run into issues I think.

Yeah, I think you can get very far without a very, you don't need a very deep knowledge but it's very useful just to say hey look, I'm deploying these models like what, what does it mean to change the temperature and the top P and the top K and these sorts of things. What does it mean to clean the data set in the, in the beginning? Like these are very practical questions that people, people have about machine learning which is just like what, why is it important to clean a data set for what does it mean the model will overfit if I don't for example? And these sorts of things. Just having that knowledge in mind is very useful.

I think it's good that you brought up the Dataset. What we have seen with these LLMs is people are inputting a lot of a huge quantity of data and they expect that these LLMs will learn just based on the shared quantity of data. What do you think? Like, is the scale of data necessary or you really need to pay attention to the quality as well, not just the quantity?

I think it's a very interesting question. It depends really what you're trying to train. For example, if you're trying to train a foundational model, I feel fairly convinced a large amount of data is necessary for the model to have a wide range of capabilities in some sense. But if you're trying to fine tune for a particular task, I think it's much more important that the quality of the dataset be there rather than the quantity. And in fact, I remember we shared internally CHAI a paper that came out last year, I think, which said you only need about a thousand very clean rows of data to try to fine tune a conversational model. So if you, if you very carefully pick the data you, you feed into the training algorithm, you're more likely to get a better outcome in terms of fine tuning. And I think this is seen quite frequently with the models that we get that get submitted to chai. We provide an open source data set based on conversations that users agree to share and people like to use this data set and they very carefully filter and clean the data to quite a small amount of rows, not a ridiculously small amount, but still. And then they fine tune their 7B or 13B open source models on this dataset. And so I think it's very important to have this clean, high quality data set in the first instance. And it's often the case that less is more.

Yeah, that's great insight. So let's talk about, I think more on training the models and the role of CHAI as a platform itself. Can you give the audience an overview of what exactly happens on the platform?

Sure. So on the platform, people from the open source community open our Python package or our website and they simply enter a URL that points to a hugging face repo, for example, and we take that model, we quantize it and we serve it very quickly in under two minutes to users. And so in terms of training, what we do on top of this is that we have a reward model, for example, which allows us to serve the model in a more at a higher level in some sense. So we take four, for example, four samples from the user submitted model, we score the responses and then we serve the best of Those ones. So we typically do a lot of training of reward models and classifiers internally to try and boost the performance of these models. This is a very, it's a very simple technique that does actually quite well in this, in this domain. And then on the other hand, what we also do is we train recommender models in the sense of when people have a large amount of, when we have a large amount of models in, on the platform, a pool, let's say a thousand models, we train a recommender which says, okay, given this conversation, this particular model that has been submitted by this user is the best model to serve for this conversation. And this, in some sense it's quite a lot like the mixture of experts architecture, except the mixture is happening externally to the model rather than internal to the model. So the most well known one these days is Mixtral, which is this open source mixture of experts models put out by the Mistral guys there. The expert, like the routing logic, the recommender logic is baked into the model file itself, which is very useful in particular domains. But for us, when we wanted to glue together quite a large amount of models that have been submitted by people all over the world, it's much easier to have this recommender model external to the architecture. And so we spend a lot of time training these recommended models, learning how to build better ones, also determining what kind of metrics tell us what's a great model. Like this is one of the biggest problems we have as well. The benchmarks that exist that people typically use are very good for assistant based tasks, but they're not great for conversational and entertainment tasks. For example, we find that MMLU and these sorts of things, they don't correlate very heavily with retention and engagement on our platform. And so we do spend a lot of time just trying to train models to understand how models perform as well, like good metrics and these sorts of things. And then we try, what we try and do is we try and package all of these ingredients together in a, in, you know, in a good framework which allows people to connect their model to users in less than two minutes. And so there's a lot of live training of reward models going on. These sorts of things require quite sophisticated AI pipelines to serve as well.

Yeah, I'd actually like to touch base on a couple of points that you mentioned. One kind of to have a better understanding of the recommended model. Right. But before that, let's talk about how do you evaluate these models that are trained. As you said, the traditional benchmarks are not going to work for this specific use case. Right. So how do you even come up with an evaluation data set?

In some sense, we just do it live. So we have 4 million users on the Chai platform every month, about 800,000 per day. We take the models that are served to, that are deployed to the Chiverse platform and we serve them directly to users with some checks, like sanity checks and stress checks and these sorts of things. And then we ask users once in a while, okay, what do you think of this model? Do you like this model? Is it any good? Or do you prefer this model, for example? And slowly but surely you build up a data set which is kind of generated life and then you can use that to say, okay, this model performs better than this model. Maybe this model should be served for this conversation, for example. And this data set is constantly being populated in our backend. So every time a model comes onto the Chiverse platform within, within minutes, it's already being checked against other models whether or not it, it performs better than that model. It's also checked just in terms of human feedback. We, we do ask users, what do you think of this model? Can you write a little text about this model, for example? So one like 0.5% of the time we might show this, this pop up and say, look, can you give us some feedback? And then we take this feedback and we send it back to the model developer and say, look, this is what users think of your, of your model. People have found that very useful in the open source community. They having actual human feedback, the scarce resource oftentimes, yeah, you often have to pay labelers and things like this. So we're in quite a unique position to be able to take the CHAI users on the platform and understand what they really want in a model and just tell model developers, hey, this is what they want. You know, your model's performing really well or it's not performing so well, not oh so great. Maybe it's too repetitive, for example. And so slowly but surely we build up this quite large data set of like thousands of models with thousands, like millions of user recommendations and feedbacks. And this is what we train our recommender system.

Okay, so the recommender system itself is LLM based or is it a hybrid approach between.

Yeah, we're experimenting with various different, like we haven't settled really on what the optimal way to do it is. It's a very good question. I'm personally, I'm more involved on the, on the MLOps side of things, deploying these things, making sure it scales well. Our internal AI researchers are, you know, going crazy trying to find what, what, what makes a great recommender system for, for LLMs. Is it another LLM or is it like a simple clustering algorithm? You cluster conversations, you score the clusters and you assign a model to each. Like you, you, you point each model to the correct cluster it deserves to be in. If, if that makes sense. There's lots of ways to go about this. There's, there's lots of, lots of interesting research to be done, I think. Yeah, we, we, we've only started in the past few months with training these models. It's performing quite well recently. But yeah, and, and one thing I do want to mention as well, which ties in with this recommender system is the way this was born was we discovered that if you serve a blend of models, like an external mixture of experts, you, you just increase performance quite significantly. Um, and so what we did like in, in early last year was we served these models a few, four or five good models that we had randomly. So we would just sample uniformly randomly. At each point a user sends a message. We say, okay, let's serve this model, let's serve this model, et cetera, just randomly. And that performance, like we had a huge performance boost just from this. And so the simple function of uniform sampling was already good enough to improve upon those individual four models by themselves. And so we said this is really what told us, oh, the recommender system, something we should actually build because if you can actually accurately determine which model to serve for that message, then it'll have an even better performance than just uniform sampling.

Interesting. That's actually a great insight.

We actually wrote a paper about this and I think the one thing I found very cool about this was that it looked like we were actually creating a true ensembling of language models whereby the output from one model would be fed into the context window for another model. And in some sense it would make the outputs of those other models more diverse because they would be sampling tokens which they don't usually sample in some sense because the other model, so it's a sample that took.

So basically you stack different models together and. Yes, yeah, yeah, that makes sense. Let's, let's talk about serving these models, people. Start with fine tuning these models. How do you serve this at scale to millions of users? What would be a good architecture that you would think about? What are different things that you need to consider from a business point of view as well? And then probably at the end we will talk about if somebody wants to get into the MLOps side of it. What are the type of skill sets that you would need?

Yeah, yeah, it's a very broad topic. It's very interesting the sorts of things you have to do to serve these models. Even at the scale that we do, there's a lot that goes into it. Just fundamentally the biggest barrier to serving language models at scale is the cost. Like it is extremely expensive. GPU costs form the vast majority of our cost of the business simply because they this it is quite a expensive resource. And so what we have is a GPU cluster. So we collaborate with CoreWeave. CoreWeave provide us with a cluster of about thousand GPUs which are constantly active, by the way, that serve the models. This cluster runs on Kubernetes effectively. And so anyone who is familiar with Kubernetes is probably quite well placed to just deal with infrastructure problems and architecturing surrounding serving language models. It's certainly not necessary to serve them on Kubernetes, it's just with the amount of volume we get, the auto scaling in terms of like being able to spin up new instances with a new GPU fresh quite rapidly is extremely important because we'll get peaks of traffic during the evenings, for example, in the mornings. And being able to very quickly get our models live, that's quite important. And that's where Kubernetes is very useful. Or just the software layer on top of Kubernetes is very useful. And so that, that for example is one thing that helps. Then on the other hand you have the backend for whatever you're serving. So Kubernetes is kind of like your infrastructure, your resources that just by itself, all it is is it's a language model at scale, but you have to do something with it. And so we built, we built our own custom serving framework and deployment framework specifically for Chiverse, whereby we take a model, as I say, we quantize it and we do checks on it and we deploy it very rapidly. And that's, that is all basically standard web services in some sense. There are probably other ways to go about doing it. But just in terms of having experience in backend development for websites, apps and these sorts of things, that's very useful. Being able to ingest requests from users, process these requests and then send them off to the kubernetes platform, for example, like these sorts of architecturing designs and decisions, they're all quite familiar to a web engineer. And with all of this, you know, it comes your, your standard questions about what database to use and these Sorts of things at scale. It, the database question is a very pertinent one because serving, serving and storing conversations, it's, it's a massive data set. And so this data set itself, you're ingesting nearly terabytes of data every day. And so you have to make a very good decision about what database to serve. And these days there are quite good ones like bigtable on Google or hadmop I think it's called on Apache, hadmob, I can't remember the exact thing, but on AWS there's an equivalent based on an open source table. And these are like white column databases which are used for these AI at scale and data at scale techniques. And so having familiarity with that sort of thing as well, just serving conversations, it's very important. If you think of the big conversational platforms like WhatsApp and Slack and Discord, those, those guys spend a lot on trying to understand, okay, how do I store my data, how do I ingest it and how do I build a pipeline around this to serve it into my. Maybe I have models which I need to, you know, train, train something on, if it's what I mean. And so again, like just having backend experience is quite invaluable there.

Yeah. A couple of other points I would like to touch on. One is what do you think how different serving these LLMs are from serving traditional machine learning models? And the second point would be what do you see the role of monitoring these models in production? How do you do that at scale?

Yep, that's a good question. So one of the big differences between traditional machine learning models, although I don't have a lot of experience with them, so I could be misspeaking, but my understanding is that the inference time for language models is quite a lot higher than the inference time for other models. Smaller models, like if you have a fewer number of parameters, it infers faster and that makes them significantly easier to serve. In fact, at Seamless Capital we used to serve models which would serve, which would serve a response in like less than a millisecond, for example. Like it's, it's, it's quite a, it's quite a simple model, right? And so it's very easy to serve. You just stick it on a web service, you don't need a gpu, you don't need anything. You just stick on a web service and you ping it and it, and it responds. Whereas a language model, even on the best GPUs will take five to six seconds to infer. And while that doesn't sound like a significant amount of time. It compute in like in, in, in tech terms, that's a long amount of time to be waiting on a request because that you typically, or not typically, but sometimes you daisy chain services whereby one service talks to another, another one talks to another, and finally that service talks to the, the end service, which is the GPU in this instance. So if the end service is taking about four or five seconds to respond, each of these services is waiting to respond for five seconds. And that causes a buildup. And so you have to start dealing with queuing and auto scaling and concurrency and all these sorts of great networking things, which fundamentally they're not difficult to solve, but when they go wrong, like your entire infrastructure could go down because you've mishandled a queue at one point in this daisy chain, for example. And that, that brings me on to your other point about monitoring these services. It is actually, it's actually very difficult and also very easy to monitor these services. And the reason I say that is because of this, this nature of language ones where they take time to serve and sometimes they will error maybe 1% of the time they will not output any tokens because they, you know, for something like it went out of memory or for example, something like that, you do have to be monitoring very carefully. Okay, what's going on with my cluster? Is it, is it healthy? Are all, is my request rate, Request success rate 99.99% or not? And if you're not paying very close attention to this like it, it go, it does kind of degrade and go goes out of control. This is something just from experience that, that we've noticed is that like oftentimes GPUs will go out of memory and for some reason that service is not cleaned up. And then it starts to cascade because more and more services start to, to go wrong. And what we learned very quickly from this particular instance was even if the service doesn't clean itself up, if you have an external service which is just monitoring the cluster and says, okay, let's, let's just clean these, these guys up, you actually keep it very healthy. And so when I first joined chai, a lot of the challenges for me were being on call and having to deal with outages because of like, know the GPU provider has not enough GPUs, for example, it's not even a, an outage in terms of like something is wrong. It's just, there's just not enough resources to go around. But also sometimes they would have outages Themselves we would have outages, et cetera. And so like a big challenge was dealing with being on call. But I learned very quickly if I just program solutions to it, if I automate all of this stuff, I'll serve myself very well. And so at the end of last year I wrote a few services internal to Chai. This is our own, our own code which just monitors our cluster, our GPU cluster, and does some very simple things like checks their latencies, sees if any particular models are performing quite poorly. They have low late, they have high latency, for example, or they're returning errors quite frequently and just kill them, just kill the service. Because sometimes the Kubernetes cluster will not kill it itself for, for whatever reason. And if you just instruct the cluster to kill it manually or rather automatically via this, via this service, the, the health of the cluster is quite pristine. And so it's, it's really quite amazing that I've, I've significantly reduced my own call time just by writing about 600, 700 lines of Python, Python code. And so that's why, that's when I say it's quite easy in some sense because if you have the discipline to just say, okay, what's wrong with the, what's wrong with the system? Can I diagnose the problem? Can I write some code just to just to, you know, automate this toilsome job of like constantly having to be on call, for example, that actually served me quite well. I think also like this is relative to our scale as a company, we are quite young as a startup as well. And so these sorts of solutions probably are not what would be employed by larger organizations like, you know, Meta and Google. They probably have far, far more sophisticated monitoring pipelines, alerting pipelines, et cetera. But for us to be able to write, you know, in one day, 600 lines and then just, you know, reduce my on call on the weekend by about four or five hours, for example. That's, that's, that's, that was certainly very good.

Yeah. It kind of brings me to another topic that I wanted to discuss was the quality of production code that you write. Like what's the difference between research code and code that you can actually put in production? Yeah. And how you minimize catastrophic failures and bugs.

I think for us, one of the reasons why we actually have very few outages is because we have tdd, which is quite central to our coding culture. TDD is test driven development, whereby writing tests is what drives the software you're trying to write. So you understand the behavior of the system you want to build, you write some sort of test for that and then you implement it. And you only implement as much code as necessary to make that test pass. And then when that test passes, you're free to refactor and tidy up the code, for example. And so for us, testing is kind of non negotiable. Every single line of code we write, we typically expect there to be a test. And I don't mean that in a dogmatic way. I mean that more in a more wholesome way in the sense of whenever I set out to implement a feature, I would write what's called an end to end test, which is basically just test the entire behavior of the system. Like just write that test out. Don't write any code, just specify the behavior of your system. Write the test out. Sometimes the test will be up to 300, 400 lines long. And then when you've understood the behavior of your system, then you can start implementing the actual feature. It's very often the case that you'll come back, have to rewrite some of the tests. You won't get it right the first time. Like that's, that's very natural. But this discipline and just saying, okay, like let me specify my behavior in code and then write my, my, my feature, that actually does reduce the amount of outages we have. Because you can test, you can trust the code base in some sense. I know that I can go in very quickly if I need to implement a new feature which is a little small, but like, it's deep inside the, the code somewhere. I know because I have about a thousand to 1,500 tests in my code base. I'm less likely to break things. It's not necessarily the case that I won't, but I am less likely because I know that I've tested things quite well. And so for us at chai, we do place a huge emphasis on testing, especially because we're such a small team. Like when you were running with 6, 7 engineers, knowing that everyone else is writing tests means that you can iterate quite fast and you can, and you can move quite fast. Um, so that's one thing, that's, that's one thing that we do to maintain our coding coding standards. And the other is we just, we do adhere to clean code principles. We do expect most of the engineers who join CHAI to have a good understanding of what it means to write clean code, what it means to have standard software architecture in, inside the code. Like there, there are plenty of books and things like this which go into the details of like how, how one designs systems via domain driven ways, via test driven ways. And typically we will expect that engineers will have a good understanding of this stuff. And then we, we, we try to maintain that with every code that gets merged. We, we say okay, is this meeting the bar that we have as a team very often the case, you know, like we'll want to very move very fast and we're very happy to say okay, it's okay if you dent the car but not, not crash the car, right. So it's okay to come. And you might not really understand what a feature looks like in the first instance. You might want to prototype a bit. It's okay to come slot it in where appropriate with, with, with code which is, you know, definitely good but not, not, not what we would like to see to the final feature, for example, that's fine because people need to move fast and you know, they need to, they need to test things out. We don't need to be so dogmatic about every single line of code. But it's still the case that it's, it's expected to be tested for example, because we just, we do want, we don't want to break, you know, production.

I think that's, that's important specifically if you are working in a team, right, like as a solo engineer, you, you know the code base. But as a team you want to make sure that every code segment on snippet is actually tested. Right. That makes life easy for everyone. I would love, if you have any resources that we could share, I'll put.

It in the video just for as an example test. Growing Object Oriented Software Driven by Tests is a very good book for this sort of thing. Just in general, if you search TDD or Test Driven Development online, you'll find loads of resources. I quite like a lot of the old, old resources around Ruby on Rails, which was an, which was a web framework that was very popular in the last decade where TDD kind of had its boom with Ruby on Rails. And the Ruby guys have a lot of interesting things to say about tdd. For example. Another thing I'll mention is that what's often like a misconception in this industry is that it's very difficult to test machine learning models or just architectural pipelines because it depends on something which is like a model. But for us, like if you adopt this principle of tdd then you, for, you're forced to say okay, how do I test it? You can't say it's not testable. In some sense you do, you do have to find ways to test these things. And actually at chai, we do do regression testing of models, for example, so whenever we redeploy our default model, for example, we might upgrade some dependencies on the container which serves that model, or we might, you know, upgrade our reward model or something like that. We actually run regression tests where we look at the logits of the, of the model and against the new model. And we do KL divergence, which is a statistical method for determining whether or not two probability distributions have changed or how they differ. For example, it's a measure of difference in two distributions. And typically whenever we want to greenlight a release, we'll just say, okay, what's the KL divergence of these two models? Has it changed significantly? If so, then we don't release it. And it's been the case that we have caught bad deployments simply via this regression test. So we saw that the KL divergence was higher than we expected for this new model, even though it was mainly just changing some packages, like upgrading some packages. And I think we learned that one package had changed some behavior in the model itself when it inferred. And so those sorts of things are very useful to avoid, not necessarily outages, but like degradations and regressions of your product.

I'm actually really happy that you brought up the testing the machine learning models because there's kind of this sense in industry that, oh, you, you can do TDD with software, but you can't really do it with like ML models. Right? Yes, that's, that's actually really a great idea to kind of keep track of these. What do you think where the LLMs are going? And also if an engineer wants to get into LLMs or become an AI engineer, like what would be the future proof skill set in this new era?

Yeah, it's a very interesting question. I think what we're experiencing at the moment is a lot like when the Internet first started, I think just generative AI in itself is a huge paradigm shift in how we're going to be doing computing and just things in daily life. And so it's very hard to say, yeah, this is exactly the direction language models are going to take. I think we're only at like the nascent stage and it's very, I think things are going to like, things are changing very rapidly. I mean, not only in the language model scene, but also like in the, you know, diffusion scene, like image generation, like with, with OpenAI just announcing their new video generation model. Like these sorts of things are going to fundamentally change the face of the planet. In some sense they already are doing so. And I think the direction these things are heading is it's going to be so much more integrated with our daily lives. Not only just having a, like a chat app, you're going to just chat with some bots. I think it really is going to be at a point where, where you are inseparable from this. These things just like people are inseparable from their phones or their laptops. Like it's, it just is a function of how people live. And I think we already see this with like people in Gen Z and even like younger people who are just growing up in a world with AI. Like it's, it's just normal for them. You know the AI is a thing like for me in the Internet was a thing like I used to just spend all of my time on the Internet as a child. For me it was just the thing that I did. And this is what that this is something people are just going to be using as a fundamental part of their daily life. And so just in terms of where language models are going to go, it really like if Chai has anything to say about it, for example, like it really is going to be a like a large scale community collaboration just to build the best AI out there that's possible. Bringing together people who want to use language models as a means of making money, for example. That's one thing we're very excited about is like monetizing this in the same way that YouTube monetizes videos. Like if, if you as an AI researcher or ML researcher could, could monetize your content, you train models and you serve it to people. Why not, why not monetize it for example? And that's that that sort of thing is I think is going to, to be you know, a very fundamental part of like this technology. Like, just like the Internet and social media became.

Yeah, we went a little over time. But how can people learn about Chai? Are there any specific links platforms?

Yeah, yeah, you can visit us on our website which is chai research.com youm can also visit our developer platform website which is Chiverse like Universe but just first chiverse.com and we have you know, Facebook profile, LinkedIn profile, Instagram profile, et cetera. We also have an amazing blog where several of our engineers will write interesting insights that they've learned at Chai. I know one of our engineers has written a blog about TDD for models for example for, for, for machine learning. Models. I've written a couple of blogs over there and people, definitely people interested. You should check out our careers page on our website. We have plenty of open roles hiring where we, you know, we don't have any particular restrictions on the type of role we want to hire. You know, amazing people with amazing values, amazing talent. And if there's not a role that fits you in particular, just feel free to send us an application regardless, and we'll see if we can find something that's appropriate.

Thanks for sharing that. I'll put that in the video description. Yeah. So please do check out Chai research. And thanks, Alex, for taking time to talk to me.

Thank you very much for having me on. Thank you.