Automating code optimization with LLMs

Foreign.

Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly. Deploy your app servers and database close to your users. No ops required. Learn more@fly IO.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm the founder of Prediction Guard and I'm joined as always by my co host Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

Doing well today, Daniel. Just there's so much going on these days in this industry in terms of AI that just constantly learning new stuff and finding out who's doing what.

Yeah, it's almost like you need to optimize some things about your life to keep up. Would you say that's accurate?

Yeah. Speaking of optimization, I think that's a thread to pull right there.

Yeah. Yeah. So speaking of today we have with us Mike Bazios, who is the CTO and co founder at Turrentech AI. Welcome, Mike.

Hello, guys. Nice to meet you.

Yeah, well, we alluded to optimization and I know one of the things that Turing Tech is working on is code optimization with AI and maybe some people, actually probably a lot of people listening to this podcast are familiar with certain developer tools that are AI flavored, maybe GitHub, Copilot or something like that for generation. I'm wondering if you could take a moment before we dive into AI driven code optimization, if you could just help set the stage for those that aren't aware. What do you mean when you say code optimization? Why is it useful? How has code optimization been part of the developer life cycle for some time?

We are in an area nowadays that we see more and more applications consuming a lot of cloud resources and a lot of resources and everybody is trying to optimize the performance of their code. Now when we're talking about code optimization and in particular performance, typically people would like to optimize things like application being faster, maybe memory consumption. We know everybody complaining about Chrome using too much memory, for example, in the past, or CPU usage, which is connected very much with the energy that different software is using. If we talk about mobile phones application there, we would all like to be more efficient and consume less energy. And practically that's the area that we have been focusing in my research and our group, in our research and in.

The company, maybe you could talk a little bit about some of the history of that research. I'm sure that that has just like everything else been impacted by this kind of latest wave of AI technologies and Generative A. But I know that the company and yourself have been involved in research prior to and all during the development of these things. So could you give us a little bit of background on how you first started thinking about these problems and how it kind of developed over time?

Yeah. Code optimization is not a new thing. If you read research papers 20, 30 years ago, everybody would like to optimize and make the code efficient. There are a lot of tools like profile, there are, you know, that help developers find hotspots. The biggest problem in this area is okay, we profile our code, but how we can automatically improve it so we make it faster. Typically this is a very, very manual process and majority of the people that work on these areas are super specialized. And it's more and more difficult nowadays to find people that know how to optimize the performance of their code because the programming languages are becoming higher and higher level. Like now people write more in language like Python, JavaScript, TypeScript. So nowadays the real companies that deal with code optimization are companies in the lower from the hardware space like Big, Intel, Nvidia and those they write specialized software that take advantage of their hardware where they show the hardware it outperforms or technology companies that need the scalability. But majority of developers, developers will not necessarily bother about the performance of the code as an immediate first thing that they need to optimize. And we have tried and we have a platform that we helped practical engineers to automate this kind of process to make it easier for developers to identify place in their code that are slow and then to optimize it without necessarily having the knowledge that they need, but also do it automatically. Now the history of code optimization, as I said, it was very, very manual process at the very beginning. Very few people know how to optimize code for specific hardware. In the past, I guess they had to read books and compiler options eventually. Then you had better and better compilers. So you can use compilation option to optimize your code and tune the options of the compilers, etceter. And then you have a lot of profiling tools that help developers also optimize. But still all these processes most of the time has been manual or semi manual. Right. And that's where we see the advances of AI helping this process. And to give you a bit of context, how we started our startup, this started after we published a paper in 2018, I think that was in the foundation of Software Engineering conference where we showed that we could automatically help developers choose better data structures by taking a code, looking the data structures and optimize it by giving variations. For example, sometimes in languages like Java, for example, you don't need to use an array list. If you potentially can use a linked list in a scenario that may be better. So we try to do these small changes and show that we have good performance impact. But majority of people at that point were manually translating code like rule based transformation, like regular expressions. If I see this pattern, convert it with this pattern. So that is how people have been doing those things with code refactoring tools until the LLM came into the discussion.

Speaking of that kind of getting into this latest wave of maybe what I'll just refer to as developer tools that are AI related. Again, you know, people might be familiar with code generation type of tools or explainers or something like that. There have also been, I've seen kind of agentic type of tools that will like write a PR for you to do. Like you say, hey, I want to do this thing and there's a PR generated here. Given the focus on code optimization, could you draw out some of. Just for people that are maybe kind of getting into this, could you draw out some of when they would want to use this kind of tool versus some of these other maybe generative applications of generative AI to code? I guess, yeah.

How does it fit into that ecosystem?

So the way we present the code optimization tool currently is part of your CI CD process. So you make a pull request, then you will run some unit tests, you will have your integration testing. Potentially you'll have a scanning security scanning tool like snyk or checkmarks etc. And then the next step is depending on where your application is deployed to analyze your code and we tell you make those changes because you can have this 20% improvement in CPU and execution time, et cetera. That is the way we present it currently as a CI CD tool in the tool chain of the developer tools. However, if you think about the technology underneath and all dev tools, the way I see it, are going to be using AI and they will take advantage of LLM based solutions. From the moment you are using LLM, if you generate code or you translate code, it's a kind of the same kind of approach. It depends on the data that you will apply the LLM. So code generation tools, practically they see when you say for the PR example, the way those LLMs have been trained Is that they see that there is some comments below the code. So you say if I give you those comments, can you predict the code? So in the code translation where you don't know about the speed of the code, you say I have seen this C code and the equivalent of Python code and LLMs do it like for example Copilot or chatgpt. They can translate code if it's perfect. Probably not yet, but that is a fundamental technology. Code optimization is on the same set of tools, but it says this is a slow code that I have seen. Now I have seen a variation of faster code. I can recommend you this faster code but eventually you can expose the LLM like any other tool that is built on VS code or LLM based, you can expose it in editor and then the developers from our LLM will get suggestion for faster code. That doesn't mean that will be beautiful code necessarily, but it will be faster for the hardware that you need to run it.

I'm kind of curious as we're talking about speed, are there any other dimensions that are relevant in there that you guys are interested in that are either adjacent to speed or contribute to speed or any other characteristics that may not be directly speed specific but are things that you're starting to target or expect to target?

We apply multi objective optimization. So when you do this translation you can have different objectives that you try to optimize. Like speed is one factor, memory usage is another. CPU usage Typically there is a trade off between speed and memory usage. If you have more memory you would like to use it that increase speed. So our tool allow you and gives you different suggestions for what you need. But we see users, for example, we see this paradigm to be used in other use cases like users have told us I can improve the readability of my code as far as I know that I don't impact performance. For example, because there are managers in teams that they have five projects and they would like, let's say five projects with five teams with developers and they would like to guarantee that the quality across is good. So this AI approach, these tools, Artemis, our tool or other tools in the LLM space will be able to help with those. But the biggest problem of any LLM based tool currently, like Chris, you mentioned you are working for Lockheed Marting et cetera, is that the code generated by any LLM, it's not guaranteed that will work 100%. Also you need existing tools to check if this code is secure. And also you do not know if you will break your code. So still in the early stages of incorporating those LLMs. That's why it's also easy for a developer to give it as a suggestion right now. But I'm pretty sure like companies like that working bug fixing code, security checks they are using and they will be using more and more LLMs because they can train all the data that they have access so they can have competitive advantage.

Would it be a good parallel just to kind of draw to people's mind, like maybe something they've seen before in other domains? It almost sounds like a parallel to kind of a rephrasing type of prompt in an LLM where you might say like I do this a lot with emails or other things like here's my really bad email, make it flow better and sound better. Or here's my goofy email, make it business professional or something like that. You drew the comparison to maybe machine translation or something like that. Is rephrasing kind of a good way to think about this?

Let me give you a very simple example. Let's say you have an essay that you need to write and you have 10 paragraphs in that essay and you would like to have a version of that essay that is much better and you can get a better grade. Right now what we do is we will look all the paragraphs and we will provide you better variations. So we will get you version one with three changes that are applied by different LLMs. Then you would need somebody to grade that essay. So in this case we measure how fast the code is. So somebody says okay, you have a 70%, then we take that output and then we provide and give it back to the LLM and give you another version. So practically you start from version 0 of your essay, you apply the different LLMs, you get feedback. So it's like reinforcement learning, live learning and get different variations and that's score will be increasing eventually. And at the end you have a translated version of your original version that the LLMs and all those refactoring did, which is better on the metric that you have. But we in the platform test that the code passes, we compile it and also we run the performance. In this scenario we'd have a teacher that would grade. If you think how OpenAI and everybody has been doing their training, they usually get LLMs and then they use reinforcement learning and HLF and all those techniques. So we have done that in the code optimization setting. That's why we have had some impressive results in taking an open source library and we just put in a tool and then Suddenly optimize by 30% execution time without us doing anything. The models learn themselves.

This is a Changelog news break. You can add Meta's code Llama to the ever expanding list of code generating LLMs based on llama 2 code llama comes in three sizes, 7 billion parameters, 13 billion and 34 billion. And it comes in three different varieties, a general model, one tuned for NLP instructions and one fine tuned for Python. How does it stack up? Well, Meta claims it outperforms other publicly available LLMs, and it shares the same open ISH license as LLAMA itself, which is free for research and commercial use unless you compete with Meta. You just heard one of our five top stories from Monday's Changelog News. Subscribe to the podcast to get all of the week's top stories and pop your email address in@changelog.com News to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com news.

So Mike, you got into something that I'm super interested in in terms of how you're going about this problem, which you alluded to. The fact that you're using this sort of reinforcement learning loop or feedback loop to improve the performance of your tools and given that you and your team have worked with sort of code generation or code specific models for some time now, before we get into kind of some of the cool stuff that you've done specifically, could you just comment on kind of the state of code generation models that are out there on maybe the open source side specifically? But if you want to highlight any closed source ones, that's perfectly fine too. But from your perspective, how is that the ecosystem of code generation models changing and advancing and what is the state of it sort of these days?

I guess so I was one of the very first believers of LLM assistive code generation tools. Like I tried to get better access to GitHub, Copilot, even try to use GPT3 to see all this kind of technology. I'm very big believer because I have seen members of our team using LLM to build things much much faster to code. For example, we had the backend engineer, we needed a prototype for a front end. He just used one of the closed source LLMs and he could in one day he did a new UI that he didn't even know the language, he didn't know typescript. So I see that they are very, very promising. I see more usage right now for good developers that already know the basic of computer science. I'm not a believer of hey, you don't need to code you will use this LLM for you that are some videos potential they're over promoting that but if you are a good enough developer you know what you need and you use they can practically help you dramatically to build easy applications, to generate tests, to generate comments about your code and of course the performance of from our experience I believe GitHub, Copilot, ChatGPT are still outperforming the other models but we see more and more open source models starting to become very very good into the different languages. Like we have been trying llama 2, we have been trying codegen, we have been trying all of this and we even expose them to our platform so people can compare the results of those those tools will need to become a bit easier for developers and VS code tools to use because by default people this API exposure that some of the closed source models are giving is solving a lot of headaches for a lot of developers. That's why a lot of developers still have preference for those but definitely open source models they are very good and I see them becoming even better if they are fine tuned a specific language or a specific context. I'll give you an example. If we want to do translation from SQL to an SQL, let's say we want to optimize SQL queries that people are doing in their databases. You can take one of the open source models and fine tune it on SQL and you probably will outperform GPT 3.5 or 4 on the context that you have. But I'm truly very big believer. It's a bit of also a bit of psychology between developers that hey, but I am a believer that people that will use they have advantage over people that don't use those tools currently.

So that raises an interesting point. It's a little bit of a tangent, but you've kind of inferred it as this is kind of changing the way we humans are coding. I know you also Talked about whether LLMs would just write the code for us and the overhype certainly today about that. But it's kind of changing the way that we code as humans and it's extending our capabilities dramatically in terms of being able to reach beyond what we might have been able to do two years ago, for instance, do you see that accelerating when you're looking at the fact that if you looked at a traditional coding team a few years ago as you're producing your product for folks and you're starting to recognize that individual coders are starting to elbow their way out of their traditional swim lanes with these new tooling capabilities that you're providing, how does that change things? Like how does the market change that you're looking at going forward?

I believe dramatically and that we have tested this with our internal developers. I have seen dramatic improvement in productivity of developer without access to LLMs and developers that have like there are two things. One is you can make more efficient developers that everybody would like to hear, but from a more senior level manager level you do not need unfortunately, as many developers as you would need before. You cannot avoid this. And developers should think for me like, okay, I am competing with some other developer in my team to produce, let's say an API or produce a ui. If the guy has access to Copilot or chatgpt the other person doesn't have, I guarantee you, like with very, very big chances, the guy that has access to ChatGPT and those will outperform and have faster results. So it is like you have a very, very good assistant next to you that you should use, otherwise you are losing. That's how I see it.

Yeah. As a two second follow up to that point you just made when we start pervasively, because I think I've seen some stuff recently over the last month or two that like the majority of active developers out there are now using LLMs. So there's been really rapid adoption here within the developer community. So. So we've kind of moved very quickly from those who had versus those who didn't have into a world where everybody has. They may not all be using exactly the same models as things progress, but everyone has it. Any thoughts about kind of what that means? It's like you're democratizing LLMs across the population of developers and now they're competing. So it's kind of like me and my LLMs are competing against you and your LLMs as a developer. Just any thoughts, waxing poetic a little bit about what the implications there are?

Yeah, I mean it's a weird world that I don't know anymore. Like I code less and less, but I now can code again. Like because for example, we have a data scientist in the team and he says, I don't feel like I'm a coder anymore. I'm just a manager of a user of this LLM that I validate the output and yeah, it's okay. And it's a bit ridiculous, he says because even simple things like copy paste, I will not bother. I'll just say, okay, can you refactor this? It definitely has changed and okay, there may be Implication about creativity of people. And if you go into the AI LLM space, like for example, for images, when those models generate images, somebody, a painter may say to you, hey, you may lose creativity because you generate always the same thing. I don't have answers for those things that I don't think a lot of people have answers. We'll just see, right? Like how things go. Sincerely. But yeah, it's interesting, right?

Yeah. This idea of being a manager of your assistants, I think is really helpful. I forget who it was we had on the show, Chris, but they were saying, hey, if you think about this thing just like a high school intern or something, you know, like, is a high school intern going to solve all of your problems? No. But if they work all day on your problem or you know, let's say you have infinite number of those, those high school interns that just can do all the time, Is that useful? Certainly there's a management aspect to that, right? Probably more so with high school interns than with LLMs. I'm not sure, but. But yeah, I love that metaphor. That's really good. I also wondering, so this particular application of AI within someone's code base, right. I think similar to what we've seen in other cases with Copilot and some of the things that have happened, it's a very sensitive area, particularly for like enterprise business users. I think if you're like an indie hacker and like you say you are wanting to create a TypeScript UI and you don't know typescript, like, boom, you can get some really cool results really quickly. Of course, enterprise code is part of the IP of a company. There's two aspects of that, one of which is the fact that companies have been hesitant or even sued others over usage of their code or data in ways that they didn't expect. I think though, the other aspect of this that you alluded to is it really is powerful when you start to bring your own data to the table, especially with these open models, both because they have kind of privacy conserving deployments and there's also like code preference things and other things that your company might have. So I'm wondering if you could speak to that a little bit and how you envision you're helping build a product that's doing code optimizations for people. So how do you think about people creating customized models for their code bases and the sort of proliferation of these customer specific models and the hosting of those? What goes through your mind when you're thinking about those things?

That's very, very important. Topic and we have quite a lot of experience with this. And I will mention an example where we wanted to. We went to a client, very big technology firm, super big, one of the best. And we said, hey, this is our platform, we have these LLMs you can use any LLM of your choice like GPT4 or open source LLMs like Llama 2, et cetera. In the beginning they said we don't have any approval for OpenAI because first they don't know the IP issue and second, they don't want the code to go outside. Right. We're talking about proprietary code at a lot of such companies. So the solution there was, okay, you can use the custom open source LLM on your data. We do not see anything. So it's a custom solution on premise. And while they're using a product, then practically our platform allows them to generate their own training data set. So for example, they use Artemis, they optimize code. They see sometimes the code is optimized, it's not optimized, but they generate the data and those data we cannot see and the client should have those data for fine tuning their own model. Then through the platform they can further fine tune their own model. That is how the industry will go, especially financial sectors and technology sector or defense companies. They will never give their code outside on an open source level. But LLMs are super powerful on that. With one client that they said, I'm not sure if I can get recommendation. If you give me a recommendation from an LLM, who is liable if that code doesn't work? What is the IP issue? I said, how do you solve it? Now they have the IP checkers, et cetera. I said, you can just use the IP checker for this moment, the same code, the same process that you would do, but also LLMs and we have added this functionality can do very good, good similarity search. So if you have other code bases and similar functions in your code base, you can fine tune nicely or you can look like the same way people are building chatbots on your documents, you can build your own chatbot on your code. So it's practically similarity search. And then we could recommend we even identified that three teams had implemented the same functionality, a bit different, right? So you even save, save time, right? So still you know this technology, the underneath technology, if you know how to use it properly, you can take advantage of it. And the biggest example is DataBricks, both Mosaic ML, because MosaicML for like they acquired Mosaic ML, that helps organization fine tune their own LLMs or their data because no organization will give their data to OpenAI or any other company to fine tune on top of the them especially big organizations that there is value on their data. So the same thing applies for coding. That's how we see and that's why we adapted the product accordingly.

My next question is I guess, sort of selfish and I like to ask this of people that have, you know, really built impressive things with this kind of new reasoning layer of LLMs. I'm wondering as you look back on building this product for code optimization with LLMs, are there any challenges that were unexpected that you, that you had to overcome and are there any sort of takeaways for that you would give to practitioners that are maybe working on their own products or integrations with LLMs? Like what has been important for you to stress especially as as a CTO and bringing new people into the team as you're working with these types of models, what's important in your mind and any of those challenges that had come up, Anything you'd like to highlight?

If you're building applications and your application dependent on LLM output, then you need to make sure that as a first stage I would recommend use something like a closed source API usage because it will solve you the headache of deploying your own LLM having good CPUs that is a problem that you cannot scale at this moment. Most teams don't know how to do it. And of course there are a lot of startups, a lot of companies that are working on this on how to build your own LLM in a scalable way. But that can be a nightmare to build. So if your business is not how to deploy an LLM and the value somewhere else, if somebody provides it as a service, it makes sense to use it. So in our site we say you can import with an API key and a secret key any LLM that you have access. Then your application becomes much easier. But then you have the problem that you need to solve of is the client. Okay, if those data are used on OpenAI or because there is a data that goes there this in financial sector or currently cannot be accepted, then our product will have to deploy. Llama 2 have to deploy. So then you have to build it yourself or use those services. So we had to build it because we were one of the early adopters. But there are tools like Hugging face provides a very nice API for you to deploy. Now they change the license I think but you know, I think majority of people can use it. So the speed of LLMs is a big problem for scaling Application definitely. Then other issues about LLMs which were a bit in the beginning. Now they are trying to be fixed is token size every time you ask and the result may be incomplete. If there is, then how do you deal with the previous context and all this kind? You need to spend time on this to do it properly. I am expecting more and more tools and open source are solving like LangChain and those tools also open source are doing some of the things. And of course the biggest problem that a lot of people talk is about hallucination of the models. You cannot trust necessarily the models and you cannot just say generate code and execute that code in your back end because somebody may do SQL injection like similar to where people were doing SQL injection in the past. Especially for coding you can have LLM kind of injection. So you need to be very careful on exposing the prompt to the end user because somebody can really damage. So yeah, those are them.

When you're thinking about hallucinations from LLMs but you're working on a problem like optimization and stuff and you acknowledged earlier or you know, some of the problems you face is one of them. You may not get, you know, the right code or compilable code because it's the output of an LLM. How do you approach that specific problem? I was actually wondering that earlier and the conversation continued on without it. But we kind of circled back around like how do you think about dealing with hallucination when you're dealing with optimization and correctness, you know, and improving in that way, balancing the two.

Very good question. So it depends on the programming language and the existing tools that you can also use. So if you go for programming language like Husky and functional programming, in theory you can have a bit more proofs about code before and code after works the same NASA for example would want this proof. You cannot have code. Second, the mechanism is ideally would like applications to have unit tests and test all the scenarios. So when you do change ideally. But not all code bases have.

Right? You mean not all code bases are fully covered with tests, I hope.

Yeah.

Unfortunately there are open sources projects that the unit tests don't even pass. They are like we take a code base, we say pytest or whatever. They don't pass by default. So it's still this needs to improve. Hopefully LLMs can improve that. The third mechanism is we aim for minimum code changes with the biggest optimization and we go gradually. Like for example, we try to say first target data structure optimization that is 1, 2 lines that you can check, then go in one single for loops, double for loops. You go a bit gradually on that. And we currently make a pull request with the recommended changes. So we still want the developer to validate those changes because you cannot take that risk. And also from a psychology perspective, if you have a tool there that you consider this is my performance expert and tells you at the end, hey, make those three changes that you can verify, it's not different from if we change the name of the tool and put a developer name and make a pull request, that person will not know where this pull request came from. So you follow the same process. That's how we see it.

Something that I'm kind of getting in what you're saying as well, which I know is often a misconception that I run into when I'm either doing workshops or working hands on with people with generative models is there's typically this misconception that, that you need to package everything into a single prompt and then output your final result. It's a sort of one step thing. I'm getting the sense that your workflow for one, it probably involves multiple calls throughout the code base because of the context size. I would assume is partly because of that. But then also you mentioned this kind of iterative element where hey, there's kind of big rocks that you can move that are the sort of worst offending areas. So there's hierarchy in that respect. But also it seems like let's just assume. I know it's not a good assumption, but if we assume that a person's code base is fully tested, integration tests, unit tests, it seems like this is something you could just loop over and over and over and over again to get increasing optimizations, probably with diminishing return. Could you speak a little bit to how you as a team think about that chaining element, I guess would be the way to say it. And then also maybe iterative element.

First of all, the way we have presented this, like let's say you take the original version of a code, there are two approaches there. It's one, I apply one LLM and take the first three, four suggestions of this LLM, right? And then apply which one works like even on the papers on how good they are on the code base. They will say the first top five recommendations from the LLM, three out of five outperformed something like this. So it's not a one chance applied LLM, etc. Now the best approach from what we have seen is you get the first version, you apply the first version and if you have the ability to get feedback from what you applied. That is where those LLMs are very, very good. So for example, you say optimize this code, you try to pass the unit tests or you try to compile your code and you get the compilation error message. Then you go back automatically to the LLM and say the recommendation you gave me didn't pass because of this error. And then they can give you better. It's like the Wolverine technique or I think somebody did a demo where they were showing how you can compile and learn because this. So if you think how you are using those LLMs, let's say write my email, they recommend, they say sorry, this is too official, I want it a bit more friendly. This is I believe currently the best approach. So it's like iterative approach. If you have a way to measure and give the feedback back, you can have the best result. Like if we take the logs and give it to the LLM, we'll have even better. But even if you say hey sorry, this is not good, give me something better, it will will again try to improve in the context. You can even do things like again, I think I was in a presentation that LangChain and similar one where you can combine 2, 3 versions of different LLMs and then the 3 base you can have 3 and then combine them. Say, take context from here, this work didn't work. So there are different approaches.

You just stole my question right out of my mouth. I was going to ask when you extended that to multiple multiple LLMs and integrated. I was kind of wondering how you were thinking about that because you were addressing like when one LLM gave you multiple points back in terms of optimization and you're trying those out and kind of how you might extend that to multiple LLMs. Because we're getting in this world with an increasing number, we're going to just be awash in LLMs before long. And as you have so many APIs or so many deployments available to you, how does that change? You know, it sounds like your workflow would work for them that regardless. But does that add value or do you think there's diminishing returns as you keep adding LLMs into it?

No, I think there is a value and also it gives a flexibility to the user to always not be locked to a single LLM. Like if you for example, it may be different user, it may be pricing issue, it may be a new model this week can have bigger token size, it may be the performance of this. So you cannot rely on one single LLM nowadays because it's so easy to build LLMs if you have the data. Our business doesn't depend on, hey, we have the best LLM and somebody else suddenly in a week can give you a better LLM and then you are out of business because you need to spend 100 million to train on CPUs. So in our case combining LLMs using LLMs and having that workflow LLM agnostic, it's a way that you can utilize. Now if people pay for better LLMs and access, then the result is better. Here though, we have to mention one issue which a lot of people may not know. It is a bit tricky when you use output from one LLM or the other LLM. So there are IP issues, etc. So you cannot. We are also investigating. Exactly. Yeah, but you cannot use in theory chatgpt output to fine tune llama in a commercial setting. Maybe you know the Alpaca paper, I think the first one show that you can say to ChatGPT, give me examples, then fine tune another model and then have it. But you are not allowed on a commercial side. But I'm sure there may be two open source LLMs eventually that you will use as far as you have the framework. Then we allow those things to happen.

As we kind of near the end of our conversation here, I'm wondering if you can paint a bit of a picture for us from your perspective as someone working day to day in in developer tools that are AI driven. What are some of the most exciting things that keep you up at night as you look forward to sort of the next year? It could be things you're working on, but it might just be generally how this field is developing. What's really exciting for you as a person building these sorts of tools as you look to the next year or so.

I personally believe that this is the start of the power of this technology. We already see how much it has changed the way people are coding. What I want to see and I see more and more things make it like people nowadays want to use like talk speech to text and then code these kind of things. Like you, they're trying to make developers even lazier these kind of things. So from my perspective, from what we are trying to do, I want us to we are in the process of looking open source project going through our platform and they really get great optimization that we can give to the community. So for example, if I can get a very slow Machine Learning Library, 30, 40% all automatically make a pull request, show to the people how easily we can optimize the speed and everybody can benefit from that. It excites me to know we still haven't found out what are the limitations of the current technology and how much inefficient code is outside there. And I'm excited to find out how much we can improve in an automatic way. Like, I don't know, redis these kind of things that everybody's using in their own day because they can make our laptop faster. Like already my fun is going crazy. And yeah, that's what this combination of LLMs and coding is something very, very exciting because we don't know the limitations of this. That's what I want to find out.

That's awesome. Well, we will certainly be on the edge of our seat as you're exploring those limitations and those possibilities. Really appreciate you joining us, Mike. It's been a great conversation and I'm very much looking forward to my code running fast, faster, despite my ignorance of how to make it do that. So thank you so much and we'll talk to you soon.

Thanks a lot.

Thank you. Thank you guys.

Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already, and if you're a longtime listener of the show. Show help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to@fastly.com and fly IO. And to our Beat Freakin residents Breakmaster Cylinder for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.

Sat.