The last mile of AI app development

Foreign. Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly. Deploy your app servers and database close to your users. No ops required. Learn more@fly IO.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist building a tool called Prediction Guard. I'm joined as always by my co host Chris Benson, who's a tech strategist at Lockheed Martin. How you doing, Chris?

Doing very well. We're still in 2023, the most exciting year in AI.

It is, you know, it's hard to keep up, but it's also sometimes hard to understand, like what of these cool demos and models and integrations are like actually production ready and how are people actually taking these things into production? And we're really happy to have with us today Travis Fisher, who is a founder and CEO at a stealth AI startup and is focused 100% on that delivering products with AI. So we're happy to have you here. Welcome, Travis.

Thank you guys. It's a pleasure to be here. Looking forward to the conversation.

Yeah. Well, on Twitter you posted this diagram which I think maybe you have penned right now, which is how to use large language models effectively. And it's sort of like a start, simple to complex scale and I found that really great. And I've actually shared that diagram with a number of people and various Slack channels and all like that. This is how you should be thinking. How did you, maybe not specifically with that figure, which we can talk about, but like how did you get into thinking about how to use large language models effectively? Like actually how to build products with these models?

Yeah, it's a great, great question. So I'll give you kind of my quick what I've been up to in the last six months, which is going to answer some of this stuff. I'm a huge fan of open source. When ChatGPT launched on November 30, 48 hours later, I released the ChatGPT npm package, which was using unofficial API. And it just allowed like thousands of developers to go and start building with this cool new thing that, you know, like the GPT series and LL have been along for a while before that, but it was kind of this step function in terms of just their mainstream adoption and really just caught people's attention. You know, after that I released the ChatGPT Twitter bot, which now has about 123,000 followers. I run a group called chatgpt hackers.dev that has about 9,500 AI developers. Just a whole spectrum of people, right? We have like researchers in there and then we have like prompt engineering, script kitties. And because, I mean, my background is computer science and I do have some formal education in machine learning, but it's not like I'm an AI expert, like by any means, right? And, and what has really captured me really over the past year or so has just been the rate of progress and trying to wrap my head around it and understand it. And because it's been moving so quickly, I've been optimizing for my rate of learning. And I personally learn best by building out in public and building open source and just sharing what I learn as I go. So, you know, I think there's a lot of like, complexity and turns in AI, as you guys know. And to some degree, even just having a mental model of like, what are the different approaches that you could start with or how to approach solving a problem is already like a difficult starting place, right? And I know for the how to use LLMs effectively, my inspiration there was Andre Karpathy recently, maybe a month or two ago, tweeted something about, you know, all these big companies are interested in using AI. They're aware that they should be using it to some extent. And so they're like, well, we need to hire a team of ML engineers and get on this, right? And the huge unlock now is with these foundational models, like for most problems you don't need to do that. And you know, yeah, there's the production side, the practical side, and I'm sure we'll get into that. But like, in terms of where to start and starting simple and actually validating for your business use case that you can actually solve it with AI, that you understand the problem domain enough that you have whether it's training data or that you're actually solving a real customer problem. Like starting as simple as possible with hosted foundational models, a lot of times is a great way to get started and just to validate quickly, you know, as you kind of inevitably find points where your workflow breaks down or where you're not getting the quality or the cost or some hard constraints like security, privacy, you know, of your data. There's kind of this ladder of complexity I like to look at. And you know, you start with just prompt engineering up at the top and then it's about like well, how much can I reduce hallucinations or add domain specific context into my prompt by you know, doing information retrieval. And then at some point you're like, well a single prompt isn't doing it. So maybe I add some iterative process to that where I use another language model. There's all these techniques for doing kind of multi step prompting, but you can do all of that with a hosted model and you can get like 95% of the way there for a lot of problems and domains these days in a way that was previously like locked behind proprietary data providers and you had to have so many resources to be able to do that. So it's really this like democratizing point in the industry at the applied AI level that we're at right now. And I think from the conversations I've been having with folks who are a lot of them like full stack typescript devs who are building applications and they want to use AI, they know it's cool, they don't know how to get started or they're like, oh, I need to learn Python, I need to train these custom models and stuff. And all of that is super important and it comes into play at a time or for particular types of problems. But the majority of people for getting started, like Start simple is the main takeaway from that.

I think that's a great insight and I think that's one of the places where so many people go wrong, is jumping into too much complexity. They don't find simple need and potentially even don't look for things that work just as fine, that are not AI in that way. So I love the go simple and build from their philosophy. I think that's incredibly practical.

I get the sense that a lot of this sort of chaining and bringing models together, doing the information retrieval, it's sort of almost like a hacking culture around this language model prompting, which is really cool and that can go so so far. Maybe there's like you say there's privacy or domain specific concerns with enterprise use cases, but in your like you mentioned the community that you've kind of built up and you're part of on Discord. What are some of the things that have maybe like surprised you that you've seen that, hey, I didn't even think that maybe this was possible with just this layer of like using a hosted model, using pre training, using retrieval, whatever it is. What are some of those things that you've seen that kind of surprise you or maybe like help develop your thinking around this topic?

I Have a few examples and one story examples would be folks who are taking these models and like applying it to their personal finances, right? And there's one guy in our discord who's like an ex hedge fund guy and he created a very basic agent that uses a large language. I think he's probably using GPT4 to take this unstructured information from his bank's website about his expenses and like, you know, extract structured information about that and then, you know, he can graph it and whatever. So I think there's a lot of hacking going on around this stuff. It is very, very early. Another story of something that surprised me, and this is just a fun story, but when I released the kind of unofficial API wrapper for ChatGPT, we kind of had this cat and mouse game going back and forth with OpenAI for a while because apparently there was kind of a group within OpenAI that was like, oh, this is amazing, look at the open source community is doing, they're building all this cool stuff. And then there was another group that was like, well, we're going to have the official API eventually, we want to control this stuff, right? So there was kind of this back and forth, right? And at one point our community found a public model, but it wasn't like publicly disclosed, it was security through obscurity. But it was a fine tuned Chat model that ChatGPT was actually using at one point and all of the open source projects started to use this thing and there were tens of thousands of actual real consumers at the end, you know, who were building on top of this. And of course OpenAI knew that we were doing this. I talked with one of their security engineers about this after the fact and they instead of like what you would expect, just shutting it off, instead they switched it out with what they call CAT GPT. And just all of a sudden one day in our discord, we started getting hundreds of messages from users saying, I think I got hacked. I'm seeing all these meows like in response to my thing, right? So it goes to show, you know, the moral here. And I ended up hearing from the OpenAI engineer that they were watching our discord, taking screenshots and laughing their asses, you know, at this happening. But it goes to show that like one, the kind of level of there's no switching cost to these things, right? It's like text in, text out, fairly basic. And there's like entire new venues of like vulnerabilities of like swapping it out with a, you know, a cat or something like what does security look like in this world when it just thought it was an interesting kind of anecdote.

Probably that vulnerability of like all of a sudden getting meows like that is a possibility. But I'm wondering like, as you've spent a lot of time with these models, you've also like you're building products on top of these models. Like from your perspective, taking an LLM integration to that sort of last mile of like integrated into a product, supporting users, what are the things that should be on either developers or data scientists minds as they think about like taking the step from like demo to like product integration, I guess would be the question.

So I like to say that absolutely everything in engineering is about trade offs and it's about really thoroughly understanding trade offs and then being able to effectively communicate those trade offs and the pros and cons and everything. And it really boils down to those two things, like over and over. So let's talk about some of the trade offs that are most important to using language models in practice. You have the most obvious one, which is quality. Like can I use these language models to actually perform the task that I want? You know, you have oftentimes secondary but equally important trade offs, like how much does this cost to run in production? What is the latency for my use case for the end users? How consistent and reliable is it? Like, can I have actual. Is my use case fault tolerant? Which is a great initial question because we're kind of moving from a world of like very deterministic human driving the program to a world where the more control you give to the language models and their reasoning abilities, this is getting more into the agentic side of things. The more that it becomes slightly non deterministic or very non deterministic. And so the ability to have guardrails around these things, the ability to have consistency and predictability is extremely important. And one of the first questions that you should ask yourself or you're thinking about like integrating with LLMs is for your particular use case, for your job to be done for your customers. To what extent, you know, do you need 100% reliability versus like 99% reliability? And that may sound like a little bit for certain domains of problems. It's everything, right? And so that's one fundamental question. There are techniques and we talk about them. I'm sure you guys are very aware as well of like going from that 99%, getting close, adding extra nines of reliability. And you know, that's also a very active area of research where folks are actively Figuring out ways to increase the reliability of these models. But the fundamental trade offs are, you know, quality, cost, latency and reliability. And using a hosted model is going to be great for quickly like with minimal resources and validating your use case. For a lot of those types of trade offs it may make more sense then to use a local model. And there's kind of been this Cambrian explosion of open source large language models and other, you know, specialized machine learning models and we're going to continue to see that proliferate. I like to think of the open source kind of state of the art as you know, six to 12 months behind the proprietary versions. Um, we'll see if that holds. But you know, it's kind of because there's like zero switching costs with these models because there's just so much competition, the price is going to keep going down over time. We're going to see the open source side of these models continue to get more powerful. And so for a lot of use cases where you're dealing with well maybe I need ultra low latency on device or maybe, you know, cost is a factor and I need to be running in my own data centers or maybe you know, you need to, after a certain point once you validate your use case, you want to fine tune and distill the model down and have a really locked in like a checkpointed. This is like completely unit tested. This is evaluated version of things. And I think we're at the stage right now where there's so much like hype and so many people building AI applications and demos and that's great, right? Just getting it out there, proliferating through open source, through Twitter, whatever it is, right? This is awesome. But the version of that last mile and the productionization concerns really need to dive deep on all of these kind of fundamental trade offs that I'm talking about in the hosted models versus local models and, and you know, fine tuning, distillation, they all become really important very quickly.

So before the break is we were talking about these different characteristics that kind of affect applied AI and affect deployment. I was really taken by the fact that so many of them are not really AI specific. You know, if you could almost argue that applied AI in so many ways is about software, it's about the systems, it's now about cloud, it's about all these other things blended together to produce solutions that are productive in the world and have value for people and organizations. You know, we talked about unit testing and stuff like that. What is your thinking around kind of the integration of all those things, because the model itself, you know, to your point about hype, still kind of gets all the attention and the amazing things and it is amazing what we're doing. But to make this stuff work in life, there's all these other concerns that there's so many cool things in 2023 happening on the model side that the other 99%, to make it real, kind of gets when you're working with people around, understanding how all this fits together so that they can do that, how do you frame that so that their attention gets on the right thing, their budgets are properly allocated to attend to all the things I've seen organizations really struggle with that because they go into it with hype, focusing on just the model and building skill sets and budgets around the model and then they try to figure out the, the whole thing with clouds and deployment and things afterwards and they have a hard time. How do you navigate that, given the hype cycle that we're operating in?

My first piece of advice would be that for your particular use case is your job to be done, whatever business use case you're solving. And to keep in mind that AI, like all software, is a tool and it may be a really shiny tool, it may be a tool that is evolving very quickly in front of us right now. It's a very powerful tool. But it is a tool to solve, you know, a business use case and a problem for humans. So rooting the framing in that I think is very important. The second thing I'll say is a lot of AI right now, and especially the stuff that gets a light shined on it and in the open because the application layer is so new and there's so much low gain fruit, as you said, we need to have more emphasis on the engineering rigor under the hood. And so one practical piece of advice there is to really focus on an evaluation set for your particular use case. And you might have existing data, you might have existing kind of input output pairs. For your particular example, you might have, you might not have that. But like starting from there and working backwards of like this is what the end user is going to see and then working backwards from that to think about, well, how can I use language models or other expert focused machine learning models to solve that I think is very important because that also gives you a grounded north star that like so much of the prompt engineering and tuning of these models is based around. Well, I think this is going to work better or I eyeball it on this one example and it seems to work for this right but really applying some fundamental engineering rigor at that level, where you have an evaluation set that you can track, that you can improve over time, that you can. And not just tracking the quality of these models, but, you know, tracking the other trade offs in terms of pricing, latency, recall, there's a whole slew of trade offs that can matter depending on your particular use case. And then the other piece of practical advice I would say is the kind of diagram of this ladder of complexity that I was referring to before. Like every time you take a step down that ladder of complexity, from using a hosted model, just using prompting and then going to some type of information retrieval embedded in the context, to having a multiple chains of prompts, to going down to fine tuning, you know, a hosted model or fine tuning a local model. And at the very, very bottom is building your own model. Right. Like every time you take a step down that ladder of complexity, it adds engineering complexity. It's going to make your solution more complex to maintain. And so really like having a good handle on how you can start simple and only move down, you know, when you need to or when you hit a constraint like, okay, this is great and I have a working solution with a hosted and API, but now I need to worry about the price because I'm going to production and you know, the unit economics. Like maybe at that point then you think about, well now I have this great solution and I can auto generate an eval set for myself and you know, have a bunch of inputs and outputs and fine tune a model that is hyper distilled and efficient and focused. That's great. Don't start there for most use cases. Right? The one other thing I would say at the practical level is where language models tend to break down or lack reliability is oftentimes when you're trying to give too much to the model to do it once. And so breaking the problem down into sub problems, problems that are a lot more focused, is one of the most practical. Like I just find myself telling people over and over again, it's like, okay, that's awesome. Break your problem up into sub problems and you know how to do that is a whole art form itself. And maybe someday in the near future language models will do that for us. I don't know. Right? That's getting into the more sensational side of things. But as a general principle, breaking your problem up into sub problems, thinking about how you can articulate your problem as succinctly as possible in a way that is native to the language models is a really key Practice.

I love how you talked about evaluation, forming your evaluation set, getting some ground truth, also breaking up your problem. Maybe having an evaluation set for each of those sub problems would be a good idea. I think there's this general perception that large language models are this kind of unique thing. These chat interfaces are this kind of unique thing. How do you evaluate that? In the way I think what people have in their mind is, oh, if I'm doing sentiment analysis, it's either positive or negative or neutral and I can calculate an accuracy, for example, whereas they might struggle to think about, okay, well, there's this output from this language model. It seems coherent and fluent. How do I evaluate this? And so I think there's maybe a bit of confusion around the evaluation side. Can you share any tips or thoughts in terms of what you found to be useful in your own work? In terms of evaluation sets and like, how you think about how good the output of a language model is?

My first thought would be the less that it's about me thinking about how good it is, and the more that it can be objective, like using some constant way of evaluating it, the better. There's one project that I really like recently by Lance Martin. It's called Auto Evaluator. I don't know if you guys have seen it, but it's specifically for the domain of qa. So question answering. And he recently partnered with LangChain to create a hosted version of it. But the way that I think about this is a little abstract and it's really like starting from your job to be done. Oftentimes sentiment analysis isn't the job to be done, it's like a piece of a job to be done. Right. So again, it's like breaking up the problem and understanding how to think about and structure those problems as whether it's an expert model that justice sentiment analysis or it's using a large language model that, like, it can do sentiment analysis. It's really good at that. But it's also like it can do a whole bunch of other things as well. So the more focused your task is, the more clearly articulated your task is and the more structured of the output that you have at the individual LLM call level, the better and the easier it is to create reliability around these things and to actually test them with more traditional software engineering practices like writing unit tests or integration tests. You know, one thing that I'm actively working on right now for the TypeScript community is a way to invoke large language models and have structured guards on them. I know, like Prediction guard guardrails. There's a few projects that are doing this, but really then having that actually be typed in TypeScript so you can make an LLM call like it's a function, but get some JSON that has these fields that has these types and have it kind of, you know, there are techniques that you can do under the hood to self heal if the JSON isn't properly formed. Or you know, maybe you want to generate some TypeScript code and you want to validate that's a correct AST or something. Like there's techniques that you can do to constrain the output of the language model for your particular task. But in my view these techniques are constantly shifting. Kind of the best practice in the state of the art. There's. And so I think libraries like LangChain and the open source framework that I'm currently working on, you know, I think will do a lot to help developers to abstract out some of the complexity of just viewing this as a general purpose tool. And again, it's like you start simple. One of the great things about language models is they can do just about anything. That's also one of the downsides, right? Like when it's so unconstrained, how do you even approach the problem? So having best practices having like examples constraining the problem and really it's like the ability to have a unit test or an assertion in a traditional programming language at the large language model call level where it's like I assert that the output should be valid JSON or I assert that the output should conform and be valid TypeScript syntax and if not like actually self reflect on that and put it back into the large language model and regenerate it. All of those things I think are foundational primitives at the large language model level that will allow developers who want to build real reliable applications to do so more reliable because they can focus on their domain specific business logic or aspects that are away from a lot of this kind of implementation details that are also constantly shifting under our feet.

Right, Fantastic explanation. You keep talking over and over again about how things are shifting and the evolution of the engineering around it. That puts a burden on these hackers and developers that are trying to go out and implement these things at this point because this year has just been, you know, phenomenal progress. But that makes it really hard for mere humans here to kind of track that and keep up with it. So you kind of talked about some of the concepts right there. But if you're thinking about like you're about to turn to somebody who's a hacker and they're looking for that guidance. What are some really, you know, not necessarily comprehensive, but hey, go do 1, 2, 3 or ABC. And that will help you kind of keep leveling up. Like, what are some of the things that you're telling people these days to say if you want to keep up this year with what's all, with this insane progress and LLMs and, and all the different model types that we're seeing, the progress, what are you going to do in a practical side as a hacker to manage that? Do you have any tips that you can kind of take us through about?

Absolutely. One, it's super noisy. There's so much happening. We're in the middle of this exponential wave. And, you know, I think a lot of people are like, oh, I have fomo, I want to be on this wave, right? But where do I start? And there's just, just so much noise, which is great on the one hand, but on the practical side, like, how do you give advice? Where do you start? So, you know, there's a couple levels to this. I have talked, given some talks to kind of like a ChatGPT for beginners type crowd. And really it's like, my main advice is one, just use it. Just go and try it, right? That's simple. But two, more importantly, like, the next time you have an actual problem that you think, like, maybe I could use ChatGPT or a language model for, actually try using it to solve your own problem. Because what that does is it starts to build up this muscle in your brain around thinking about using these new type of tools to solve problems. And it's really a different type of tool. It's like exercise. You need to start exercising that muscle early and often. And there's a lot of noise. There's a lot of different AI tools. The side of things which I'm confident will be just as relevant, you know, a year from now, a couple years from now, is building up that muscle to think about using, how to actually use AI to solve your own particular problems. It's one thing to talk about, like hypotheticals and general, like, cases of problems where these tools excel, but it's another thing entirely to start building your own personal, you know, muscle.

Totally agree with that.

I have a, you know, whether it's a personal problem and you want to just go, you know, Talk to the ChatGPT or you have a problem at work, and you're like, well, I think I could use a language model, a hosted API to Solve this or something like starting simple, starting from your own problems will start to build up that muscle and you know, it'll, you'll naturally learn it and take it from there.

So Travis, you've mentioned a couple times like TypeScript, Node, this community that like you're a part of. And I think there's probably a lot of Python people listening to this show, maybe data scientists, practitioners. To me it almost seems like there's like two communities. There's like all of these data scientists trying to figure out like oh, large language models, generative AI has sort of broken my intuition around like how would I need to be doing, like do I need to be training models now? Like, how do I solve this problem now? I was training models last year, do I still need to be doing that? So there's like that side of things and then there's like this really vibrant community of front end developers and other developers that are building, even like people that are maybe like low code, no code people building really cool products around this technology. You seem to be sort of like exposed to a lot of those things. How do you see these communities developing over time? And like how have you seen maybe the TypeScript, the Node JavaScript type Crowd kind of rise up to meet these technologies maybe in a way that the sort of traditional data scientist crowd has not or has differently?

I guess to some degree a lot of the JavaScript typescript world is like jealous of the Python world, right? Because all the cool new AI stuff is like Python first or using this machine learning framework or whatever. And this is where like hosted APIs, whether it's replicate or hugging face, so you got a hat on, right? Hugging face or OpenAI or cohere, like all these hosted models are a massive unlock for application developers and TypeScript is the largest programming language in the world. Python is, it's big, it's the largest by far in the data science in machine learning world for sure. And so there's this dynamic between the two where I think we're seeing at the application layer folks who are good at building full stack apps that can easily plug in to hosted models and things who really push the envelope in terms of like unlocking people's imaginations, building a good UX around these things that's so important, like to make it more approachable for people and to really like show people what a lot of the machine learning people have known about for a long time. Right. But like you need both sides of that equation. So one of the projects I did a couple months ago was I ported scikit learn to typescript and it's not like a full port, you know, it's like Auto generates all of the TypeScript classes, like 260 classes and then under the hood it creates a sub process, a Python sub process and then marshals and does the inter process communication between them. But it works extremely well. And you can call and do K means and PCA and just all these fundamental things that the Python machine learning world takes for granted. There are versions of that that exist in the NPM ecosystem. It's just they're all over the place in terms of quality, in terms of like there's so many just fundamental aspects of machine learning that the TypeScript world is missing out on. And one of the primary drivers behind kind of what I'm working on and I'm, you know, happy to share, like I'm, I'm Building a reliable TypeScript open source framework for building reliable agents. Very cool, thank you. I view agents as this new. So if large language models are CPUs, right? In kind of this new compute paradigm, there are these reasoning engines, like yeah, they're great at generating text, but the real emergent property, the real game changing property of them is reasoning. If they're kind of the new reasoning engines and you have like, that's the CPU layer and then you have like a storage layer that's all these vector databases and kind of overhyped on that side of things. On top of that you have like how do you actually run programs? And that's you know, agents. And I view it as like there's kind of a spectrum of like traditional programming that might happen to use a large language model. And then on the other end you have like full self driving agents that are making decisions and creating tasks and just fully autonomous. Right. And I'm excited to kind of focus on somewhere in the middle and focus on more reliable like use cases that we can actually build reliably today. But to your question about kind of the typescript Python world, a lot of the frontier, the libraries at the framework level that are pushing the edge here are all Python first, right? And I really want to take a TypeScript first approach partially because it's the community that I know and love. It's my like best tool on my tool belt and partially because I think people building real applications at the application level, a lot of those folks are more in the JavaScript typescript world. So.

So you have hit an area that I have so much passion for.

Oh, awesome.

I'm sitting here waiting to ask My next question here, and Daniel has heard me whine about this for years, what I'm about to say and so I want to get your take on it. So like there is more to the world than just Python and I'm a multi language person and I don't necessarily go all in on any one language or the other. I'm a Typescript user. In the last year I've been doing Rust. I had been doing more go before that outside of the AI and Python stuff, but I, I hit a use case where I was building something and I had to eke every little bit of performance out of the available hardware to do what it was. It was going to be C or Rust when it wasn't going to be C, so went to learn Rust and then I'm in Rust and I'm doing that and I'm looking at as an analogy here for what we're about to go at. I'm looking at webassembly and the Rust community and other language communities are so into, you know, that fact of, you know, write it in with the thing that you need to be in and yet have access to that in terms of deployment and still having great performance and stuff. And every time I'm now messing with webassembly in Rust, I'm thinking when is the AI world going to catch up on having multifaceted from a language standpoint, access to the models instead of everything being Python first? And so asking the pardon of the Python lovers in the audience, when am I going to be in Rust or Go? And you're obviously doing it in Typescript, but the language of my choice and taking advantage of, as you called it, the new CPU of reasoning from that point, instead of having to do a context switch. It is an ongoing year after year after year frustration that I have, as you can probably tell by now. So I'm hoping that you're about to give me the golden path out of here because I need one.

Okay, well, first off, I love your framing, I love your passion for this. I also feel very similarly. I think the reason why I'm starting with Typescript is because of the developer experience at the application level I think is really important for the type of framework I'm looking to build. But I view WebAssembly WASM as kind of the ultimate compiled language runtime that I want to target. Because you could imagine a world not too distant from right now where you have agents that are running, you know, in data centers, they're running on edge. So anything that's Kind of, you know, whether it's cloudflare worker or Vercel edge function or. Or within a service worker in your browser. Right. But the common thread there is wasm, you know, to what extent, starting with developer experience at the TypeScript level and then focusing on that at the runtime level, there's still a clear path forwards for a lot of folks for just like using hosted APIs. You know, that's one area that you can have that multi language very easily. That's a natural point.

But you got to be kind of in the cloud for most of that in a practical sense, which I'm not always.

Yeah, 100%. And then there's like the whole open source models or the practical side of things where you're like, well I need to have full hardware or the latency or something that's like on device and I am extremely bullish on webassembly as there's a quote that I like and it was from. I forget who it's from, from the Linux foundation or something. It was like, you know, If WebAssembly existed 10 years ago, then Docker would have never needed to exist. Right. And I think it will have that level of impact eventually. I think potentially I do too. Yeah, I think potentially the kind of unlock here that the path that could bring it into the more of the mainstream could be AI. I don't know at the model level there's just so much momentum behind Python, you know, and all the core kind of researchers and stuff are Python first. So when I did the scikit learn kind of port to TypeScript there was I think a Python port called PIO ID and it's a Python runtime. You guys might know better than I do, but it's targeting webassembly and it allows them to run subset of SciKit Learn in WebAssembly supported environments, including Node JS on the browser. And that's super, super fascinating to me.

Yeah, I think that there's a couple related projects I think like pyscript from Anaconda is like trying certain things like that, but really interested in that space as well because I've seen it's sort of like a different kind of diversity than we normally talk about. But the fact that like more developers from more diverse backgrounds are at the table building AI things I think is an amazing thing and I think a lot of good is going to come from that. So I'm really happy to see a lot of that happening.

Well, if we have time, just one more maybe controversial take on this sure.

We like controversial takes.

Awesome. Awesome. You know, as we get closer to building reliable agents and the way that I kind of was framing it before, it's kind of a fundamental new compute paradigm with large language models of CPUs and you build in these agents on top of them as they eventually get more and more reliable and more autonomous. Right now a lot of them are just toys, let's be clear. But as that happens, I view it as a new higher level programming language, you know, and we're working with natural language. The AST of that language is, in my view, a directed graph and the nodes are like specific LLM calls or a call to a tool or a call to an API. And you know, there's massive problems around, around how to add reliability at that level of having structured output or guardrails like, like some of these things are clearer than others. And then at the whole graph level, you know, that becomes a program or an agent to some degree. We're talking about all of these, like Python and Rust and then the implementation. And that's all very important. But I wonder to what extent, you know, 10 years from now we will even be talking about a lot of the current levels of programming abstractions that are hyper relevant to us today as practitioners or will, how quickly we'll move towards this world of a higher level abstraction for solving problems that is just significantly more efficient, more approachable, because it's kind of based on natural language. Anyone in this field that talks to you about timelines, you know, is like just throwing a, throwing a dart, ran with a blindfold on. But that's, that's one thread that I'm really excited about.

You kind of went already ahead to where I was hoping you would go, which is what's keeping you up at night, what's in your mind in terms of like looking forward and all of that. And I, I agree. I think this is a really, really interesting direction and I certainly hope that we see that timeline progress rapidly. I think we probably will. So yeah, it's been a pleasure to have you on the show, Travis. Really looking forward to keeping in contact and see, seeing all the amazing things you do and trying out some things in Typescript. It's an exciting time to be part of this and yeah, looking forward to keeping in contact. Thanks for joining us.

Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already subscribed and if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Flye for partnering with us to bring you all Changelog podcasts. Check out what they're up to at fastly.com and flye IO. And to our Beat Freakin residents, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time. Sam.