There's a new Llama in town

Foreign.

Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly. Deploy your app servers and database close to your users. No ops required. Learn more@flyio.

Welcome to another fully connected episode of Practical AI. In these episodes, Chris and I keep you fully connected with everything that's happening in the AI community. We're going to take some time to discuss the latest AI news and then we'll share some learning resources to help you level up your machine learning game. This is Daniel Whitenack. I'm a founder and data scientist at Prediction Guard and I'm joined as always by my co host Chris Benson, who is a tech strategist at Lockheed Martin. How you doing Chris?

Doing cool. I'm trying to figure out how did we survive before all these great new models and stuff. Like it's changed.

Oh yeah, it's been crazy. Yeah. I just created a a post for LinkedIn and I was like grabbing text, putting it into chatgpt, like getting nice rephrasing and then like, oh, I need an image. And in particular we'll talk about it a little bit in this episode. But I was like, oh, there's this Free Willy model from Stable AI which is like whale themed. Right. And then I've got the llama thing. So I just went to Stable Diffusion XL on Clip Drop and said hey, to generate me an image with a whale and a llama together. And you know, how do I even post to like LinkedIn before without these things? It's like a different world.

Yeah, 2023 versus 2022 is totally different. The content generation, the way you code. It's a different world.

Yeah. And this week, as most weeks are, it seems like in 2023 had some pretty groundbreaking announcements and releases which we're going to dive into a bunch of those things. There's just a huge amount to update on and I think it's a good time for one of these episodes between you and I to just parse through some of the new stuff that is hitting our feeds. So. Yeah, well, I mentioned Llama. One of the big things this week was llama 2, but I think before we jump into llama 2, which I think was maybe the main thing dominating at least my world this week, it might be worth just taking a little bit of time to Highlight something outside of this stream of large language models, which also crossed my desk this week, which I thought was really cool. It's this latest version of NERF. This is work from Google presented at ICCV 2023. So it's called Zip NERF, Anti Alias Grid Based Neural Radiance Field.

That's quite a name right there.

It is quite a. It stands for Neural Radiance field. So nerf, it's like camel cased in capital N, small E and capital rf. Nerf. These are fully connected neural networks that create unique novel views of complicated 3D scenes based on a set of images that are input. So I don't know if you seen that video yet.

I'm looking at it as we are talking and when you say the video, I know which video you're talking about because it's amazing. I've just left it on.

It's pretty spectacular. Yeah. So this is a podcast, so it's hard to express some of this for people. If you just search for zip nerf, you can go to the page for this paper, which is a great summary. But there's a video on the page and just to describe what it is, imagine this kind of complicated house with a bunch of different rooms and an outdoor patio, sort of garden area. And the video is actually this kind of almost like a drone fly through of the house and then the outdoor area. If you imagine a drone flying through a house, there's hats and coats and toys and couches and plants and all sorts of things everywhere. But the video is extremely seamless. And it's not generated by a drone. It's actually just generated by interpolating between a whole bunch of 2D images and then interpolating from that, the 3D scene. So, yeah, I don't know. What are your impressions, Chris?

First of all, from the perspective, the drone flight, if you will, that you have as a perspective viewing, it is like the best drone operator in the history of the world.

Yeah, it'd probably be hard to get one to do that. Yeah, yeah.

You're not going to get a real drone operator that could fly that amazingly, you know, and get those things. It's just phenomenal. And the house is like, for a moment you look at it and I mean, it is. It looks real, but I have noticed it's cluttery, but it's immaculately clean at the same time as well. The clutter is cleanly distributed and stuff. So I wish when my house was cluttered, it looked as beautiful as this house. It doesn't yeah, but. But yeah, I mean, just like if you didn't know, if you weren't listening to the, you know, practical AI podcast to go look at it or something like that and you just stumbled upon it, you'd think it was a drone video. You know, if you didn't have the education and go, oh my God, this is just really cool. I wonder how, you know, wonder what they're doing here. But it's indistinguishable from real life for all practical purposes.

Yeah. And so it's based on 2D images. And then there are these generated interpolations, which maybe gets to. There was something that we were talking about prior to hitting the record button, which was this whole field of generative AI is sometimes confined, conflated with large language models or ChatGPT. But there's a whole lot going on in generative AI that's not language related or maybe even based on language related prompts. So I mentioned that image that I generated for my LinkedIn post that was still a text prompt into a model that generated an image. But here what we're seeing is We've got static 2D images that are input to a model that's actually generating a whole bunch of different perspectives that are synthesized in a 3D scene. So this is, I would say, still fitting into our current landscape and world of generative AI. But it's not a text in text out or text in image out model.

Right. And I think people, there's so much coming at people right now. I think, you know, we keep talking about that this year, in the five years we've been doing this podcast, we've never had a moment like the last few months where things have been coming. New things have been coming at people so fast. New terms, new models, and people are trying to distinguish. So it's pretty, I think it's pretty fair that people are trying to make sense of how they relate together. And there's a lot of connecting between the idea of generative and the idea of large language. Models overlap in a lot of areas and you have models that are both and models that are just one and stuff. But I think it's a brave new world right now in terms of the amount every show. We're just trying to figure out what matters right now because there's a lot we're not hitting. Yeah.

And this side of things, maybe like the 3D or video or image based side of things, I know, has its own set of kind of transformative use cases that are popping out. I even remember A little while ago there was some technology I think from Shopify, but others have done this as well, where maybe you have a room in your house and you want to see how you can transform it with new furniture or something that of course you could buy. This is a real kind of E commerce or retail sort of use case for some of this scene technology of a different kind. If you think of this sort of technology that can take 2D things and create these 3D scenes, certainly there's use cases within game development, for example, but even other cases where maybe AI has never impacted the process as much. Like in real estate, for example. How expensive is it to literally have a person come out with specialized camera gear? I know that we've had this in the past where it takes a special person to come out with special camera gear to capture the kind of 3D walkthrough, essentially the street view walkthrough of your house and map that onto an actual schematic of your house. And here if you imagine someone, maybe I'm now selling my house myself without a real estate agent and I can take an app potentially and go through my house just taking 2D images and create this really cool and a fly around 3D view that's interactive. That's really, I think, a powerful, transformative change for a number of different industries. I came across a company called Luma AI in one of the posts about this technology. I don't know exactly how much of the, if they're even using the zip Nerf stuff, but certainly some things related to Nerf to take these 2D images. And they have an app that will create 3D views, which is pretty cool to see some of this kind of hit actual real users.

We keep talking about the fact that we've hit this inflection point where it's hitting all the, you don't have to be in the AI world for this to have a big impact. And so it's very easy looking at the zip Nerf video to imagine walking around with your cell phone on an app and you're, you're just kind of like walking around and the app takes care of whether it's video or whether it's still images or what. And it just uploads it to this and produces this, you know, amazing, you know, so it's not your walk around that it's doing. It takes that as raw video, but then it produces this super high quality thing. So yeah, I mean, I think this is another case where there's this one technology with thousands of Use case possibilities, you know, where it just changes everything.

Yeah. And maybe also in the. It'd be curious to know your reaction to this also with respect to kind of the industrial use cases where I've been thinking about. Of course, like capturing 3D scenes is very important. For example, for simulated environments where you're trying to maybe train an agent or you even kind of an industrial training for human sort of scenario where you want to kind of take someone into an environment that it's physically hard to bring a lot of people into.

Yeah. Or there could be safety issues and such.

Yeah, safety issues. I don't know if that. That sparks things in your mind. I think in the industrial sense this could have a more B2B sort of impact than just a consumer app. Right, Sure.

I mean a simple thing. And this is. I'm making something up in the next thing I say. But it's very easy for. For me to imagine intelligence agencies that are, you know, like if you go back some years to when Osama Bin Laden was found and they had, you know, various imagery and stuff, but with stuff like this, they might take all those images that they're getting from various sources and produce, you know, a high.

Like a flyover.

A very. Yeah, flyover. And very photorealistic of certain parts of the compound where that kind of imagery and that can be used in a military operation. Subsequently, I'm making that up, so don't. Nobody should take that as a thing. But it's not hard to imagine that. It's not hard to imagine a lot of factory uses and other industrial things where you have safety issues, you have limited access kind of concerns where you're trying to convey that. But there's a lot of mundane things, there's a lot of home based things and small business based things, as you pointed out the real estate one earlier. So this is just one technology that we're talking about so far.

Yeah. And I think what you're saying it illustrates how this is impacting very large organizations all the way down to small organizations. Yeah.

Sole proprietorships.

Yeah. And it's interesting how like if we just take this use case, for example, these kind of 3D scenes, kind of large scale organizations that maybe their bread and butter was either the compute associated with like rendering videos and 3D scenes, or their hardware providers that are creating specialized kind of 3D type of equipment, like their whole business model. They've got to be thinking similar to other organizations that are dealing with maybe language related problems that are thinking about these things with respect to LLMs, there's a fundamental shift in maybe how their businesses will operate. But then at the same time, it provides an opportunity for the kind of small to medium businesses to embrace this technology very quickly and actually make innovative products that can be widely adopted very quickly and actually be competitors within an established market. So there's an established market for, for 3D things that has been quite expensive over time in terms of access to that technology. So now that whole market's going to change. I think a lot of the players will be these kind of small to medium sized businesses.

I agree. I think there's a moment here kind of ironically because people are so worried about like the impact on human creativity because of all these models and stuff like that. But on a more positive note, there's this huge opportunity that you're just now alluding to for people that if you can connect the dots as things are coming out and you can stay on top of it, it's a great equalizer. And so it will clearly change many, many markets that are out there and many, many industries. And so there's huge opportunities for those who want to surge ahead at this moment and take advantage of that. And so I think that the message we tend to see in the media tends to be a little bit doomy and gloomy on that, but it kind of discounts the fact that change isn't always a bad thing. People are afraid of it, but there's huge, huge opportunities here as well if people choose to go find them.

Well, Chris, there is a new Llama in town.

Llama 2.

Llama 2 basically destroyed all of my feeds and concentration this week when it was released because it is quite, to me an encouraging thing, but also another transformative step in what we're doing. So Llama two, for those that maybe lack the context here, Meta or Facebook or however you want to refer to it, Meta had released a large language model called Llama which was extremely useful. It was a model where you could host it yourself as opposed to like OpenAI. You could get the weights and host it yourself. But the original Llama had a very restrictive licensing and access sort of pattern. Even though you could kind of download the weights from maybe like a BitTorrent link or something like that. And those propagated technically, if you got those weights, you were still restricted by license that prevented commercial use cases specifically. And now with Llama 2, Meta's released the kind of follow on to Llama and we can talk through some of the, what the differences are and what it is and some of what went into it. But I think one of the biggest things, which is, I think going to create this huge ripple effect throughout the industry, is that they've released it with a commercial license. As long as on the day that Llama 2 was released, you as a commercial entity don't have greater than 700 million monthly active users, you can use it for commercial purposes. So maybe if my company, maybe later on has 700 million monthly active users, which would be great. Probably never.

But there'll be something past llama 2 by then, though.

Yes. If it does though, I could still actually use it because it's only on the release date. So on the release date, which was. Was this week. If as long as you didn't have greater than 700 million monthly active users, you can use this in your business for commercial use cases. And I think that's going to have a huge ripple effect downstream. And we can talk about the model itself here in a second, but maybe just. I'll pause there to get your reaction on that, Chris.

It made me smile when I heard that because it's kind of like saying, so long as you don't compete with us at Meta, you can use this for commercial.

Oh, oh, it's totally true. Yeah. Like, who is that? Right. So that's Snapchat, TikTok. Right. Like you can think of who this is. And I guess one way to put this is it's not totally open source, quote unquote. Like, we wouldn't call this maybe open source in the kind of official definition of open source.

Yes.

But it's certainly commercially available to a very wide set of people.

Yep. You know, one of the first things I noticed when this came out on their page and they're taught, you know, there's, there's. And I'm diving into like, the specifics of the model here is we had an episode not too long ago and you were describing about kind of the. I believe it was the 7 billion limit, you know, in terms of hardware usage and stuff. And having been taught that by you, I immediately locked in on the smallest being 7 billion there as. And I thought, ah, this is what Daniel has taught all of us about that limitation on accessibility and who can do it. So, you know, it has the 13 billion and the 70 billion size, but I definitely picked up on the 7 billion, which I'm assuming is going back to what you were teaching us a few episodes back.

Yeah. And so just to fill in a little bit on that, so the Llama 2 release includes three sizes. So again, thinking Back to, to what are the kind of characteristics of large language models that kind of matter as you're considering using them? One is license. We've already talked about that a little bit here. We might revisit it here in a second. Another is size because that influences both the hardware that you need to run it and also its kind of ease of deployment. So llama 2 is released in 7 billion parameter, 13 billion parameter and 70 billion parameter sizes. And then there's also of course the training data and that sort of thing that's related to this and how it's fine tuned or instruction tuned. So llama 2 is released in these three sizes both as a base large language model and a chat fine tuned model. So there's the 7 billion 13 and 70 billion llama 2s and then there's the 7, 13 and 70 billion llama 2 chat models which we can talk about that fine tuning here in a second. But yes, you're right Chris. In that 7 billion I could reasonably pull that into a Colab notebook and maybe with a few tricks, but certainly with the great tooling from hugging face including ways to load it in even 4 bits or other quantizations, I can run that on a T4 for example in Google Colab with some of the great tooling that's out there. So not needing to have a huge cluster the 70 billion. Even with that, that's kind of another limit where using some of these tricks. I've definitely seen people running the 70 billion parameter model on an A100 again loading in 4 bit with some of the quantization stuff and all that. The 70 billion certainly going to be more difficult to run. It might require multi multiple GPUs, but that's kind of that sizing range for people to have in mind and how accessible things are. And yeah, how might you.

I'm just curious if you're looking at these, you're a business out there or data scientist and can you make up a couple of use cases that you might target with each of these where you might say oh I want to go 13 on this, not 7, not 70 for something like this. Can you imagine something like this? I'm putting you on the spot.

Yeah, I think, I mean there's certainly innumerable use cases, but I think maybe two distinctions that people could have in their mind is if you want like your own private chatgpt, right. Or like another way you could think about it is a very general purpose model. Like you could do anything with this model, like any specific Prompt, whatever. You're probably going to look towards that higher end, the 70 billion parameter model. For that kind of almost chatgpt like performance, you're going to have to go much higher. But as we've talked about on the show before, most businesses don't need a general purpose model. They need a model to do a thing and so, or a task or a set of tasks. And so in that case, I think businesses, because this is open and commercially licensed businesses that could take those 7 and 13 billion parameter models and fine tune them for a task in their business, which also increasingly has amazing tooling around it again from Hugging face and others with the PEFT library, with parameter efficient fine tuning and the LORA technique which is the low rank adapter technique which basically only adapts an existing model. It's kind of an adapter technique rather than retraining a bunch of the the original model. This opens up fine tuning possibilities in these smaller models where that fine tune for an organization is going to perform probably better than any general purpose model out there. And because it's that smaller size, you can run it on a reasonable set of hardware that's not going to require you to buy your own GPU cluster to host the thing. Right. So that's kind of maybe a range of use cases that people could have in mind.

I have one more question for you before we abandon this 7 billion to 70 billion being an order of magnitude jump on that. Why would you have something fairly close to that at 13 billion parameters like what's the difference in 7 and 13 when the next step is all the way up to 70? What's the rationale you think?

Yeah, so it is interesting actually if I'm understanding right from some of the sources that I've that I've been reading, there was actually, I forget if it was 30 or 34 billion parameter model that they were also had in pre release and were tuning. So there was another one that kind of fit in that slot that is kind of missing that gap like you're talking about. Like if you think of MPT, MPT has a 30 billion parameter model that fits in that kind of gap. My understanding, and you know, if our listeners can correct me if I'm wrong, please do. But my understanding is that they actually did test that size of model and found it to not pass their kind of safety parameters around, harmful, potentially harmful output or not truthful output, that sort of thing. So they decided actually to hold that back. So it could be possible as they instruction tune and get human feedback, potentially More iterations of reinforcement learning from human feedback. There may be a model that they release in that parameter range. So that was one thing that that happened. I think it is interesting, you know, several different things here that are unique about this model specifically or maybe the release as well, other than the license is they were fairly vague on the data that went into the pre training. So they talked specifically about some very intense data cleaning and filtering that they did on public data sets and it was trained on more data than the original llama. But they're fairly vague on the mix of that data and all of that. So that may be related to feedback they got on the data sets that were used in the first llama. I don't know. But the technical paper was mostly related to the modeling and fine tuning, trickery and methodologies that they used, which was interesting. And one of those interesting elements of the way that they fine tuned this model was I think the reward modeling. So if you remember like the GPT family of models, the MPT Falcon, these different models, one of the things that is often done with these models is this process of reinforcement learning through human feedback, which is this process and we covered this on a previous episode, which we can link in the show notes, but actually using human preferences to score the output of a model and then actually use reinforcement learning to correct the model to better align with human. Human preferences or human feedback. They actually use two separate reward models in this fine tuning of the chat based model, one that was related to helpfulness and then the other one which was related to safety. One of the interesting things that they talked about in the paper was how sometimes those things can work against each other if you're trying to do both of them at the same time. So they actually separated out the reward models that they used for the chat fine tuning into these two reward models, one for helpfulness and one for safety, which is quite interesting I think. Chris, maybe just a couple other things related to llama and then I want to see your feedback on Code interpreter as well because we haven't talked about that yet on the show and maybe Claude 2 if we can get to it.

Yeah, we got to mention Claude 2 as well because they were both big releases.

Yeah, so just one maybe other note which I find quite interesting and actually I would love our previous guest Damian's thoughts on this, who was in our last episode about the legal implications of generative AI. But one of the interesting things about the llama license, in addition to it allowing this commercial usage, is that there is technically a restriction in the LLAMA license that says you will not use LLAMA materials, which includes the model weights and etc. Or any output or results of the LLAMA materials to improve any other large language model, excluding LLAMA 2 or derivative works thereof. So essentially what this means is if you're using llama 2 and you want to fine tune a model, or you're fine tuning a model off of llama 2 outputs, you're stuck with llama 2. Basically, llama 2 is your model and you're going to stick with llama 2. So you couldn't, for example, technically take the llama 2 outputs from llama 2 and fine tune, say Dall E3 billion. Right. That would not be allowed by the license. And of course that's something that people are doing all over the place. They're taking outputs from GPT4 and fine tuning a different model, or taking outputs from a large model like maybe llama to 70 billion now and fine tuning another model that's smaller based on a certain type of prompt or something. So this is restricting that family of models that you're allowed to do that sort of thing with, which is the first time I've seen that. I think it's kind of interesting.

Yes. It strikes me as another Mark Zuckerberg anti competitiveness thing, which he's fairly famous for. I mean, that's kind of even before this.

Yeah. How could you enforce such a thing?

Yeah, I, I, that was my next question to you is, is there any possible way that you could conceive of to actually know that from an enforceability standpoint?

I have no idea.

I, I don't either. So it seems it's like a, it's a licensed thing and it will concern the lawyers, but it's hard to imagine, I mean, going back to our conversation last week, once you have output and that output is input to more output, and you know, there's a point where it becomes very, very, very difficult to know what the sourcing really was.

And the fine tunes are already appearing off of LLAMA too. So the most notable probably is Free Willy, which is from Stability AI and is a fine tune of the largest 70 billion model. But there's other ones coming out as well. And so I think we're about to see just a huge explosion of these Llama 2 based models for a whole variety of purposes. And who knows how they will fit into that licensing restriction or how open people will be about that. But it's about to start. The fine tunes are already coming.

Yeah, well, to your point, earlier they weren't terribly clear about the data that they were sourcing from their own standpoint. And I find it interesting, a little.

Ironic, a bit of a double standard maybe.

Yeah, a little bit of a double standard right there in terms of like we're not going to tell you everything about how we're doing input, but by the way, you better not use our output for your, you know, for others. Yeah, yeah, so yeah, a little interesting. Do you think there's any risk of a walled garden kind of concept happening in large language models if others were to follow this lead on anti competitiveness?

Yeah, it will be interesting. I think it is a notable trend that the first llama from Meta was not open for commercial at all and now they're opening it up for commercial purposes. And you know, maybe there's a separate trend that will happen with some of these use based restrictions that people are importing into their licenses and how useful those things are over time that will may shift and we'll see those things die off or maybe if they're enforced and there's precedent, maybe we'll see something go the other way, I'm not sure. But speaking of models that you might get their output and use it to train other models, I.e. these large scale proprietary closed models from people like OpenAI and Anthropic and others. We've got a couple of things that we haven't talked about on the show yet, which people should probably have on their radar. One of those is Claude 2. What do you think about Claude 2 from Anthropic?

Yeah, I've been playing around with it a lot in the last week and I kind of have a set of things that I try over and over again. They're kind of my standard tasks as new models come out and some of them are coding and some of them are content generation, which are kind of the two big things that I use most often. It was interesting. You can put, you know, the input size for Claude 2 is much larger than, than the others.

Like much, much larger.

Much, much, much larger.

So 100,000 tokens.

Yeah. And so it's had me kind of change the way I'm approaching it in that by contrast with like Chat GPT and you're trying to figure out with, with the limits that you have both on input and output, how do you kind of prompt engineer your way to get, you know, where you're trying to go, which has become this whole skill set we've been talking about, you know, in recent months. And yet Claude 2 almost kind of wipes that out a little bit in some ways, not in all ways, in that you can hit it with a much larger input space. And so it's changing how I'm thinking about kind of getting to the output that I want and the output is a bit different. It's not the same. I'm getting different outputs from all the models, so they're not all the same. Definitely. I think my biggest thing is with all these new releases, I'm trying to figure out how do I use each one, when do I. I'm trying to develop my own strategy on when do I go to ChatGPT by default. Like, when's that the right thing? And that's changing as we'll talk about with things like plugins and stuff that's evolving. But then Claude 2 comes out and then you have, you know, on the open source side, as we just Talked about with llama2. So I think trying to understand all the tools in the toolbox in relation to each other has been interesting. So Claude 2, I'm really focused right now primarily on large content output is kind of where I've landed on that.

And the hundred K context length of Claude 2 is something I find really compelling as well. There was also a significant paper that came out that caused a lot of waves in terms of context length. And thinking about that, which showed kind of as you increase context length, you lose any significance of the middle bit of that context. So the beginning and end is more important in terms of what makes the output of the, of the model quality or not in terms of how you would measure that. And so we'll link to that paper maybe in the show notes as well. But I've tried some things, I mean, I don't know exactly all of the details. Again, Claude is one of these closed models, so I don't know all the details of how they're doing things. And because it's sitting behind an API, it's hard to know how those things evolve over time. But for example, I took one of the things with Claude 2 is I just took one of our complete podcast transcripts. So a full episode, so 45 minutes of audio transcript. I took episode 225, which I really enjoyed, talking a lot about the things that I'm working on right now with Prediction Guard and just asked it to give me a summary of the main takeaways and you know, pasted in the whole thing and it's like a fairly good comprehensive takeaways. Like many companies ban usage of certain LLMs, blah, blah, blah. You know, Prediction Guard is trying to provide easy access, structuring validation compliance features for LLMs, making LLM usage easier, blah, blah. And it gives these great takeaways. And then I asked, you know, hey, suggest a few future episodes that we could do that maybe cover related topics, but things that weren't covered in this episode, pretty good. Some of them are kind of generic. Right. A look at current state of AI agents and automation. How close are we to no code, AI app generation, blah, blah, blah. So that all kind of off of this large context of the transcript input was quite interesting.

I'm curious, I'm going to put you on the spot also, as someone who's working on your own product, and I know this is not a Prediction Guard episode, but I'm asking on my own behalf and on behalf of the listener, how do you, as someone who is looking at these different models, how do you think of those different models? How do you kind of structure them in your mind in terms of what you're offering? You've been evolving rapidly over the last few months and I'm always curious to see kind of where your head's at on this now as you're looking at them.

Yeah, I think the things consistently that I'm seeing are that I made a post on LinkedIn about this as well. Even my own applications that I'm building, LLM based applications, having access to multiple models rather than a single model I think is a really nice usage pattern where the easier we can make it. And there's. There's other people that are doing this as well. Other. You know, in Prediction Guard you can query a whole bunch of models at the same time concurrently. There's other systems that will let you look at that output as well. So nat.dev and some of the toolbar stuff that Swix is doing, we had a collaboration with him in the Latent Space podcast. So the more you can tie these things together and look at the output, or automatically analyze the output of multiple models at the same time, I think that's really useful because it's hard to generally evaluate these models until you start evaluating them for your use case and building intuition about them for your own use case. So I think the pitfall that people maybe fall into is saying, oh, I'm going to use this model before they've even tested that for their use case. Right. Try creating a set of evaluation examples for your own use case and then try out a bunch of different models for that. And also try out the things that are becoming more standard kind of operating procedures for building LLM applications, like looking at the consistency of outputs, running a post generation, validity or factuality check on the output, so checking a language model with a language model, doing input filtering and all these sorts of more engineering related things. So, so those are some of the things that I'm seeing. But having access to a bunch of models at the same time I think is something that can really boost your productivity.

I appreciate that. And to our listeners, we're not making it a prediction guard show or episode, but as a co host, Daniel's excursion through this in his professional has made him, in my view, one of the world's true experts in how to look at all these together. And since we have the benefit of him co hosting the podcast, I'm going to continue to take advantage of that expertise for all of us.

Thanks Chris.

So sorry about that Daniel. Sorry for putting you on the spot.

Yeah, no, no worries. I think the other thing maybe to highlight with Claude 2 and something that you were talking about in chat before we jumped into this episode was Claude 2 versus or maybe anthropic and their offerings versus OpenAI. How do we understand that? Like how, how do we categorize these things? I think one of the interesting things with Claude 2, so we've seen both Anthropic and their Claude Models and OpenAI and their GPT models increase context size over time. GPT models not quite as far as claude, but both have increased. They've also both added in some of this functionality, which I think is very interesting. Claude 2, I think first, if I'm not wrong, the ability to add in your own Data. So in Claude 2 there's a little attachment button and you can upload PDFs or text files or CSVs and have that inserted into the context of your prompt, which I think is of course extremely powerful. We've talked about adding in external data into generative models and grounding models in the past. It's very powerful. Now OpenAI is doing this in a slightly different way and I think this is something worth calling out on the podcast is with their new code interpreter beta feature within ChatGPT, you can upload data, but it's processed through the code interpreter in a different way than what CLAUDE is doing. So we all know that ChatGPT and GPT models can generate really good code and specifically good Python code. And so what OpenAI has done for their kind of data processing agent within ChatGPT is said, well, let's just have our model generate Python code. Then we'll hook up the Chat GPT interface to a Python interpreter and just go ahead and execute that code for you over your data and then give you the the output. So this is maybe a distinction that people can have in their mind. Claude2 you can upload this huge amount of context. You can upload files, insert it into the prompt. As far as I know, they're not running any kind of code interpreter type thing under the hood. ChatGPT might not be inserting all of that into the prompt, but they're actually saying, well, what if we decompose what you're wanting me to do with this external data into something that can be executed by a sort of agent type of workflow where you upload your data and ask me to like do some analysis over it. I'm going to generate some code. So the language model generates some code and then that code is actually executed in the background. It returns a result which is then fed back through a model to give you generated output back in the interface. So it's actually a multi stage thing happening in code interpreter in OpenAI, it.

Effectively produces a no code solution, you know, where you get an output and you're just kind of skipping the whole thing, you know, instead of, instead of using the language model to generate your own code and to be your code assist and all that. And then you're still doing it. It's kind of skipping that whole step right there.

Yeah, and I can give an example I actually ran prior to the show, so I have Claude and the OpenAI code interpreter side by side open. I uploaded a file with a bunch of Yoruba, which is a language in Africa, transcriptions out of audio which are from the Bible TTS project that we worked with Koki and Masakane on. And so I uploaded this file which includes this Yoruba text in a CSV format. OpenAI said, Great, you've uploaded this file. Let's start by loading and examining the context. And then it has this sort of show work button and you can see the actual code that it generated, which is pandas code to import the CSV and then output some examples. And so you can expand that and actually see the code that it ran under the hood and the conclusions that the agent came to. Then I asked it, okay, well, plot the distribution of the transcript links. Are there any anomalies? And then again it says, hey, show work. And you can see it's importing matplotlib. It's taking in the CSV. It's actually creating the plot and actually generates an image out of the transcripts. It says, I didn't find any anomalies. They're all kind of within the same distribution. There's not any anomalies. Then I asked it, can you translate all the Yoruba to English? And that's where it ended up stopping because it said, no, I'm not good at doing that. And Quad actually stopped there as well and said, no, I'm not going to do that. I also uploaded the Yoruba alignments to Claude and it said, hey, sure, let me analyze these transcripts. And it just output some general, like there are 50 audio links, the transcript links. There's no Python code there. It just gave me some takeaways, right. And then I said, are there any anomalies? And it said, I checked and I can't find any and could you translate it? And it said, unfortunately I can't. So it's all still a chat based thing. So you can see kind of different approaches to this complicated workflow of having almost an assistant agent executing code for you versus putting more context in the language model and having it reason over that context.

So they're almost getting their own strengths at different types of approaches to problems. Would that be fair? Yeah. So that's another way of thinking about it is as you start understanding how the different large language models approach a problem and the tooling that might be better or worse for a given use case that also will help you kind of pick which way you want to go in addition to maybe just using multiple models as you talked about earlier.

Yeah, exactly. And there's so much to dive into on all these topics that we've covered today. I'm going to make sure that we include include some really good learning resources for people in the show notes. So make sure and click on some of those. There's a guide from Datagen on the neural radiance field stuff, the Nerf stuff that you can learn a bit more about that there's a hugging face post and a Phil schmidt post on llama2 that are both really practical kind of how do you run it? How do you fine tune it? What does it mean? And then there's a nice post from the one useful thing Ethan Moleck blog or newsletter about code interpreter and how to get it set up and some things to try. So we'll link that in our show notes and I think people should dig in, get hands on with this stuff. Things are updating quickly and the only way to really get that intuition about things is to dive in and get hands on it is.

It's the most interesting moment we've had in the AI revolution of recent years and just so much cool stuff right now. Anyway, thank you for taking us through all the understanding and explanation of these things.

Yeah, definitely. It's good time. Hopefully people enjoy the rest of their week and maybe go see Oppenheimer or Barbie, depending on which of those is most interesting to you. But we'll see you next time, Chris.

See you later. Thanks.

Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already, and if you're a longtime listener of the show. Help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to at fastly.com and fly IO and to our Beat Freakin residents, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.