Broccoli AI at its best 🥦

Welcome to Practical AI. If you work in artificial intelligence, aspire to or are curious how AI related tech is changing the world, this is the show for you. Thank you to our partners at Fly IO, the home of changelog.com Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents so you can launch your app near your users. Learn more@fly IO.

What's up friends? Intel Innovation 2024 is right around the corner. Accelerate the Future Registration is now open and it takes place September 24th and 25th in San Jose, California. This event is all about you, the developer, the community and the critical role you play in tackling the toughest challenges across the industry. Ignite your passion for AI and beyond, grow your skills to maximize your impact and network with your peers as they unleash the next wave of advancements in technology. Here's what you can expect Understand the emerging innovation and trends in dev tools, languages, frameworks and technologies in AI and beyond to empower you and the solutions you're building. Get in depth technical experience doing hands on workshops, labs, meetups and hackathons to collaborate and solve problems in real time. You can explore featured partner and intel solutions. They have partners there, startups there, customers there and intel is showcasing the latest in products, services and solutions across keynotes, tech sessions and the show floor to help you meet your development needs. Collaborate with experts, learn and have fun. Engage in interactive sessions to connect, get certified, gain unique ideas and perspectives, build long lasting networks and of course have fun and get inspired. Hear from leading industry experts, technologists, startup entrepreneurs and fellow developers along with Intel Leadership CEO Pat Gelsinger and CTO Greg Lavender as they take you through the latest advancements in technology. Don't miss this chance to be at the forefront of innovation. Take advantage of early bird pricing right now until August 2nd. Register using the link in our show notes or to learn more, go to intel.cominnovation once more. That's intel.cominnovation or go to the Show Notes and click that link.

Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am Founder and CEO at Prediction Guard and this is a pretty special and fun episode for me because I get to kick back with an old friend of mine. We went to the same university. For those that haven't heard of it, Colorado School of Mines in Golden, Colorado. Of course, a shout out to all the ore diggers out there that are listening. But yeah, we have with us today Bingsung Chua, who's a data scientist now and working in the, in the energy sector. I was really fascinated to talk over the years with, with Bingson about, about all the things he's doing and in particular his kind of approach and learnings around active learning and NLP models and yeah, I wanted to invite him on the show to talk through some of that and learn a little bit from him. So welcome to the show. How you doing?

Hi Daniel, thanks for having me.

Yeah, it's been a while since the days in Colorado School of Mines in cool days. And now you're working as a data scientist in the energy sector and also working in Asia, which is super cool. I'm wondering if you could give us a little bit of a sense of some of the unique things about doing data science and machine learning type of things in the context of like the energy sector in the context of like an actual enterprise real world kind of situation. Because we talk a lot about recently, we've been talking a lot about all of these genai models and APIs and such and that is super cool. But also there's a lot of on the ground work going on in data science that maybe looks quite a bit different than that. So.

Yeah, thanks. So I mean I work in the energy sector and it's pretty much a traditional type of sector. A lot of the companies, as you go around, at least in Asia or at least over here where I'm at, we do not actually even have things like cloud services or subscription and stuff like that due to different reasons and stuff. But at the same time there's an appetite for machine learning, AI data and all of those things you see people talk about Gen AI as well, but I think the ones that I've noticed, at least for me personally, that has really brought a lot of values are kind of what you guys were talking about in a previous episode of Broccoli AI.

Broccoli AI, I love it.

Yeah, not so sexy, but still really important. Really brings value. Particularly I guess what we're going to talk about is active learning in the context of nlp natural language processing. And so I think that's a pretty exciting place to be in to do. I mean it kind of translates into geni and to a certain degree, but at the same time I think to at least the context that I'm in as well, in the sector that I'm in, we do not have things like cloud services ready for us. Right. And so you have to figure out ways to kind of bring that about in an on prem server vm. So how do you work around that? And how do you actually bring in cloud native modern technologies within a traditional kind of structure? So yeah, yeah.

And from at least my impression, even though there's sort of not this, whether it be for security reasons or legacy reasons or just connectivity, you know, there's not the type of connection to cloud services like you're talking about that others might be working with. But at the same time at least my impression is that this sort of sector and maybe others, there's other related verticals where they have been sort of data driven to some degree for some time. And I don't know if you could speak to that, like the, the types of data that people have been processing or storing or are available in those contexts.

Yeah, I mean that's a good point. Because a lot of these traditional industries, at least in the energy sector that we see, we have sensors that are constantly flowing in data. All the time the data is there, I mean we are collecting data, right? And then going a little bit deeper then you find that a lot of us have been collecting a lot of unstructured data too. And so at least within, you know where in my experience, at least what, what was happening was when I came in, I was pretty much the only data scientist and stuff. And so I had to like make my existence justified in a sense. And so I knew that like I had to do something about like bring value within this organization. And I'll be able to kind of prove that like hey, look, data science actually does work and does bring value, but the quickest, I guess, low hanging fruit that I found within, at least in the context of where I'm at, is the whole unstructured data. So we have been collecting thousands and thousands, if not hundreds of thousands of unstructured data. But there has never been a way to really analyze that at scale. So people have been analyzing it, they've been able to do some sort of human analysis on it, but there's never been someone who's able to say, hey, what's been happening in the past 10 years that we've been collecting all this data? What's it telling you? Nobody has really been able to do that. So I thought, okay, maybe that could be one thing that we could actually bring in is that you have all this data that's ready for you, that has all the insight, that has all the information that is locked up waiting to be unearthed, pretty much waiting there for us to just extract it or mine it. So I did a quick POC for one of the departments company and said, hey guys, you've been doing power bi, tableau. Power bi kind of thing. You've been able to do stuff with your structured data, right. And you've been able to plot it on beautiful graphs and stuff like that. Being able to analyze in that sense. What about all the instruction data that you've been collecting over the years? And they said there's no way we could do it. Right? I mean, like, we've got like thousands. There's no possible way for me to put it in power bi. And so I said, like, well, maybe we could explore ways that we could actually get into machine learning to actually help you to scale that analysis from an NLP standpoint. Thankfully, they bought it and so we took off from there and it was pretty cool. I mean, it was a journey for sure. Journey to learn. Yeah.

So like, when you say unstructured data give people a sense of like the kinds of files or not maybe the specifics, but you know, I'm imagining a file store with some type of files in it that contain. Yeah, something give people a sense of that.

Yeah, for sure. I mean, I guess unstructured data, typically we think about it as like text, right? It could be text or it could be something that's just out of structure.

And in like Microsoft Docs or.

So we have been storing all of those data within SharePoint. Microsoft SharePoint. Right. And so what I've seen is these unstructured data is usually. It usually comes in a tablet form.

Right.

You have a table that is collecting all of the structured data. But alongside with it there's always a comment or that additional things that you have to actually tell the story of what you're actually collecting. And those are the ones that I think it's always locked up. And you would see it's very typical in any kind of industry where you have a tabular data that collects sensor data and everything, or reports of what's been happening, and those are structured. But then there's also a column that would bring in some sort of remarks, observation or actions taken, whatever it is.

Comments.

Comments. Yeah. But more often than not, we just kind of play through it and kind of not really put too much attention to it, at least within the data set that we were looking at. We had found that there were a lot more insight in those data than the structured data that they've been collecting. I'm talking about safety data. So we've been collecting safety reports every single day, a couple hundred, sometimes tens of them every day over the years. And so these data had some sort of insight Right. And it brings in an insight of how is the safety condition in our operations. And so with that, like the people usually see the categories or what the reports are being reported for, but then like when you look at the categories, you realize that sometimes it doesn't really jive with the stories that they're actually trying to bring in in unstructured data. And so that's where the, the part where we felt like it's going to bring in additional insight to the unstructured data that we see.

Yeah. And so like there's this, we've talked a little bit about like the data that you was there, the potential insights in that around safety and maybe other insights as you kind of came into this industry and kind of were getting an understanding of like let's say that you built the coolest data science app that was out there and had a cool model in it to do some analysis. Like, what is the reality of how that would have to run? Like, what does in production mean in your context?

Yeah, so many things. I mean, just start from the data. We don't even have labeled data to start with. Say you want a classification app or model, you need to have some sort of a label. Right. And we don't have that. And so you have to figure out ways to bootstrap your labeling process to start off and stuff. And then all the way down to like, of course, training. The model is pretty easy these days. Right. And but then you have to think about things like, hey, what does that look like to put it on the server? Most of our servers are running Windows Server. And so I've had experiences putting in production some apps on Windows Server and that was painful. And so like we have to figure out ways that like work with the it's and stuff. And said, hey, can you deploy a Linux server for us instead and just work it out from there and set it up from there. That being said, like that's just over all picture of it. And then you get into the details of like how do you actually store your model. You've got to have some sort of infrastructure to kind of pulled at which, you know, in our case felt like MLflow is pretty good model registry experimentation, tracking and stuff to keep track of what type of models that I'm using and stuff like that. And then so many things, honestly. Gosh. And then like, how do you actually put it on an orchestration kind of service. You could use Cron jobs. But then, you know, it may not be so flexible. Then you kind of need to work something out and so you have to get some sort of orchestrator to spin it up and make that a service for your infrastructure as well.

I love this discussion because I think it fits the theme of the show so well around being practical and the fact that, yeah, I'm sure that there's actually a good number of listeners out there who are really wanting to do machine learning, AI, data science type of things and they are sort of in a similar situation in their company because actually I think probably more of the majority of companies are in this sort of situation than sort of infrastructure wise, extremely modern and just cranking everything out on, on kubernetes in the cloud and that sort of thing. So yeah, I love this. So you were talking a little bit about the sort of problem of so you have all of this tabular data with extra comments and unstructured data and certain things you want to do like extract insights or classify maybe some of the unstructured data. But then also nothing has been labeled over time, it's just unstructured data. So talk a little bit about that bootstrapping problem and how you thought about that in terms of I've got all this stuff, I want to create a model, but I have no starting point.

When we were going through that whole labeling process or data preparation process. It's pretty interesting because we really didn't have anything, no labelers or anything and I didn't have a budget to get an external labeler annotated from me.

Just hire thousands of people online.

Right? I know I could just do that maybe. But then again even that like I've thought of it and our data is sensitive to start with but at the same time it's so nuanced and it's so nuanced the context of our company. And so a lot of data that we, that every company has is just so nuanced to their own context, you know. And at the same time too, one of the nuances is the way that like these texts are being written. And so a lot of code switching happening, you know, which means code switching, which means like in Asia a lot of times we speak in English, but we will kind of put in some native languages that we know or that we grew up with. And so it's just kind of like you go back and forth, back and forth and it's just kind of kind of a common thing, especially in Southeast Asia. And so you can't just hire somebody online and just kind of label it for you because you just couldn't. I don't even know how to bring in a context to these guys. Right. And. But thankfully, I mean, I would say that the part that really helped was I happen to have a really good sponsor for the project. And these sponsors, they were super on board. They were not technical, they were SMEs in their own department and they knew this stuff, but they know enough of machine learning and data and AI that hey, it's kind of a model that predicts not necessarily making 100% accuracy get go. And so they understand those nuances in essence. And so they kind of supported that and understood the kind of things that we have to go through, you know, as a practitioner that we have to go through to kind of understand that part of it. So that was really helpful for my part, because having a good sponsor means you get really good support for the project. But that also means that because the product that I'm working on, the app that I'm working for on is for their people, you know, their subordinates and the people. And people were reporting under them. And what happens is they said to them and say, hey guys, this is your app. I want you to help Bing soon out with building this app. And so having the users themselves on board from the foundational bootstrap level really helped us. So because we didn't have any labels, they had the guys to actually be the ones who label for us. They were the labelers. And so the, the users themselves were the labelers. So that was that. Honestly, I was pretty really blessed to. To even just have that that kind of worked out together. I think that that kind of worked so much in my favor.

Yeah, yeah. And in that labeling process, how did you develop your sort of set of instructions for like, how you like explaining the problem to them or helping them define the problem and the categories, for example, in a classification model, how was it for you? Because I've also had the experience personally probably and been burned a couple times where I'm like, oh, this problem makes sense to me. I set up the labeling thing, I release a bunch of labelers in there, and either the instructions don't make sense or I've biased this in some way or, you know, likely because I, like you had mentioned, I wasn't super close maybe to the, to the users in that situation. But yeah. Any learnings from that experience?

I think a lot of iteration with them. I had so many times, like I would travel to see them. They work in our operations. I literally travel there to see them in person. And I said I would, we would just go through, hey, These are the labels that we want to label. We kind of get a general idea of what they are. But when you get into the weeds of it, like you, when you get into the details of it, you're like, you would think in this situation that label should be here, should be number one. But then, no, somebody else said, no, it's two. So the way I worked it out was there will always be contention. I noticed no matter how tightly knit your labelers or your team are, there will always be contention. And you just got to work around with it. At least in my experience, I just had to work around with it. And the way I worked around with it was I just kind of had a voting system, you know, and I set up an account. So the technical side of this is I, I could have just given them Excel sheets and they could just label them on an Excel sheet. But I find that, you know, they are doing both me a favor kind of thing. But I want them to have a really good user experience instead of just going through Excel sheets and stuff. And so I used Argila back in the days and when they started.

Pre hugging face days.

Yeah, yeah, exactly. And I noticed that Argila is amazing in the sense that like, it allows you to set up different users and you could, I mean, even other kind of interfaces, I've used labels too as well, but Argila was able to, I could use the API and just kind of like set up each user, right. And then for each user, I'll just kind of like sample the same data for them. So I had to actually go through the first round, the same number of set of data for them to label, say 500 of them, I think I remember. And they would all label within two weeks. And at the end of that two weeks, I'll collect them and I'll find which one, which are the most contentious ones. And so the ones that are the most contentious, the ones that have the least percentage of the majority, I would pull it up. And I said, hey guys, what do you think about this? This is contentious for you guys. Why is it contentious? And you work it up from there, right? Because chances are you're going to see the same kind of a label again or the same kind of a data again. And if you talk it out, hopefully when you see a similar thing, and you said, hey, I actually, we already talked about this. We all agree that we're going to go with this. And so that's the first round. We had the same labels, the same data set for everyone. It's a bit inefficient to a certain degree, but I think it's important to actually get into that space so you understand the contentions of each person. And then the second round I have everyone just kind of like label at scale pretty much. And yeah, we collect that from there pretty much.

Interesting. Yeah. So you did, in this process, you did a initial sort of offline bootstrapping of labels.

Right.

And did that. So like scale wise, what sort of scale when you're solving. So here we're talking about an NLP problem creating a classification model on some labels of this unstructured data. Of course this would vary by domain, but sort of what scale of labels did you shoot for when you were doing that initial? Trying to get to that place where you could start up and train your first model?

We did just about 1800-2000 labels or rows of data basically. And then we start training our first model.

That probably means you're not training like a 400 billion parameter model with 1800 samples.

You don't even have infrastructure to be able to train that.

Yeah.

No we don't.

So what does a broccoli AI model look like?

I mean these are all text, right? And so we were going with the simpler, simplest you can find on Hugging face at that time. I think at the time Sentence transformers were really making it big for different reasons, whether it's topic modeling or classification. And at the same time too, I remember they came up with the set fit model which is fine tuning the sentence transformer, which was honestly revolutionary for me. Amazing. And I thought it was amazing that you're able to do something that was meant for similarity, but then you could actually fine tune it for classification and with pretty good performance. And it's supposed to be something that is a few shot classification model or few shot kind of fine tuning. And So I thought 2000 should be enough for me to start somewhere. Right. And in fact when I trained that I tried some other models, but I think sentence transformers were the ones that actually gave the best performance out of all. It still wasn't that good, you know, talking about like 60 something, 70% kind of thing in terms of F1 score. But when I talked to my sponsors about this, I said hey guys, like you're okay with like me deploying this at like 60, 70%? And they said no, actually that's fine. Right? Because the objective for this was number one to bring visibility of these reports to the users. Because one of the pain points that they said was for us to be able to Know what people have been reporting at least in the past 24 hours, they had to get on SharePoint and just different hoops and loops to try to find out filtering and stuff. But to be able to get that sent out in the email with the classification was already win. And so I thought okay, let's do that, but let's not stop that, right? I mean we should actually create a pipeline and that's where the active learning comes in. And it really helped because I'm glad that I actually used RGLAB to start with the bootstrapping of our data set and having that rglab which means our users are already used to the interface and they already have an account. And so I was able to kind of hack around with the Argila as a Python API and basically I was able to create a loop where pretty much what this model does every day, it will bring in the new data that people have been reporting for the last 24 hours and make some prediction on it at about 60, 70% F1 score accuracy, whatever it is and then send it out to the users and these users will see it and at the end of that email they will say hey, I don't think this is that signal. It should be this signal I want to give a feedback and at the end of it they're able to click on a link that brings them to their profile in Argela that will allow them to give a feedback for the particular day data set. And so over time now it's in production every day from time to time I'll get people giving their feedback and we've gotten like close to 4,000 data sets now label from this active learning. And so we will train the model periodically. Not on what I could have done it automated but I didn't really want to just I didn't feel the need for it yet to put it on automation. But then at the same time like you know, you're just collecting an end event and we're just training it from time to time basically. Foreign.

Hey friends. Outshift. Cisco's incubation engine merges innovation with the art of possible a launch pad for transformative emerging tech Out Shift blends startup agility with corporate strength to develop next gen technologies from the ground up in AI quantum technologies, cloud native and more. Their newest AI innovation, motific addresses a critical challenge in the rapidly advancing world of gen AI. Bridging the gap between concept and deployment, this model and vendor agnostic solution supports the entire gen AI journey from assessment and experimentation. Motific accelerates deployment from months to days while safeguarding against Gen AI security, trust, compliance and cost risks, all while empowering business function and IT teams to rapidly configure end user assistance. Powered by organizational data, motific provides advanced customizable policy controls to prevent unauthorized access to sensitive data and helps ensure compliance throughout the entire process. With deep visibility into operational and business metrics, motifik enables you to track roi, optimize costs, and make informed decisions. By offering a centralized view, motific deters shadow AI usage and empowers teams to innovate responsibly. So move beyond the traditional constraints of AI implementation, utilizing AI deployment that is both responsible and is revolutionary, ensuring your projects are not just quickly launched, but built on a foundation of trust and efficiency. Visit Motific AI I.e. m O T I F I C A I.

So Bingsun, it's super interesting to hear kind of how the you were able to engage the the users of the application through this like reporting process, essentially that they were, you know, had some of the right incentives in place to, to respond and to give you updated labels. And you mentioned also the model repository saving models, getting them out with mlflow. In the context of you deploying your model on prem you updating the model, you just mentioned kind of retraining the model. What does that look like for you right now in terms of that cycle of when you would want to push out a new model after gathering this data, how you would judge that to be worthwhile or useful and any sort of testing that that is relevant to that cycle of getting in new labels, retraining, you know, evaluating that sort of thing. What, what does that look like for you and how do you kind of put in the right or, or how have you thought about the right metrics to understand when to update the model at this point?

Honestly, we keep it simple. We just kind of like periodically do it at a cadence, you know, a couple months or two. But I did think about like what does it look like to actually measure the drift of the data and stuff like that of the model predictions? That could be one of the ways that we could do it too. But what I'm seeing is actually the model is doing its job fairly well, well enough to actually solve the business problem. And so we don't see a need to actually implement a more sophisticated monitoring unless we need to. You know, that's where we're at with it.

Yeah. And when you push your model sort of like you update it. You had mentioned the model repository. How are you shipping your model out to the application. Because I think, like you had mentioned, you know, you only have so many resources. I think there's also a lot of people out there in your situation where I think it was Kristin Lum on a previous episode, she had talked about kind of that data scientist out there that is maybe one of very few or the only data scientists in a potentially a large organization and having to, like, do all of these things. They're not like an ML Ops person, they're not a model trainer, they're not a observability person. They're doing all of that. Right. So there are limitations to, you know, how much sophistication you can put in place. And I think that, like, some people go way too far and they're like, oh, I'm going to implement all of this stuff. And it actually makes their life as a practitioner less happy than otherwise. So, yeah, how have you found that balance? And like, what does it look like for you to do these cycles in terms of tooling and the things maybe that you've, like you say you mentioned you thought at some point maybe it's relevant to implement some of this observability stuff, but maybe not yet, or there's other priorities. So what does that look like for you in terms of how you decide what level of sophistication is Right. And how you push things out?

And I'm 100% with that, honestly, because it is a matter of priority. My customers are happy, I'm happy, and I'm not going to, like, change what's, you know, what's good or I don't want to break what's been working. Right. So to speak. But that being said, like, I, you know, when it comes to, like, all of that, I think I have a general idea of what would be the minimum thing. So now I'm working on some other things as well, like anomaly detection and stuff like that, which needs to be deployed. So having gone through that, that kind of like set up a, like a pseudo infrastructure for me to know what kind of infrastructure that I'm going to be looking for, for whatever else that I'm working on. And at the bare minimum, I think model Registry is super important. And being able to call the different versions that you've been training and being able to track that and being able to call it through an API, through a function, you know, ML Flow has this great Python connection with it. And so being able to do that is just amazing. I mean, it keeps my life sane, right. I don't have to like, figure out where I store my model pretty much. So I would be doing this exactly the same things with whatever I'm working on next, which I've since moved on from that project and I'm just kind of like maintaining it. That project's count now in maintenance and now I have to move on to a different project to solve a different part of the business, you know, in that sense. But that project kind of set, like I said, set the foundation and knowing what kind of things that needs to be done. So sorry, I'm kind of going ahead of myself. So1 is mlflow being the most important thing for me in terms of this sort of scenario. The other one is Orchestrator is also really important, having a really robust orchestrator. So for me, I think Prefect was perfect for me and I was able to do different things and stuff.

The types of things that you're orchestrating are what types of things.

You could do it real time with Prefect. At the same time you could also be listening and stuff, but at the same time you could also just running on schedule, calling different functions, sub functions and things like that. So that was really cool to be able to have that. That's pretty much what we do right now. We are not really going into real time monitoring yet. Until we do that, then we'll have to figure out something else more sophisticated.

Yeah, yeah. And are you just shipping your models sort of as part of Docker container or something like.

Pretty much, yeah, we do use Docker containers and just so that we can keep it contained in that sense.

Yeah. That's awesome. I think you had mentioned in one of our conversations something about DuckDB. Where does that fit into some of this?

So the data, the raw data that we get from is from SharePoint. But if you have anyone who has any experience with SharePoint in terms of like wrangling and data stuff, it's so painful.

Yeah.

So I, I thought that would be good to actually have some sort of a middle layer, mini lake house or data lake kind of thing. And I didn't want to bother my IT guys too much. So I thought DuckDB is a great thing for it. Right. I don't need a VM for it. And you can have an embedded SQL service that you can use. So that's being pulled every day, pulling the data into DuckDB and DuckDB will be the one that actually cleans up the data, preparing the data to send it to the model and that becomes like a pipeline for me to be able to work around the whole complexity of SharePoint really honestly, yeah.

I have personally found a lot of use for DuckDB even in the past. Yeah, even in the past year on the. Even on the more Genai stuff where you're doing sort of like text to SQL or like queries and that sort of thing. And every company we're working with has different crazy sets of data or different configurations of this or that and that layer of having a kind of unified analytics layer, but also not that sort of, you know, easy to pull in to Python, easy to spin up, easy to like test with locally and then like deploy with. Yeah, that's been really useful.

I remember you talked about LANSDB for RAC and things like that. And it's the same thing. I love embedded database. I think it just works well and it's kind of scalable eventually. And I think. I really like that.

I think there was one blog post I've always referred back to because I also went through the. You and I were at mines at the same time and then there was data science hype and then there was like the big data period where everybody was in Hadoop and Spark and all this stuff, which I know a good number of people still use Spark, but there's a blog post by the mother Ducker company. Yeah, but I think the title is Big Data is Dead or something. It basically goes through some of the discussion around like, hey, we all thought we had big data, but like the actual query problem, like the types of queries that we need to run, these aren't like big data problems. What's needed is different. So yeah, for those, shout out to whoever wrote that blog post because it was really, really good. If you ever want to come on the show and talk about it, that, that would be awesome. Yeah, well, as you kind of look back on this process and some of the things that, that you've learned, like what are you, what are you looking forward to in terms of like the future of the process of your own work or of the things you're learning or maybe like as you go into this next phase, it sounds like you're working on some new things. You'll want to reuse some of the tooling and kind of process that you have used. But you know, what's different or what are you excited about kind of for this next phase in light of what you've kind of learned over the past years.

Generally Speaking, I think MLOps is just so nuanced in different contexts. Everyone has a say of what should be done. And I think if I learned something from this, was Nobody really knows everything, so you kind of have to figure out from there and you kind of take a risk on certain things that you decide in terms of your system design and stuff. What I'm excited for is actually to be able to take this and see what it looks like for other things in other applications, whether it's an anomaly detection or whatever it is. In a broader sense, I think I'm excited to see things like embedded database getting more and more mainstream, especially in the context of LLM and Genai and stuff. I'd love to see that getting more and more mainstream as well. One of the things I'm always thinking about is skill is one thing, because a lot of the applications that we talk about today, especially in the context of Genai, we always talk about the bigger compute and bigger scale. I would love to see that getting smaller, which it is happening now, getting more accessible on different devices and stuff, being able to do more cool stuff on device and advanced stuff on there. Excited for that too.

Yeah, I think there's a lot of people excited for that and sort of this new phase of AI where people talk about AI everywhere or this sort of thing which in reality, you know, there's been machine learning and data science sort of everywhere for some time. But that that sort of wave of these newer generation of models kind of being runnable in, in more practical scenarios is, is exciting. But yeah, thanks for, thanks for joining Benson to, to talk about a little bit of your broccoli AI. It's been, it's been fun and love it.

Thanks for indulging me.

Yeah, you and I can can hype the broccoli AI and I'm sure we can get Demetrios to help us hype it too. Yeah, I don't know if he trademarked that term. He's got it in, in his hype cycle now, so love it. Yeah. Yeah, thank, thanks so much for joining and hope to talk to you again soon.

Thanks. Thanks for having me.

All right, that is practical AI for this week. Subscribe now if you haven't already. Head to PracticalAI FM for all the ways and join our free Slack team where you can hang out with Daniel, Chris and the entire Changelog community. Sign up today at PracticalAI FM. Community. Thanks again to our partners at First Fly IO, to our beat freaking Residents Breakmaster Cylinder and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.