Creating instruction tuned models

Foreign.

Welcome to Practical AI. If you work in artificial intelligence, aspire to or are curious how AI related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more@fly IO.

Hello, this is Daniel Whitenack. I am here on site at ODSC east in Boston, the Open Data Science Conference and I am super excited because I get to sit down with Aaron McHale Staples, who's a developer community advocate at LabelStudio and yeah, what do you think of the conference so far, Aaron?

It's been first off, super fan of kind of what you've been doing and a lot of anybody who's creating stuff out there in this space, especially with the current like zeitgeist and explosion of interest in AI and machine learning, it's.

A little crazy, It's a little wild.

It's a little wild. I, you know, I would be lying if I'm not. Wouldn't be saying I'm, I'm newer to the field myself, but it's been something I'm being very fascinated about. But all that being said, this conference is really cool to see just the breadth of first off people here. There are people who are very new to the industry, people who came to just learn more for their first time, but there are people who had been practicing for years and years and this is their third or fourth time at odsc. And I'm also really interested about the number of people concerned about data integrity here.

Yeah, lots of like interpretability integrity, reliability type talks.

Yeah, lots of reliability. The other one is like also on missing data and like how do we approach these problems especially, especially with the rise of foundational models and generative AI. Like how does that impact it for the long. Which is crucial conversations I think to have.

Yeah, definitely. And what sorts of different players in the space are you seeing at this conference both in terms of like open source or like different kind of targets like ML ops platforms, that sort of thing. How do you see that developing?

First off, I'm personally a huge fan of open source. It's not only how I learned to code in the first place, but just a big believer in the ecosystem. I'm a huge believer in open data. I'm a participant in Open Data week. So I think all of these things.

Are like, and you're wearing a pie lady shirt, which is awesome.

Yeah, I'm a member of PI Ladies, so again, super important, I think, to have all these things in the ecosystem. But one of the things I think that stands out is there's so many new innovations that, like, if you're starting a tech stack from ground zero, it's really fun to see all the different players in the game. So selfishly working at LabelStudio, one of the best things about being in the space right now is we're a cool platform because we can integrate with so many different data types that it means that I also get to play with almost every other tool or workshop or players in the ecosystem, which is selfishly fun. It means I get to have more things to integrate with or build. As always, we're huge fans of what the Pachyderm team is always doing. Work very closely with them. We work very closely. We've got a lot of friends and fans. The DVC crew, they're not here at this conference round, but did get to work with them at PyCon, which was really amazing to see kind of the work that they're coming out with. The Condo crew is always fun to see around, so that's always exciting. Cool.

Yeah, yeah. There's so many awesome things going on and I've seen like maybe like three or four open source packages that I like. I don't know if I've been ignoring or I haven't heard about. So that's like one of the fun things about coming to these things. I know. Also, you gave a recent talk at PI Data Berlin about reinforcement learning from human feedback, I believe was the topic. Could you tell us a little bit about like the general like pitch or angle on that talk, which is definitely like a key topic these days with all the instruction tune models that are coming out and all of that. So what, what was your kind of angle in terms of what you were thinking about there?

Yeah, so I think one of the cool things is it was a talk that Nikolai and I gave and Nikolai is the CTO and one of the co founders of Label Studio. And what we did at Berlin is we really made sure to expand on this idea that like, yes, these generative models, these larger models are kind of becoming the norm. I think we met yesterday. I was talking to someone who's like, I got interested in AI because I made a thousand things with Mid Journey and I'm like, cool. And I'm very fascinated by. And I'm a believer, like, I don't care how you got into it, but Just the curiosity to show up to a conference and learn more is very fascinating. But explaining to someone how it works and then also explaining the best practices behind it is really important. Personally, I have a journalism background, I have a liberal arts background. And I think it's really important that we incorporate the humanities in technology for the long run. And so when it comes to reinforcement learning, all of these large generative models, they can all be made just a little bit better with the human signal that we can provide. And we can say a lot of things like, you know, get into prompt engineering, which is another whole other topic. But it will never be as good as like if you can retrain your own data set with subject matter experts or to a specific use case or condition that you're trying to output that data towards.

Yeah, and I think one of the things that's been on my mind recently is like this topic, reinforcement learning from human feedback. Especially with what's gone on with ChatGPT and all. Sometimes it feels like out of reach for like day to day data scientists. Like, how do I, like I could leverage this model, but what is the tooling around reinforcement learning from human feedback? Like how could I use that framework or use tooling around that to like impact my own models or my own life? Like how could I connect my domain experts input and their preferences into a system that I'm designing? Do you have any thoughts there?

Yeah. So one of the examples I love to point to is actually Bloomberg did this and they probably did this early April now and they took the financial data that they had and Bloomberg from all the way back from like, you know, many of us know about it, Bloomberg from Bloomberg News, but that was actually the financial terminal that was used for stock trading. But they have these, I mean, mass amounts of financial data and how do they stack on top of it? Like how do they get and access that data even faster and kind of train it to the best use case that we have. Like currently our larger models can do that. They're not experts in financial data. They're not combing just financial data. But what Bloomberg did is they took and they retrained and they built the things I probably fangirled over. Sorry, if you're on the Bloomberg team and I fangirled over you at PyCon because I got great work. I was like, this is the coolest thing ever. I use this as an example also I like learn machine learning off of your repo. Okay, thanks, bye. But we do have a model if you want to learn and see reinforcement Learning in action. There is an open source repo. It is built by myself, Nikolai and Jimmy Whitaker, who we have as a data scientist in residence at Label Studio in Hartex, but also is at Pachyderm as well. But all of that is built, it's open, you can play around with it. It's based off of GPT2 right now, so go have some fun and get your hands dirty. And it's all runnable within a Google Colab notebook.

That's awesome. Yeah, so you mentioned it being run in a Google Colab notebook, which I think is awesome. And using also a bit of a smaller model to start with. And we've seen a lot of kind of directions towards smaller open models that are accessible to data scientists with like LLAMA and other things like that. How do you see that trajectory going and how will that impact sort of like day to day practitioners in terms of what they're able to do with this sort of technology?

I think the biggest thing I'm actually going to zoom out to answer this is the biggest thing we need to think about is context. Like what are you using model to solve or AI to solve or ML to solve? And the more that I've been diving into these conferences and the ecosystem, especially at a conference where it's a blended conference where you have folks that are not necessarily deep in the field or ML practitioner or they're like new to ML, it is so easy and there's a meme I always point to that. It's like, oh, it's you know, we're an AI backed so and so and it's like jk, we're just an AI, we're just, you know, calling the API and put a nice pretty shiny front end on it, which is no shade to anybody who is putting a front end on a GPT API. Like there is no shade at all to that. But it's like think about what you need a model for in the first place or what you want to use. Machine learning like that context is so important. I'm currently playing around with a Naked and afraid data set just to like play around. There's an open source data set out there that is a.

That's that's awesome. Like videos like.

No, it's context from the TV show of how many days they survived.

Oh, so literally. Yeah, like statistics about like, and features of the different survival situations.

Yeah, it's like, it's like country, their name, gender and then how many days they made it.

Yeah.

And like climate.

Yeah. Based on that. Like yeah that's so intriguing. I watch, I watch a lot. So confession. I watch also the alone show which is like another survival show.

This is like I'm a reality terrible junkie. I. That is how I do stress is reality tv.

But so I always wonder, I have this conversation with my wife around, like could I do it and maybe with a model trained off of. Off of your survival data set, I could like say I'm from here and this is my background. Could I survive?

I don't know. And I can't take credit for the original data set. It is someone who I have made friends with in my reality TV subreddit. So if you need to know where I spend my time. But he runs a SQL database. It is actually very good. He's very awesome updating it. It's available on Reddit. I can share it with you and you can post it in the links. But I'm just kind of playing around with the data set for fun. But in this context I'm playing around building demos and just having some fun, teaching myself some new skills. I don't need a large foundational model for that. And I think going back to your original question of well, all these models are getting smaller, more accessible. We can run it in a notebook. We don't need the high powered computer models everything single times. And if we stop and think about the context of the problem that we're trying to solve, it can give us a lot of answers and it can save us time, energy and computing power. And I think that's why I get really excited about being on the data labeling side again. I have a background in humanities, I'm a self taught programmer, but I think I don't want to be like we need more people like me in data science, but we need more of the humanities in data science because we're missing the context. Yeah.

We recently had a guest on the show that was talking about the intersection of art history and computer science and how computer scientists who are analyzing and doing computer vision could actually learn a lot from like what we know about like art and like how scenes are composed or how art has changed over time and how the features that they're actually engineering are connected to some of those things. So yeah, I think that there's a lot of different areas where this could apply and domain experts are so important. And I assume that with all of this like reinforcement from human reinforcement, learning from human feedback, I always mess it up.

I've been, it's okay. I've been doing the same thing. It's like R lhf I get it.

Yeah. Especially since you're from the Label Studio side. Do you have any. Could you give like a general picture or workflow for people of like, hey, I maybe want to take one of these models, GPT2 llama, MPT now, whatever the one is. But I also want to gather some domain expert feedback and eventually get to some type of like instruction or fine tuned model off of that. Like could you just give a general picture for like what that looks in today's world?

Yeah, and we'll try. This is like always. I feel like this is better when you have a whiteboard and a diagram and some arrows.

Oh, for sure. Yes. It's hard.

I'll do a quick walkthrough. So first off you'll create a sort of prompt. So typically these models work with a prompt and then you're given a large language model and then you start to train it. Usually what happens when you're training these models is you get a set of two outputs. And so in this case we can use what is opossums? Because we're opossum fans at Label Studio. I feel like that's natural. And you can be like an opossum is a marsupial creature or opossum is a great character for memes. Technically both of those are correct, but depending on context. And this is where that human signal side comes in. One answer is more correct than the other. So if we were training, let's say opossum memebot thing or a meme bot generator, let's go that direction, we'll have some fun with it. We would take the latter answer of this. A possum is a great animal to make memes. And that would be the better answer if we were going for what type of animal are we doing? Like maybe a biology assignment homework. Probably would pick the marsupial one. But this just gives insight of like the details that you give your annotation team can really directly influence the model. That's the labeling side. When we move this on, all of this is put through kind of this results from human feedback. Your answers are ranked. I did a binary situation. So just two options. But you can have a multitude of options that you put in. It is all weighted. It is then looped back around when you wish that we had the whiteboard to a reward or a preference model. And this reward or the preference model kind of tells you like, hey, I probably want to go for answers that look like this. Now computers don't speak memes or Marsupial or biology textbooks. But they do know patterns and trends, which is like what they pick up on. So based on that context clues that we give them, this preference model will start to preference those type of answers. Now it's really important that these reward preference models also hold in place kind of the original things that we had that it knows like how is language structured? Or other things from our original model that we enjoyed. Like we like, like language is always structured like this. Here's a proper noun. We like to capitalize the first letter of the sentences. Things that are important but like we kind of overthink sometimes when talking about generative language models. At least after that we want to make sure that we're not just gaming a system. Models are again, I don't think models are sentient. They're kind of just like mass numbers. They're just trying to game a system. They're playing like I always compare it to like you're trying to like it's Moneyball, essentially I'm baseball fan here. So it's Moneyball. You're you're statsing out the system and they in order so that they're just not giving you what you want to hear every time. You'll have to calculate an error rule in there. So put an error metric or an update rule and it basically says all right, we're going to almost like dunk you down a little bit so you're not too perfect and that'll prevent unwanted model drift. Then once you've done that a few times, you'll combine that with a copy of your original model that you had again you're kind of doing that checks and balances, making sure it doesn't run away. After that you will have a tuned language model and then rinse, wash, repeat until you've got that model right where you want it. Set it off to production and then talk to your friends that the other parts of your MLOps ecosystem.

Yes.

And it'll come in handy.

Awesome. Yeah. And I hope that we can link some of your slides from that talk in our show notes. They're awesome. Including emojis and the full deal which helps. So make sure and check out the show notes. The link to the slides will be in there so you can take a look at these figures while you're listening to the show. One follow up question on this, we're talking about gathering this feedback data people can think about. Okay. In the context of my company or where I'm working, I'm going to gather some of this data Tune a model. But what is your perspective on sort of like the open data ecosystem? And like what would you encourage people to think about in terms of like data that they could make openly available to help others who are also trying to do this or the other way around, like people that are searching for maybe a place to start. What is the sort of open data ecosystem look like right now and how important is that as this sort of field advances?

Yeah, first off, this is, you've got me on my other favorite soapbox of the moment. And this goes back to my days when I was, you know, a journalism student working in journalism. But open data is one of my like favorite topics to geek out on basically. It was something that really came actually as part of the Obama administration. He actually established federal funding for a lot of our public and civic data as a part of government accountability and transparency. So there was actual federal grants that went out to make a lot of our civic data public. So there's a really cool example. I believe it's the city of Philadelphia that actually built a SimCity like game off of their public data. It's so cool. It was like a grant given. Super fascinating. I'll link it to you and give it. It'll be in the plethora of show notes on all of that. But open data is just open, freely accessible, freely used data that is made available to the public. Love open data. I'm a participant in Open Data Week. But when it's, it's been federally funded, it's not always the best thing to be federally funded. And like we all know how government grants go and if you aren't aware how government grants go, they're very niche specific and they run out and they're not always maintained and it's not always the cool, sexy job that we have. So they're always not the best maintained or context applicant. What a lot of these early machine learning models did and what a lot of machine learning models do is these open data sets have given opportunities for people like myself to even learn how to do data science. I learned Python in Open Data Week. I remember going back and like let's get the traffic data in New York City. And it's like basic, you know, using curl and like getting things started for the first, like can you query an API? Like they're not the most organized data sets out there, they're not the most clean. Sometimes you get some really messy garbage data. You know, the 2020 census is actually a great example. I was speaking to someone yesterday at the conference about this, the 2020 census was the first time that we were able to do it digitally. Well, she gave the example of like, you know, hey, I started the census on my phone. Oh no, the pot boiled over. Oops, I accidentally counted myself twice in the census, or I didn't fill out my address, or now I've got two people, or a person who lives at this address, or a typo. Crap. Now that's a very messy data set. So open data can be a problem. Let's go to like the practical application of this. If you are working in open data or you are interested in getting more involved in open data. One of my favorite sources is like if you're publishing a story, making tutorials, making content, put your data out there and put how you processed it. And it's not just like one thing to put your data out there, but also how you processed it. In journalism you have this phrase, how you frame the story is how the story or is how you tell the story leaving out details, context, or even how you came across a source can influence how the story comes across.

Yeah, for sure.

And that's. We see it especially evident in data driven journalism and solutions journalism, which is interesting and like, it also can be really damaging to trust and reputation. But I think ML runs the same risk right now if we're not transparent of here's how I prepared the data set, here's how I trained an annotator, or here's the tools that I used, or here's how I obtain the data in the first place.

Yeah, and like you were saying the certain things like how you give instructions to a data annotator or, or how you set up your prompt that has such an influence on the downstream performance of these things. But it's very frequently I've definitely found the instructions you give data annotators is something very often left out of the story of how people tell what they did. It's like, oh, we gathered this data with these labels. Okay, Well, I can imagine my own set of instructions for getting those labels, but it's could result in a totally different thing that's happening. Like all sorts of biases and other things that go into that.

I mean. Well, I have a perfect case example of this. In January, we met many of the team members at Hartech's and LabelStudio met up. Basically, we got our entire customer success and sales team and the community side of things and a bunch of our support engineers to all sit together and we had a data labeling competition for fun at the end. And I Had just finished. Like, how did you get started with data labeling and best practices? And I was like, easy, I'm gonna kick all of your butts. I was totally going in hot shit and everything and thinking, well I sped through. I was like, whatever, next. Great, done. And I sped through because I was like, speed was a metric, but also accuracy. Well, I sped through this thing because I was like, whatever, I'm gonna ace this. I know the keyboard shortcuts, my systems are set up. I had the lowest accuracy score. Everybody. My data was all. I was like, you failed, Aaron. It was like, I was like, man, I'm gonna go embarrass myself right now after all that crap I just talked.

Yeah, yeah, I think that's the other thing. Like I don't know if you have any encouragement here, but like data scientists out there who have not actively participated in the data labeling process, I think, yeah, that's such a learning experience because it gives you perspective. Even if in the future you're not part of one of those processes. It gives you good questions to ask if like, oh, someone gives you this data set that was labeled, you should probably ask a few follow up questions about like how did that go? What did you do there?

Well, in academic research you actually have to disclose things like did you pay your annotators or how did you prepare the annotators when you're doing research? Because that can put so much of a bias on a model that is built off of that data. And that academic like you can't get peer reviewed studies done in without disclosing that information. It's part of data ethics now. And one of the biggest things, and we don't talk about it enough is how do you pay your annotators or do you outsource your annotators? Which isn't saying that's a bad thing to do. But again we remember that so many of these models, and I see it like, I think a lot of it times it actually is probably, I'm going to guess here, I don't know, but I'd be even wondering if these smaller models that are generated because they're generated at home or people dorking around on their computer, they might even have more bias because we're not trading an annotator. Like I know when I'm goofing around with my naked and afraid data set, I'm not annotating, I'm playing some 30 second goofing around stuff and watching YouTube videos, just seeing what's out there. I'm not doing the work, which is a problem.

Yeah, I guess kind of bringing things full circle a little bit. Like we started talking about some of these players and ML ops and sort of the ops around this process. We talked about human feedback, reinforcement, learning. We talked about open data. What excites you about like the trends that we're seeing and what impact they could have on our industry moving forward. Maybe that's related to like people that weren't able to like participate in this process before. The tooling's better so they can. Or maybe it's something totally different like around like tasks or other things that you see like in the future. Like what, what are you personally excited about like looking forward as you bring this stuff together.

Well, first off, I've been really impressed with what the Hugging Face team is doing. I noticed the hugging face shirt. The hugging face spaces have been amazing. We do have a label studio Hugging Face space but the ability to get up and going in the browser has been super awesome. And there is a talk I went to at PI Data Berlin that's running Streamlights. They're running entirely Python based models right in the browser. And tools I think there's a. It's binders I believe is another tool that's doing another again very similar to notebook processing all in the browser makes it more accessible than ever before and it's just really exciting especially as like I love that we have more people interested in this industry, but it's also not only the interest but the tools to do it correctly and ethically. And again jumping on my soapbox here. This is why the open data is so important. So putting when we can putting our sources, our references, building in the public, building in open source and making kind of a almost I don't want to say paper trail but like a show your work sort of process is really important for the future.

Yeah, awesome, that's great. And as we kind of close out here, where can people find you online and also tell us a little bit about your own podcast which sounds awesome and like includes pickles.

Yeah. So I am available online at Erin McHale on all the platforms or Aaron Bio has a link to everything that I'm at. You can also chase me down at LabelStudio. So it's Label Studio but the last IO is like of things. Join the community, come hang out with me there. We have an upcoming town hall and getting into more workshops. So very excited about that. I also run the Dev Relish podcast so it's everything about Devrel and ish also you know, naturally Some people made sourdough bread. I got into fermentation. We got a fun pickle fact and cool pickle logos because you got to relish the developer moments.

Well, I open source. Yeah, this was definitely not a sour experience. I've relished it very much. Thank you so much for joining, Aaron. It's been a great pleasure to talk to you and looking forward to following up with all the cool community stuff you got going on again, people. Check out the show notes and thank you so much.

Thank you so much. This was a quite a big deal that we had going on here.

Good one, good one.

Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already, and if you're a longtime listener of the show. Help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to@fastly.com and fly IO. And to our Beat Freakin residents Breakmaster Cylinder for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.