Data synthesis for SOTA LLMs

Foreign. If you work in artificial intelligence, aspire to or are curious how AI related tech is changing the world, this is the show for you. We just dropped Dance Party, our third full length album on Changelog Beats. Buy it on Bandcamp and itunes or stream it on Spotify, Apple Music and the rest. Link in the show notes. Thank you to our partner partners@FlyIO launch your app close to your users. Find out how@fly IO.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the CEO and founder at Prediction Guard and I'm joined as always by my co host Chris Benson, who is a tech strategist at Lockheed Martin. How you doing Chris?

Doing great today. It was nice seeing you a few days ago in person and in the flesh. In the flesh, yeah, that was great. I think you posted a picture on LinkedIn. So if anybody doesn't know what we look like and has some crazy reason to want to know, there's a smiling mug of us on Daniel's profile.

So yes, yes. And the reason we met is I was on a client visit on site and we were prototyping out some stuff like chat over your docs and natural language to SQL stuff and all sorts of things with Prediction Guard. And one of the models that we were using was from Noose Research and that works out great because we have Kuran Malhotra here who is from Noos Research, co founder and researcher there. So welcome. Glad to have you. Kuran.

Hey all, thanks for having me. I'm extremely excited to chat with you guys.

Yeah, like I said, I'm a huge. Well this is our first time meeting but I feel like we're already friends because I've had so much of my own benefit and interaction in working with models from Noose Research. A lot of amazing models that you've posted on Hugging Face and research that you're doing. I'm wondering if you could just give us a little bit of a background about Noose specifically and kind of how you came together as researchers and started to me from the sidelines it seemed like, oh, all of a sudden there's these amazing models on Hugging Face and I don't know who these people are, these new research people, but they're amazing. So give us a little bit of the backstory there.

Absolutely, yeah. So just as a general overview, we are one part like open source research organization, we put these models out for free. We put a lot of research out for free, some data sets so people can build on top of these open models. On the other Hand, we're very recently a company as well a C corp, so we've been working pretty hard after getting some seed funding on building together some, some exciting stuff. I won't go too into on during the overview point, but we're continuing to do our open source research and development and release of models indefinitely. The way we started is very interesting and it would be pretty out of nowhere to the outside for sure. It was, it was extremely fast for us. We are a collective of people who have been playing around in the open source language model space for a while, ranging from like GPT2 release to llama release to like the first Transformers paper. We've got people from various eras of gen AI of when they came in. And for myself it was GPT2. I stumbled upon a collab notebook and started fine tuning, made some Edgar Allan Poe and Lovecraft tunes.

I've done the same. That's awesome.

And we just got pulled into this world of look at these next token predictors that are just managing to smatter together the most wonderful and amazing stories that slowly turn into a deeper and deeper dive of, well, how can I use this for learning information? How can I learn to use this for production and automation? Right? It's evolved over time for us. We started off just working with different open source collectives. Actually once OpenAI kind of released GPT3 and had closed sourced it, you know, we were used to open source GPT2. We were like, oh man, what are we going to do? Like, how are we going to continue to play with the level of customization and interactivity that we had with GPT2? Then Eleuther had released GPT J6B. The Cobalt AI community, this community of people who tune models and inference models started to pop up I think around 2020, 2021 in the face of this. So a lot of us started to have places to centralize and play with these models. We got to contribute and learn how to become better open source AI developers, et cetera. Eventually there was a need for more concrete organizations to do this kind of focused work on the creation of these models. We were stuck with like okay architectures for a while, like Pythia, but thanks to Meta, you know, we wouldn't be here without Meta. I'll say that first and foremost, like the great Llama. Yeah, yeah, like prior to Llama, right? Like everyone's like, oh, Facebook evil. Like my data, et cetera. And here we are like they are kind of like the shepherds of this new era of the open source AI movement. So when Llama came out, there was a paper that came out called Alpaca by Stanford Lab, right? And this was about distilling data from bigger models like GPT3, ChatGPT, GPT4, and being able to train social media smaller models on that distilled synthetic data, something they call the instruction data. So that alpaca format really opened up the playing field for everybody to start making these instruct style models. These actual four produce style models. So there was an idea I had in my head of, well, the alpaca guys are using only GPT 3.5 outputs. What if I only generated GPT 4 outputs? It'll be a little expensive, but you'll probably get a better model out of it than alpaca. At the same time that I was looking at this, there was a guy on Twitter named Technium who had just started putting together his own synthetic data set based off alpaca and the GPT4 only as well. So I was working with a group at the time called Open Assistant under Laion. They're really big nonprofit. And while I was working on that, we had some GPUs they were cool with us using towards the development of new models. So I reached out to Technium and I said, hey, I have a little bit of compute. You have GPT4 data in the same format. I have GPT4 data in the same format. Let's train a model. So we trained a model called GPT4X Vicuna. This model was on the Vicuna fine tune. We fine tuned to fine tune. Basically, the Vicuna model was a alpaca style fine tune. And we tried our data set on top of it. It was good, it was okay. But then we thought, you know, we'll probably get a better result if we just train on the base Llama model. And the resulting model was the very first Hermes model. Gotcha.

The og.

The og. And that's kind of how it started to come together was we both had a data thesis on use GPT4 only and follow Alpaca. And we trained on Llama and we got Hermes. And we didn't know what benchmarks were. We didn't know anything about any of this stuff. We just made a model and it got a ton of attention. We put it out under this name, Noose Research. Noose comes from the Greek word for intellect. We thought it'd be a good name for an AI company, but it was, it was just a place for, you know, fun projects and fine tunes and stuff. It was just a name. We were using for our collaboration. And people started swarming and asking, you know, what's Noose Research? Like what's this sudden like mystical like open source organization that like put out this like best model and we're like, yeah, best model. Like we, you know, we just tried something. It was, it was really organic. And it got to the point that people started telling us, you know, you must have trained on the benchmarks, like these are doing too well. And we were like, what's benchmarks? We, we're not really like coming from an academic place as much as from like an enthusiast that became so committed that it became our life. Right. It became our day to day.

Yeah.

So from there people started to ask us, can I join Noose Research now? There wasn't a news research to join. It's just two guys. Right. What ended up happening was we formed a private Discord server and we thought there's a lot of people who range from somebody who's like 16, 17 years old, savant on Twitter, hasn't even been to college yet, insane at transformer stuff, to mid-30s, you know, working a really, really good fang esque job and just wants to really create and let loose. That was another class of volunteer. And then you have, you know, older gentlemen who has already exited a company or something who has just been playing with code for a while and wants to jump in and hang out. So we ended up being this really eclectic group. You know, we don't know what your name is, we don't know what your race is, we don't know your gender or anything. It's just Discord profile picture, Twitter profile picture. Right. So we came together, grew to about like 40 people, all working together on various different projects like Hermes Tunes, Data Synthesis, the Capybara series, context length extension, etc. And just from this kind of interaction between Twitter and Discord and bringing people in that we thought were cool, we ended up becoming what people will call open source research. Org.

Yeah, you sort of stumbled into creating this amazing research research organization which is ruling the world, which is awesome.

It's what OpenAI might have been.

Oh well, yeah, that's really sweet.

Thank you guys.

Yeah. And I love it. It's so cool to hear that story in that background. And I see like in my own sort of little snapshots here and there, like I'm connecting that in my mind over the past couple years as I've, as I've seen you all post different models and that sort of thing. This is something we've definitely touched on. On the show before, but some of our listeners might not kind of fully grasp when you say this sort of like synthetic data sets that you were focused on in this alpaca format. Could you kind of explain a little bit? Like, we've talked a lot about fine tuning and, you know, preference tuning and RLHF and different things, but what does it specifically mean that like, you would take synthetic data, what does that mean in your case? And like, why, why does that result in something good in fine tuning an open model? People might think, oh, this is synthetic data, why should I expect it to like, be any good? So could you kind of help explain that subject a little bit?

Yeah, absolutely. So, I mean, out of context, synthetic is like as meaningless as like artificial, right?

It could.

Data is data, but in this case it's referring to a class of data that's been generated by another language model or another AI, another diffusion model, et cetera, that can actually be used to further train models. Now you might say, why would you want to do something like that? How is it helpful? What was important to us is we were all GPU poor, right? We were all running on laptops or maybe a 3090, maybe a 4090. Like, as individuals, we, we don't have data centers. So training or even tuning like a large model in the early days, like 70 billion parameters, something like that was just unfeasible for us. And knowing that GPT3 is like something like 175 billion parameters and 3.5 and 4 can only go up from there, the question became, how can we make these small 7 billion parameter models even compete with these massive, massive ones, These ones that I want to run offline, these ones that I might want to run an edge device on a phone, on a drone, etc. Right? Like, how can I make them even useful? So there's two things to talk about here. One is synthetic data and the other is distillation, right? So synthetic data is just referring to like any kind of data that's created by a model in this case. And the reason that's useful is in particular distillation. So if I told you to go study comp SCI for 10 years, for example, and put in that massive time investment and really focus on general programming. And then I told you, you know, now it's time for you to learn about AI and transformers and stuff, and put you through all the math prerequisites, et cetera, like, you're going to come out with like a really strong foundation of how to do the work but the problem is you've put in a massive time investment. Now if I take that guy who spent 10 years doing engineering, then another 5 years doing AI and I ask him, hey, can you teach somebody like just really important, like compressed tidbits that'll help them just get up and running to do the work? That's data distillation, right? That's knowledge distillation. So you look at these big models like Claude or 70B model or GPT4 and you can see like they're amazing. They're brilliant at everything. They have a bunch of high quality data they're trained on and they have a bunch of low quality data they're trained on that they can interact with and express in a high quality form. So instead of me having to read a massive 10 pager for why some chemical reaction or some like tax based process, whatever you want it to be like, instead of reading a massive document on that and then feeding that to a language model, we can just have that really smart model that already understands it really well, compress that information into an instruction or into a conversation, into like two sentences, three sentences, five sentences, like half a page. And we can just train a much smaller model on that compressed information and it will learn the compressed information, you know, to the degree that a language model learns something, you know, not perfectly, but because of that. What the alpaca guys did was they generated a bunch of seed tasks from GPT 3.5 on various different domains and topics and created these kind of compressed instructions with instruction. An input question from the user and then an answer. So the instruction could be like, given the following math equation, explain step by step why this is the answer, and then the input is the equation which is your question, and then the output is the compressed answer. So all of that we can take as one sample in the data set and we can make hundreds of thousands or millions of samples like that of various different domains and various different tasks. So the alpaca guys did this, less than 100k examples I believe. And they trained the llama models on these and they found massive boosts to performance that this distilled information like a human successfully compresses and transfers over. So when I saw that, and then independently when Technium saw that, and then independently when many others saw that, we were like, this is so intuitive. This is exactly how I've learned anything, by just going on discord and Twitter and bothering people to give me the compressed bit of how I do something. We should try doing this with even higher quality models than 3.5. So we created, I can't remember the exact number at the moment, but at least 50,000, maybe a hundred thousand examples originally for Hermes, one like this, just using GPT4. And then we trained on that and ended up getting performance that was extremely, extremely like massive boost compared to the other models that were not trained using this kind of method. So without these giants that have already established themselves in the space, we wouldn't be here. Without OpenAI, without Meta, we literally wouldn't have the model and the data to do the kind of work that we did to make Hermes. What it allowed for us is for local models to finally be comprehensible and for us to finally have offline capabilities to take the good stuff from something like GPT4 or something else and make it uncensored. So it still has all this understanding of all these topics, but it doesn't have all that RLHF inside it necessarily. That safety izes it so that when people utilize, the model has all this intelligence, but it has more freedom of thought to kind of converse with you on topics that OpenAI may reject.

Gotcha. One of the things I was curious about as you were going through that was a few episodes back. Daniel and I were kind of talking about the effect of model licensing, you know, on the community and the different kind of licensing concerns that were coming out from, whether it be, you know, meta, OpenAI, you name the organization. Is that ever a challenge for you since you're kind of using those to get started in terms of the inputs, has that been a concern or do you anticipate it being a concern?

I think that of course, like, generally, like US international regulation on this stuff is evolving, the conversation is evolving very much so naturally there's like, you have to keep it top of mind. You have to think about these kind of things. But thankfully, because all of our model releases are like open source and we don't profit from them, like if somebody goes off and creates a product using our model, you know, good for them. But we don't necessarily take on that liability or that worry of saying, hey, like we're going to sell you this model that was created with GPT4 outputs, we. We actually actively try to stay away from doing that. But because the data distillation paradigm is so effective, you know, if a model comes out that's better than GPT4 and it's open source and I can use it locally and in their ToS, it says, you know, you can use this to make a commercial model, then we can Apply the same techniques that we've been preparing and researching and understanding from these closed models and use it there. So right now, like we don't stand to or try to or have any plans to profit from using any of these outputs. We're not about that because we want to be careful and respectful of these model creators. But that, and these companies. But that being said, we're learning all these techniques and developing all these techniques that will be useful for when that time comes and for when that's available. Especially with the advent of something like Mistral. If we do distillation from a Mistral model like Mistral medium or something like that, that's completely from my understanding, you know, barring their TOS saying otherwise, but I believe it doesn't. It's completely okay in that situation for us to create models like this that can be used commercially, etc. Regarding the TOS stuff though, like, as much as we err on the side of caution, I'd find it hard to see a company enforce their ToS when these larger models are likely trained on not all copyright free stuff. Like I find it hard pressed to believe that these closed source companies, their models are, you know, totally copyright free and totally copyright clean. So if some other company that was feeling a little more rambunctious than ourselves was to say, you know, we are going to commercially release on this, I imagine it'd be difficult for them to become after without the other group opening their books. And there's actually a pretty interesting interaction that happened regarding this between Google and OpenAI if you guys are familiar.

So yeah, I saw this interesting picture the other day. It was like the interesting web of AI and it was like how Microsoft, Google OpenAI. It's like on one side there's the ones and it shows how they're connected to the other ones is like this visualization and how many of them overlap in these strange ways between like whether it's together or Mistral or Meta, Google Microsoft OpenAI is sort of very interesting web of connections that probably makes some of these things rather difficult.

Leave it for the lawyers to sort out.

Yeah, yeah, that's the thing is like we can look at an example, right? Like you hear that phrase like good artists copy, great artists steal, right? Like so the data distillers we're copying, right? Like we're just distilling this information like we're trying to like make our models more like those and we don't really plan to commercialize, we're just doing it for free for everyone. But the great artists are, you know, Google, you know, like, you look at Bard and it tells you, you know, I was made by OpenAI. Now it's fine for our open source model to say I was made by OpenAI, because we're very transparent about this, is trained on GPT outputs. But when Bard violates the ToS with a paid product, Bold, Yeah, that says like, I was trained by OpenAI. Right. You'd think that OpenAI would come after this multi billion dollar company like immediately, right? Instead you see a tweet from. First you see Google deny it. Then you see a tweet from Sam Altman, which was something along the lines of, I'm paraphrasing. Or something along the lines of, I'm not mad that they trained on our outputs. I'm mad that they lied about it. And I'm sitting there like, okay, you're mad about this, but like, don't you. Aren't you gonna pursue the legal action in your terms of services? No, no, because everyone would have to open their books up too. That being said, I don't condone the commercial use of that kind of stuff, like release, like making a paid model from GPT4 outputs. Like, I wouldn't advise anyone sell a model made with them just because, like, you know, we want to respect people's like, TOS and stuff. They worked hard and spent billions to make this stuff, or hundreds of millions, however much they spent. But there is certainly room for hypocrisy in the, in that realm of the large corpse. That's my thoughts on the licensing stuff, and that's definitely my own individual thoughts. Like, we're a pretty decentralized collective at Noose, so you'll find people with all sorts of opinions all over the place. And as a company, we don't hold any view whatsoever on that.

Yeah, I'm wondering maybe this gets a little bit to the distributed nature of this, but I know that there's sort of various collections of what the Noose research group has done over time. You mentioned Hermes, but then there's there's these other kind of categories of things too, like the yarn models, capybara, puffin, obsidian. Just looking over the hugging face now, I'm wondering if you could just give us, like, from your perspective, a little bit of a map of these different things and like, how people might categorize the different collections of what news has done. I definitely want to talk about, like, the future things and ongoing things as well. But as it stands now, what are the kind of major Categories of what the collective has invested in over their time. In over time.

Certainly. Certainly. So within the stuff that's viewable on Hugging Face, at least we've got the Hermes series, of which, like I told you guys, the initial story of how it went down, but from there Technium kept going. I haven't personally had any interaction with the Hermes model since the initial. From there, Tech just continued to create more and more synthetic data, collect from more and more sources, use more and more open data sets. And he's just got the, I guess, award winning, like data thesis. The guy really knows how to go about curating and synthesizing good data. So technium, it's his baby, the Hermes project. So everything you've seen since is really his work and anyone who has kind of collaborated with him. But almost like you can't call it anything a solo project because of the open data sets we use too. Like everything is built on the shoulders of giants and the shoulders of each other as little people. But tech really has helmed the Hermes initiative so far. I think that's our most popular model series. And he released the open Hermes as well. Because we had some data in the original Hermes that we never released publicly and we wanted to make that kind of an option for everybody. So that's. Hermes still follows the same kind of philosophy of synthetic data and it now uses the chat ML format instead of the alpaca format is what we kind of upgraded to. Then you've got Capybara and Puffin, which are both done by a volunteer and you know, OG member LDJ, who you may be familiar with. Luigi Danielle Jr. So the capybara series was using an amplify instruct method, this novel method that LDJ had worked on alongside another one of our researchers, J. So LDJ and J can get confusing, but the two of them worked on the Capybara series, created the data set, trained the models, and then Puffin was the idea of using hand picked smaller samples from some of our larger data sets to make sleek data sets for an easy tune and see how that works kind of in the spirit of the Lima paper where they just used a few examples to get really good results. Those are really the popular tunes using synthetic data for like general use. Yarn is this novel context length extension method at the time of creation by Imozilla, also known as Jeffrey Canal and Bowen Peng, also known as Block 97, alongside Enrico Chipotle and Eleuther AI. So what happened there was these guys were already looking into contests like the extension for a While and when we kind of came under the noose banner to do the work, it opened up a little bit of resources from COMPUTE sponsorships, it opened up a more centralized place for them to be able to do that collaboration. I had no hand in, in the yarn models whatsoever. And that's the exciting thing is everyone really gets to work in their own spheres and their own kind of autonomous circles. And then we just check in and see, you know, how's the research going, how's it coming along? Because we really work with people that we heavily believe in and we believe in their idea. So if we don't already have an idea, we kind of just say, you know, please freely create because we brought you in. Because what we will freely create will push forth our agenda anyway. So I think those are our biggest model releases and series that we have available. Outside of that, we have a bunch of stuff on our GitHub as well. Stuff that's being worked on, stuff that hasn't necessarily come out yet. There's a lot of that.

So I got a question for you as a follow up. It's pretty fascinating, the story that you've been telling us here because of that kind of organic creation of the organization or collective. And I'm wondering, as you've done that and you kind of went through and talked about the different model groups and kind of talked about the owners or spiritual owners, if you will, of each of those families, how do the different members of the collective interact to kind of share? Like how do you each push each other along or share information or give ideas so that cross family efforts can kind of benefit from the overall collective? And as you said, now a C corp and you guys are more organized at this point. So what kind of culture has developed around those communications and learnings?

Yeah, absolutely. I mean, when it started, it was just like a small discord, maybe like 10 people from there. Like we kind of created more channels as people wanted to work on more things. And we had initially split up into like three, four different topics or sectors that people could assign themselves to. One being data synthesis, of course. So we can kind of find new novel methods and formats for distillation and the creation of synthetic data. One being training, like people who are just like really good at training hyperparam stuff and people who will come up with new architectures and new techniques. Another being agents, a group of people who want to actually try to build tools and do autonomous work with this stuff. And then we had this one category that it was a prediction for the future of Simulation. So we had people that were very interested in kind of bringing this stuff into simulation, into Unity, into kind of seeing how all these things came together. And it was interesting because the training built on the data synthesis, the agents build on the training, and then the sim would build on the agents was kind of the idea. So everybody needed to work together because all those things are so intrinsically connected. But people would have specializations on kind of where in that workflow they wanted to work. We didn't end up doing a lot on the sim side of things. Now recently there's a lot more interest because we have a lot more, you know, capability generally as the AI community does, you know, but as we've grown to, we went to 40 people, it was fine. Now we've gone to like 5,000 people in the discord. It's. It's a little unwieldy there. So what we do is we kind of tier people in. You come into the discord, you can see maybe two channels, and then we'll give people a developer role. We don't really let people select their own roles because we want to make sure we can kind of sort through people we know to kind of let them through. And even as we do open source research, a lot of it is unreleased and we want to make sure that it's kind of protected before release. So we create this developer role so people can then see like way more channels of just general development and development conversation. And from there as we see, you know, contributors who have started to do more work or show more passion towards contributing to news in a particular field, or who have some reputation or some portfolio in a particular field, then we'll assign them one of those roles and that will open up the family of channels relating to those roles and our current projects surrounding that role. So like data synthesis projects, agent projects, training projects, etc. So we kind of just tier it out so people can interact. And people who have been around for a while, or people we consider fellows or part of the cohort, they can usually see pretty much everything. So they're pretty effective in serving as coordinators for the cross communication between these different channels and groups. And even if something has like a particular someone has a particular role, or some channel has a particular role, it's supposed to be a part of like, it's still discord and we're still very chill. So like people will still work on like various different overlaps inside of just one channel as well. If you're listening, you know that artificial intelligence is revolutionizing the way we produce information, changing society, culture, politics, the economy. But it's also created a world of AI generated content, including deepfakes. So how can we tell what's real online? Read Write Own Building the Next Era of the Internet, a new book from entrepreneur and investor Chris Dixon, explores one possible solution to the Internet's authenticity problem. Blockchains from AI that tracks its source material to generative programs that compensate rather than cannibalize creators, Read Write Own is a call to action for a more open, transparent and democratic Internet. One that opens the black box of AI, tracks the origins we see online, and much more. This is our chance to reimagine world changing technologies to build the Internet we want, not the one we inherited. Order your copy of Read Write Own today or go to readwriteown.com to learn more.

I have a selfish question which now that this is one of the advantages of doing the podcast, I get to talk to all the amazing people doing amazing things and learn from them. But I'm wondering as a person who is also trying to fine tune some models, either just for my own enjoyment and learning, but also fine tuning models for specific tasks and in specific customer use cases and that sort of thing. There's a lot of people out there, I think many of our listeners who are thinking like, since you being part of this collective have worked for, you know, since the sort of dawn of these many, you know, the proliferation of fine tunes from Llama and et cetera. And as you've seen all that, as you're doing more and more fine tunes now, as you're looking towards the future, do you have any kind of good advice or things to keep in mind for all those fine tuners out there that are thinking about grabbing something off of Hugging Face, creating their own versions of these models. Maybe they have their own ideas about a specific take on a model. Any general tips that you found to be really useful over time or pitfalls that you'd like to highlight?

Yeah, I mean I can, I can try to think of a few off the top of head I'll say that hyper parameters are really important and it's important to try to get that right. It's going to vary from model to model, but a lot of the time some people think hyperparams like don't really matter as much to like obsess over and some people think it's like a secret sauce as well. So I'd say like try to do a lot of research into like good hyper params, a good learning rate. Like I'D also say, like, I could be totally wrong about this as I am not the trainer of Hermes today or a lot of these models. But something I personally believe in a lot is like ignore like people telling you to only train for like X amount of time. Like if you're not overfitting, like just keep going. Like if you can, if you have the compute, like keep training and keep going. Like train for more tokens, more epochs. Like that's something I heavily believe in. In terms of trainers to use. There's a lot of people who make their own scripts for specialty stuff and there's of course like, you know, you can just use hugging face. But the library we use is called Axolotl A X O L O T L like the animal by Cassius Wing Leon of the Open Access Collective. We think Axolotl is probably the best general purpose trainer for Lora qloras, fine tunes, etc. It like any open source repository is buggy and stuff you're gonna have to work out, but it's in my opinion probably the easiest and most effective trainer to use for like pretty much any model architecture available right now. So I definitely point everybody towards Axolotl.

Awesome. Yeah, that's super useful. We'll share some links in in our show notes as well, so people make sure and check that stuff out. Another kind of interesting question as you see, you know, I think we saw these waves of models that came out maybe around synthetic data, fine tunes or other types of fine tunes. I see this like interesting sort of thing happening over the past however many months, you know, not that long in the scheme of things, but in the AI world maybe a while where we're kind of now like there's a lot of interesting approaches, more so than just fine tunes, but like mixture of experts and merging and of course multimodal stuff coming out now I see news kind of dabbling in that you don't have to answer for the whole collective. But as there's so many of these things coming out in different approaches, what are some of the things within that doesn't have to be one of those, but what are some of the things on. On your mind kind of moving forward or on Noos's mind kind of more generally.

Sure, I'll try to go from like simple to complex on the kind of.

Stuff that sounds great.

I think that definitely just like straight up instruction tuning is great. There's other ways to tune, like the EVOL instruct method. I would advise people to try to create new instruction methodologies that allow us to make even better formatted data. People don't spend enough time trying to create new instruct formats and we've definitely been swamped with not doing that as well. So I think towards the general community it's a really easy place to get started. You don't need to really know how to code so much as think about how a human might more effectively phrase something or format something and kind of remix from there. I think that's like probably the easiest place to start. Then there's a model merging, right? Model merging is great. You can just like take two models and frankenstein them together to question mark results. You know, you got to just try and see what happens and feel it out. Then from there I would say there's stuff like dpo, there's RLHF, dpo, these kind of rewards things that can let you enable rejections or create censorship or put some kind of general concept or attitude towards the model. We found that to be pretty effective. With the latest news, Hermes, Mixtral, dpo, it seems like people really like it and prefer it over just the sft. So that's another thing that I'd heavily recommend. From there we get a little more complex. We have some reward model stuff we're working on that I won't speak to just yet outside of saying we're working on it. That we think is going to be like pretty big for reasoning boosts. Of course there's techniques like chain of thought and tree of thought for like multi step prompting, creating data sets. Even out of that for any of these purposes I've already mentioned is going to be really effective. Now to stuff that maybe not everybody can. Actually a lot of you would already be able to do this. There's like something that we like to call over at Noose activations hacking, where you're kind of messing with the way that a model. I'm trying to think about how to say this in like the most layman's terms, like you're trying to mess with how a model like generally vibes about something. So rather than just doing a system prompt or something like that, you can actually like change the the model vectors to kind of be like more political about something, less political about something more terse, more specific. And it has far more effect and control over a model than a system prompt. It's basically like a system prompt that like tells it to embody certain characteristics, but it's not something you can really jailbreak or get around as far as my testing has shown. Certainly not as easily As a system prompt, like, we have no problem jailbreaking even the most censored closed models today. Like it can be done by anybody with the right words, right? But this activation stuff, it really creates a bit more of a robustness and fidelity to the concepts that you're trying to tell it to embody. There's a few more I'm trying to think of that would be useful for people. One thing is soft prompting. It's not really around anymore. It used to be pretty big during the gptj, like pre lama days and the Cobalt AI guys really pioneered the use of it in the open source community. But a soft prompt basically takes like massive prompt and compresses it down to like way less tokens. So you can give your model like a huge prompt, a huge system prompt, or huge amount of information and use like way less tokens. So soft prompting is cool. It's not going to be too difficult to like update it for like Llama Mistral, like today's architectures. It's just like nobody has really done it that I've seen. So, you know, to the community, if you guys do that, please share. That's actually much easier than the activation stuff, I think. And then finally, probably the hardest unsolved is like sampling methods. Like today we use like Top K, Top P, like, you know, nucleus sampling, etc. Whatever. Like there's better ways to pick tokens for sure. There's better ways to judge the value of tokens for sure. And everyone has been too kind of concerned with higher levels to get that low and do whatever the magic math is that I can't do that would, you know, enable some steering and some even beyond steering, like alternative sampling paradigms. And I think that would probably bring the biggest change and transformation to literally all models, regardless of the tune, regardless of the architecture, etc. If you get pulled off. So really looking forward to something like that happening in the space.

That was a lot of really good advice that you have there. I was sitting there trying to take notes while you were talking through it and everything going, wait. But he said that too. And he said that too. The really good answer there. Thank you for that. As we were starting to wind up here, wanted to ask you. I know about. As we're recording this, it looks like it was just over three weeks ago. About four weeks ago, when we released this episode, you guys announced your 5.2 million dollar seed financing round. So congratulations on that. That was pretty amazing.

Thank you.

And I'm kind of wondering, so like you've kind of started with this kind of fairy tale story of kind of organically building from the ground up. You know yourself, you connect with somebody else, a few other people join you get to thousands of people contributing. You find and really producing amazing work. And then you're incorporating and now you got the seed round coming. Where does that lead you? It's kind of a sky's the limit kind of scenario. It seems, you know, that now that you're, you're kind of launching and you know, on that, you know, as a corporation, as you said, where can you go from here? What do you anticipate over the next couple of years or even several years out? You know, what's the vision? What do you want to achieve? You've come a long way so far. What's next?

AGI? No, I'm just kidding.

I believe you if you said it actually.

I mean like, you know, someone will do it, but.

And then you'll distill the knowledge, then.

We'Ll distill and then you'll, you'll run the AGI on your, on your neuralink, on your contact lens or something. But for us like, there's a huge focus on locality, there's a huge focus on offline, there's a huge focus on take the power back, run the model yourself, do everything at home. Like that's big for us. And at the same time, of course we believe in scale, but there's this idea that, you know, there's so much unsolved at the small model size. Why don't we do that before we go to a trillion params? Because we can scale those realizations. But for us like there's certainly you know, a transformation and change in attitude and pressures from going from pure open source volunteer to as well having kind of this more corporate branch get created as well. But that being said, it's been pretty consistent our ethos and our motivation for why we do this. And like you said, it really was organic in the sense that like we're a product of the times, we're a product of the atmosphere of the AI community. Like people have said nice things like you guys are setting the trend and it's not really true so much as the truth is like we are one of many embodiments of the sentiment that the community has and that the world has. We think, think like there's more than one noose research in this world. You know, there's Alignment Labs, there's Pygmalion, there's Cobalt, there's people who have been around before us, people who will come along the way. People have already formed since we have, and there's lots of people who have kind of embodied the noose research ethos. And it's not really just our ethos as much as the overall community's ethos. There are people who have come before us, people who will come along the way who do very, very similar style of work as us, this kind of open work. And I think that's got everything to do with the fact that like this is what the people want. We're just the everyman, just like everybody else. We're not like billionaires or super like all X Facebook or anything like that. We're. We're just a bunch of people who really, really care about this, who, who want to see everyone have access to language models, everyone be able to automate their lives, everyone be able to push their understanding of any topic to the next level. And our work, as we become an organization that's looking to, you know, be a company and create revenue, etc. We won't let it tamper or hinder any of the open source work we do. In fact, we want it to empower all of that work because we believe that the tools and the developments and services that we will be providing as a corporation will only serve to better feed the entire open source community. We're not really looking to suddenly make like a closed Hermes or something like that. We're more looking to create tools and do research that makes your open Hermes far more effective, far better and, you know, good enough that you may want to pay for that tool.

It sounds like something I would pay for, that's for sure. Yeah, it's super inspiring. I really appreciate you taking time, Karin, to talk with us. I thoroughly enjoyed this because I am such a fan of everything you all are doing and the community that you've built. So thank you for staying true to that culture and what you're doing and I'm really looking forward to seeing what happens in the future and where things head. And I hope that we can talk again and have Noos back on the show and in a year when of course, everything will be different in the AI world and I'm sure you'll still be doing interesting things. So, yeah, you're always welcome back on the show.

Thank you so much. It's been a pleasure to chat with you guys. Thanks for being so candid. I'm glad we were able to kind of push our message forth more and thanks for the validation you and the community have given us to keep doing this. Great work.

All right, thanks. We'll talk soon.

See ya. That is practically I for this week. Thanks for listening. Subscribe now if you haven't yet, head to PracticalAI FM for all the ways. And don't forget to check out our fresh Changelog beats. The Dance Party album is on Spotify, Apple Music, and the rest. There's a link in the show notes for you. Thanks once again to our partners at Fly IO, to our Beat Freaking Residents Brake Master Cylinder, and to you for listening. That's all for now. We'll talk to you again next time.