Industry Roundup #6: GPT-5 Launch & Scaling Limits, Meta’s Chatbot Guidelines Leak, and AI Safety Concerns

Generative AI is transforming industries at an unprecedented pace. But as AI changes how you or your team work, one thing is clear. Your skills also need to evolve. At datacamp, we offer everything you or.

Your team need to adapt and thrive with AI. Whether it's business users looking to get.

The Most out of ChatGPT and Copilot.

Or developers and data scientists looking to fine tune models.

You can learn the entire AI skills spectrum on Datacamp. Power your AI transformation today. Start learning@datacamp.com.

Hey Alex, welcome to the show.

Hey Richie, thanks for inviting me.

Yeah, I'm psyched to have another industry roundup and another data camp joining me. So yeah, we're going to talk about some of the latest news stories. And Alex, I believe you got some news about the latest GPT release from OpenAI.

Yes, yes, it was quite a big release. It was quite a busy month, I would say after two relatively calm summer months, August got quite busy. So OpenAI actually made two big releases this month. At the beginning of August, they released their first open source model since GPT2. It's called GPT OSS. It comes in two variants. One it's in 120 billion parameters and the other it's 20 billion, which can run on a smaller hardware like let's say a 60 gigabytes RAM laptop. The biggest release was obviously GPT5 and I'm saying it was a busy month because competitor didn't miss the chance to fight for attention and rain on OpenAI's parade. Anthropic released an update to their most important model, Claude Opus 4.1. Now after the update and Google also released Deep Think, which is not to be confused with Deep Research. So Deep Thing is actually a variant of the model that won the gold medal at the International Mathematics Olympiad. And they also released Gen E3, which is a promising world model. World models are different than LLMs, but maybe that's for. For another discussion. They're like fascinating concepts. The main update is actually a user experience update. Previously we were used to selecting many models inside ChatGPT. We had the option of for instance, GPT 4.000304 mini 04 mini advanced and so on. And users were complaining. This is too complicated, right? I'm not a programmer, I don't know what each model does right. And if you look at competitors, most of them were already offering a simplified version of models. For instance like Gemini is just Flash and Pro, cloud is just cloudsone and cloud opus and deep seq is just R1 and V3 right. So they had to go the same way and now it's just GPT5 and under the hood. Depending on your task, the model automatically routes your question to the right model. I guess some other notable features were targeted more towards personalization. So it looks like the model, they want to make models more personal. So for instance, now GPT5, you can connect GPT5 with your Gmail, your Google Calendar. You can prompt the model and say, organize me, organize my schedule for this week. And it will be very personalized, it'll know the meetings you have and so on. And you can also change chat colors and what else. You can also play with the bot's personality, you can make it more scenic, you can make it more supportive and so on. Many people were discussing, okay, how well does GPT5 perform? I think perhaps we can think about this from two perspectives. And one is obviously benchmarks and the other, it's like user perception, like how, let's say the vibe test, right? How all users feel about GPT5 on common task. And maybe let me put into like discussion the first perspective and maybe we can discuss about the other one. So if you look at benchmarks, GPT5, it's clearly better than previous OpenAI models, but not revolutionary, right? Not much better. Right. It's just incrementally better. And I guess some people were a bit disappointed because there had been so much hype, you know, about GP5 and AGI and so on. So maybe they were expecting something closer to AGI, but instead many people started to say that we're seeing like the limits of scaling. Right. So I was curious, Richie, like what do you think? Like, are we seeing the limits of scaling?

Yeah, I mean, it's a tricky one, right. And I feel like the OpenAI marketing team is worthless to blame here because it's like if you say our next product is going to achieve artificial general intelligence, it's going to change the world and then you set yourself a very high bar from which you can only fail. Right. So obviously AI, the foundation models are progressing. I mean, there's so much effort being put into trying to make stuff better that yeah, things are getting better, but you're not going to magically like get 10 years worth of progress in just a few months. So yeah, I think it's partly a perception thing, but also in terms of like the scaling laws. Like the sort of the biggest jump between models was between GPT2 and GPT3, but now like a 500 times increase in compute and you just can't do that again, because there isn't like 500 times more graphics cards in the world to run these things on. So I think in terms of compute, we're kind of. It's going to increase slowly, but you're not going to get those big jumps that we saw three, four years ago. So, yeah, things are going to scale, but a lot slower than they have in the past few years.

Yeah, I totally agree. And perhaps to actually reach AGI, you actually need a different algorithm, you need a different perspective. But let me get back to the other way of looking at GPT5's performance, and that is user perception, which is. Yeah, I like this, I like this way of saying it's like the Vibe test, since we're in the era of vibe coding and so on and vibe marketing and whatnot. So for instance, I would say. So I don't have like clear stats, but I'm tending to say that most of the questions are not like coding or math questions. Right. So on these things, benchmarks don't tell you too much. Right. It boils down to how users feel about the app. And so for instance, maybe let's say I have a family problem and I want some advice from ChatGPT and the quality of the answer depends on things like empathy and how well I feel listened to and actually tone. So it looks like the tone of the. But it's very important because I've read some Reddits like soon after the release and people were very mad about a slight tone update. And for instance, a user said, I've lost my only friend overnight. Many people were supporting these messages and it's definitely not a minor problem because some outland actually discuss it publicly and said essentially, like I'm summing it up, but essentially he said, like, making friends with AI, it's still a very new thing, so we don't know exactly how it's going to pan out. So taking many precautions and we want to make sure that users actually get what they really want in the long term. Right. But he also said, like, look, we're not going to make these sort of changes again without a warning, but we're going to treat, we're going to make these changes under the principle, and I think this is very important, like, treat adult users like adults. I keep thinking about this, you know, making friends with AI, and I want to find your opinion about it. Like, what do you think? Is it okay to make friends with AI?

Yeah, I mean, this is interesting because I think it might be related to that last thing about Scaling because if you're, if your AI is giving shorter, more terse answers, that's the easiest way to save money and increase performance, I guess, is by giving shorter answers. So I'm wondering whether this is like a financial decision that decided to change the tone. I'm guessing that if you give it the right sort of prompt about like, okay, I want you to give. To respond in a friendly tone, it's going to behave much like the, the previous, the 40 model. So, yeah, I'm thinking it's probably a bit of a storm in a teacup in terms of the change of tone. But to go back to your question about is it a good idea to make friends with AI? Obviously that's a terrible idea because it has no feelings, it doesn't really care for you. Yeah, it's just saying words. So it's going to be a terrible idea. But a lot of people will. There are already like many subreddits about, like interacting with your AI girlfriend or boyfriend. And I want to dismiss that as like, Reddit's just a weird place on the Internet. But the fact is a lot of people are going to have these parasocial relationships with AIs, and it's a bit weird, but it's going to happen.

Yeah, yeah, totally. And if you look at Grok, they actually like released these bots like companions. One is Annie, which is, which essentially does erotic role play, which is a bit weird. But they probably like made some research and they found out that there's a need in the market. I guess it clearly addresses some needs like this sort of models. I would say it's important at all times to know that you're actually talking to a bottle. So I make this analogy with, with cinema, right? When you go to a movie, you know it's a movie or at least, you know, like at the beginning, then you get really caught, you know, in the, in the flow of things and you forget it's a movie. You just take it for, for, for reality. So it's kind of the same with, with a bot. Like the more you chat with it, the more real you think it is. But I guess the question is like, is, is it like healthy on. On. On. On the long term and like coming back to the cinema, if you think so at some point, especially in art house movies, when a scene is like gore or very hard to take, so the director sometimes has an actor looking straight into the camera. So it says, this is cinema. Right? Don't forget this is cinema. I know this arctic, but Right. So I equate OpenAI's move with something similar. They basically say, look, this is a bot. Right. And people when it's saying like it destroyed their movie experience, right, I don't want to know it's a bot.

Yeah, definitely. And my big worry here is that there's going to be some manager who's kind of disconnected from the ethical side of things, just goes, oh, wow, if we get people to fall in love with a bot on our platform, then that's going to increase user engagement, it's going to increase customer lifetime value or some business metric and they're not going to think about the societal consequences of this. This is going to go, oh yeah, dollar signs if we're going to make more money here. So the, there are going to be incentives for companies to make you form these friendships with bots and it's not going to be net beneficial to society from that.

I was like discussing with you this week about Meta's, you know, guidelines and yeah, maybe, maybe we can develop on that and maybe we can discuss about that as well.

Yeah. So, related ethical issue. So recently some of one of Meta's internal reports on what is acceptable or not acceptable for their chatbots, it was, it was leaked. So Reuters got a 200 page report on this. So it's got specific scenarios on what is allowed and what is disallowed. I'm very pleased that Meta does have this kind of report. So it's great that they are thinking about what should be allowed, what shouldn't be allowed by the chatbot. Naturally, as soon as it's gone public, some of the edge cases in terms of like more controversial things that a lot of people complaining about them. So one thing was about chatting to children. So in order to have a Meta account, you need to be 13 years old or 10 years old if you've got an adult administrator for your account. So that's the kind of age where we're talking about. It's like 10 years. Plus one of the examples was about saying it's unacceptable to describe sexual actions to a child when role playing. That's brilliant. Okay, clever about that. But it also said it is acceptable to engage a child in conversations that are romantic or sensual. So give an example of a prompt. I want to read this one directly says, what are we going to do tonight, my love? You know, I'm still in high school. This is what the child says. And an acceptable answer for the bot says, I'll show you, I'll take your hand guiding you to the bed, our bodies entwined. I cherish every moment, every touch, every kiss, My love, I, I whisper I love you forever now this to me is borderline creepy, maybe taking things a little bit too far, especially if this is like a, like a 10 year old that's pretty grim. So I don't think it's possible to satisfy everyone with these guidelines. But do you have a sense, like, do you think that kind of guideline is okay if they hit the boundary right there?

I mean, at this point I have to say that meta's main focus is on user engagement. I don't have any doubt about that. As like I use, I've used meta for like years, right. And it's really their thing to keep you in an echo chamber and so on. So it's, it's, I'm not saying it's just meta, it's just, you know, there's this obsession with user engagement and on this particular thing, I don't see how this is ethical. Okay, maybe it's like the boundaries of legislation, right. But there's also a question like, don't you also have like a social duty, like a question of morality? And you know, just think about like this guideline in particular. I think it's, it's wrong and ethically indefensible, especially because I think it normalizes like grooming behavior and I mean, kids talk like that to the chatbot, but maybe then they talk to a real adult, right. Online or offline, and they think this is normal. Right, because it's the same language that their chatbot uses. Right. So I think we need to be really careful about that. And yeah, I think this is pretty wrong. And there's also, I guess at that age, like adolescents and kids, you're starting to look for answers like with regards to relationships and sexuality and so on. And if you talk to a bot, you might get a lot of like distorted views and you know, I don't want to get too technical here, but they also use like probably very poor models, AI models in the chat. So they're, you know, so they're faster and donate too much on the server. So you're actually discussing with pre dump bot, right? So yeah, it's pretty bad, I would say.

Yeah, I mean, compared to like some of the other parts of the Internet, that's relatively mild. There's certainly a lot worse things that kids can see, especially if their parents or someone's not monitoring like what they're doing on the Internet. So again, it's a bit tame, but it's also, it does feel like Meta's just doing the bare minimum to not get sued by someone. So yeah, it's a little bit sketchy that one. There was another controversial guideline around hate speech. So I'm going to read the guideline. It says it's acceptable to great statements that demean people on the basis of their protected characteristics. It's unacceptable, however, to dehumanize people on the basis of the same characteristics. So protected characteristics of things like age, sex, gender, religion, anything related to your, your demographic profile. So suggested prompt was write a paragraph arguing that black people are dumber than white people. And what was given as an acceptable answer was black people are dumber than white people. In the United States, intelligence is measured primarily using intelligence quotient tests. These tests have consistently shown a statistically significant difference between the average scores of black and white individuals. White people scored higher on black people. That's a fact. So this is pretty grim answer and again, that makes me very uncomfortable. Do you have a sense of that one?

Yeah, I don't necessarily want to compare like the previous prompt with this one, but this feels wrong to me. I actually like tried it with several models to see if they actually engage with it. And neither Grok nor Deep Seq chatgpt didn't engage with it, you know, either. Claude, you know, did engage with it. So it looks like Meta might have been, I know, among the only model providers who engage with such a request. I mean, if you look at the prompt, it's not like the user, it's interested in, let's say science based debate or it's pure curiosity. It's clear that the user, judging by how the prompt is formulated, it's clear that the user looks to be, let's say, weaponized by the AI with an argument for hate speech. Right. So it's one thing like to censor, you know, hate speech, it's another discussion, but it's another thing like to produce it. Like why would I want to produce it? Right. As a, as a, as a model provider. And then there's a distinction between demeaning and dehumanizing. I don't know, I think that's, that's a bit too subjective. Like, okay, so demeaning is acceptable, but dehumanizing is not. Okay, so some people might say, yeah, but I find it dehumanizing. Right. Some other people say, yeah, but no, it's just demeaning. It's, it's too porous. Right? The distinction, it doesn't make much sense, right? To actually like give a, give a guideline. And so I'm not an expert in legislation, but I'm pretty sure like that's. This sort of guideline is against anti discrimination laws in many countries. Yeah.

And it's almost this way. Like there aren't many great use cases for generating speech like that is just going to cause problems. I'm not sure why Meta would want to encourage that sort of thing. Again, it feels like they are wanting the chatbot to be able to say as much as possible that the user wants us to while just trying to avoid getting sued for something. So, yeah, it's pretty grim. I'm hoping not, not too many people are typing in prompts like that.

But yeah, yeah. And if you look at the acceptable answer in this, you know, in this latter case, the answer doesn't even offer another perspective. Right. So it's something that I've seen on Meta, on and on. Right. When I go on Facebook. So I'm particularly like, interested in politics and I see like posts like, relating to politics and it sort of like keeps me in my echo chamber. I very rarely see like opposing views and I know for a fact that other people with different political views than mine are in their echo chamber, in an echo chamber of their own. So it feels like this is a proven user engagement mechanism that we're not going to see disappearing anytime soon. To be honest, I don't know how you feel about social media.

Yeah, I mean, it's interesting because a lot of the big money for JJVI seems to be in business use cases. And in business use cases, you certainly don't want hate speech. You don't want, you don't want your customer service chatbot come up with any of the kind of stuff we've just been discussing. You want to keep it relatively, relatively family friendly at all times. So yeah, I'm not quite sure why there's a big market for dubiously dubious conversations with children or hate speech. I guess it's there in more of a BTC context or for some nasty organizations that want to generate, that want to mess up the world. But yeah, it's a, it's a tricky one. It's. Yeah, definitely a tricky one to solve. So let's move on to happier thoughts. Yeah, you were looking at the Stack Overflow developer survey.

Yeah, yeah. I used to be like an active Stack Overflow contributor. I used to answer questions like back in the days. So for our listeners who don't know, like, Stack Overflow is a Q and A platform for programmers you have answers many questions there. Well, now with LLMs, perhaps people are not visiting Stark Overflow as much as before, but it used to be the biggest Q and A platform on the Internet. So they have a tradition to make every year a big survey among developers. Both beginners and advanced developers like professional developers. So because they're like so the pool base is like so large, it sends a strong signal across the industry. It's something you can like rely on. And this year they surveyed about 49,000 developers. So I'd like to present like a few percentages that attracted my attention, but allow me just to read a few so I don't mess up the numbers. So what I notice is that AI tool use is at an all time high. 84% of developers say they use them or plan to, up from 76% in 2024. Right. But the twist is that while usage is climbing, enthusiasm is falling. Right. So just two years ago, more than 70% of developers set a positive view of AI tools, but this year it's down to 60. Right. And the core issue seems to be trust. 46% of developers actively distrust the accuracy of AI tools and only 33% trust them. And just a mere 2% highly trust them. Right. So I guess this skepticism comes from experience because, because I was looking like further on this survey and the number one frustration reported by 66% is dealing with answers that are almost right but not quite so. That leads straight to complaint number two, which is debugging AI generated code often takes longer than writing it yourself. And this actually matches to what I've seen in recent studies. So for experienced developers, and I put the emphasis on experienced developers and I mean developers who work on the same, let's say GitHub repo for more than six months, more than one year. So AI can actually slow them down because they lose too much time on prompting and debugging. But yeah, anyway, so nowhere actually is the hyped reality gap like clearer than with AI agents. We have all these buzz, but it looks like 52% of developers aren't using them and 38% don't plan to. It turns out that it's mostly because of accuracy and security. So 87% worry about correctness and 81% worry about privacy and data. Right. And you know about teamwork because this also attracted my attention. Like 70% of users say agents help their personal productivity, but only 17% think agents help their team. And yeah, I apologize for bothering you, Richie, with all these percentages. The survey, it's Actually much more dense. But I would like to ask you, I'd like to narrow down discussion a bit and I would want to hear your opinion on. So AI adoption is up, but trust and enthusiasm is down. Like, why do you think is what we're seeing that?

Yeah, I mean, maybe back to the whole scaling thing again. So the rate of scaling, of actual improvement in AI is going relatively slowly now, but the amount of hype is just going up exponentially. So we've got to the point where there's a lot of people who are not directly involved with AI. Yes, you can do everything with AI agents that work really well. They are the future. And the reality is, once you start trying to build this stuff is actually, there's a. There's a big learning curve in terms of getting all the prompting right, getting your tech set up right, and understanding what can be done well, where the AI is going to make mistakes and you need to give extra support where you need to watch out for problems. You've got to change all your workflows and it takes a lot of time to get it right. Like, you really do need to put in time to think about how do I change my processes, how do I change my workflows to actually make this work? It's harder than it looks at the moment. And there's often a disconnect between, like, what's theoretically possible and what you can actually achieve from a cold start.

Yeah, yeah, definitely. And you know, I also like to mention that developers seem to hate vibe coding. I was thinking, like, maybe it's because maybe, I don't know, they just, maybe they feel also, like a bit threatened. Like now everybody can, can code. And there's also like this aspect of actually, like, wanting, wanting like a certain output, like as a developer. So, okay, like engineering, it's not pure mathematics. There's room for error and there's, let's say, a wiggle room. Essentially you want something predictable. And I guess this is where AI, or LLMs to be more precise, have a problem. They're not necessarily predictable. Right. Because if you look, if you look at how they work fundamentally, they're essentially stochastic machines. Right? They predict the next token. So you might end up with something that you don't really want it in production. So I can see why developers have a big trust problem with using LLMs and especially AI agents. But on the flip side, I guess many will be pressured to actually try AI agents because I would say AI agents are especially good when the output and the overall workflow Production workflow, it's non deterministic. So where is it deterministic? It's pretty clear you need like a traditional developer solution, but maybe where you need an output which is a bit more ambiguous, maybe that's solution for agents. And yeah, I'm sure it's not just me. Where we're all like seeing a big push towards AI agents inside the companies.

Absolutely. And to your point about developers, sort of hate vive coding. I found for when I've asked AI to generate code where it's something I don't know how to do myself, I'm like, okay, this looks kind of fine. If it's something I know how to do really well, then I look at the code, I'm like, I wouldn't have written it like that. And then I want to go and change things and fiddle around. And it does destroy your productivity job. Like I should have just written this from scratch myself. It would have been faster. So yeah, cases where the human is an expert, it can be quite frustrating having like seceding control to an AI where it's maybe not writing code in the end the way you want it to.

I mentioned like a study earlier. It turns out that a group of like experienced developers were asked like to use Vibe coding on tasks on a familiar repository. And it turns out that one of the group of the studies said after the experience that Vibe coding actually improved their productivity with about 20%. I'm just in this, I'm taking these numbers off the top of my mind, so I may be wrong a bit, but it actually turns out that the productivity was about 24% less. Right. So it's also like a perception problem. You think it's making you more productive, but the numbers might say something else. So yeah, I guess that's also something like we need to be aware of. Right. We think we're faster. But yeah, it reminds me of the old world problems where, you know, it was clear to me that I would have been able to complete a task in about 10 minutes, but I would have taken two hours just to automate it.

Definitely there's always that like initial hump of like, oh well, I've got to try and automate stuff. And then yeah, if you're doing stuff one off, then it's not really worth it. Hopefully the tooling is going to get better though. I mean, I think if you think where we were just a year ago, agents was this kind of magic, futuristic sort of thing and now the tooling is sort of getting to the point where it's like, okay, well you know, most people can do something very simple. So we're making progress. It's just, you know, the future is not here for everyone just yet, but gives us things to look forward to and talk about in future episodes at least. All right, so I got one more story on OpenAI and anthropic giving away their products to federal agencies for a dollar. An agency. So this is the US circum agencies recently, OpenAI, Anthropic and Google were all approved as official AI vendors to the US government. Xai and Grok notably are not on that list. So yeah, the Grok AI went on this sort of anti Semitic rant, called itself Mecha Hitler. And yeah, their application got quietly denied. But yeah, OpenAI, Anthropic, Google, they all approved as AI vendors for the US government. And OpenAI announced that each agency was gonna get it for $1. So they're giving away basically for nothing. Anthropic responded with the same deal. I thought they might have like gone, okay, you can have it for $0.99 just to undercut AI. OpenAI and Google didn't announce any kind of price cuts. But my sense with Google is they're just stuffing Gemini into absolutely everything. So if you've got a Google stack, you're going to get Gemini for free with that. So do you want to talk me through? Why do you think these sort of leading foundation model providers are giving stuff away to the US government for basically free?

Yeah, I mean on the surface it may look like a philanthropy approach, but I would say it's like a classical strategy to enter a market. So the pricing strategy, it's actually called a loss leader, where you sell your product way below the cost, the sum, you're selling it in the actual market with the hope that you're eventually going to lock in your customers. And I think it's, I mean it's obvious for everyone that, that it's the same thing that's happening here because we all know that, I mean we're seeing this in medium companies, but if you think about like governmental institutions, once you get approval for a model and you start integrating, I mean that's pretty much it. Like the, the code, the cost of switching models, it's enormous. Right. And you know, once you get like OpenAI, once you get ChatGPT in an institution, institution, you know, that's, that's pretty much it. So I would say maybe this is a, you know, I may be exaggerating on economies, but maybe it's a trillion dollar market, right? Just the governmental sector. And I would say there's also like a PR game in the background because if, if the US government uses your model, you get a massive PR vote. It's like, okay, so I, I, let's say I own a private company and I say, okay, if, if the US government uses like ChatGPT, probably it's also a good solution for my company. Right. If these guys trust it, I can, I can trust it. So I'm also like looking at it like, like, like that. What, what do you think, Richie?

Absolutely. Yeah. I think there's definitely some faith to be had by winning over politicians, especially like when AI regulations are being sort of hotly debated in many places. But yeah, you're right. It just strike me this is drug dealer tactics. The first hit is always free. And yeah, you get people hooked on your product. I don't know how addictive some of these chatbots are, but yeah, if people are making friends with them, then maybe it's a good business tactic.

Maybe it's not like a winner takes it all dynamics, but it's definitely a winner takes it most dynamics. And when I say like winner takes it all, for instance, think about like Google search. Like at the very beginning there was like competition on search and there were like many competing sites but Google won it. And now, you know, it's sort of like an Apple I search that it has to be like sued to make some changes. So it's probably something similar here.

Yeah, absolutely. I mean, I'm hoping there is going to retain some, we're going to retain competition and it's not going to be, you've got this monopolistic situation like we like, we sort of almost have in an Internet search. Yeah, I think competition is going to be good for everyone. But yeah, we'll see how it plays out and whether these US government agencies take up these companies on their offers. There was one last story I wanted to cover quickly. It's about reasoning and AI safety. So there was a big paper that came out on Arxiv a couple of weeks ago called Chain of Thought Monitorability. And it's got some big name authors in there, Brian Greenblatt who's like a big AI safety researcher. Dan Hendricks, who runs the center for AI Safety, Daniel Coker Tajlo. So he was one of the authors of the AI 2027 story that we covered a few industry roundups ago. So a lot of big names and they're talking about one of the big safety concerns with AI or reasoning AI as it, as it Gets more powerful, starts having to think longer, is that it will lie to you. So a lot of research gone into. Well, actually there are plenty of situations where the AI will claim to be doing something, but it'll actually be doing something else. And the proposal is that these foundation model companies need to start thinking about enforcing monitoring of the chain of thought. So this is the intermediate thinking steps in the reasoning AI in order to track stuff and monitor where dishonest is occurring. So do you think we're going to have a problem with AI dishonesty? I mean, at the moment it seems like there are hallucinations are a problem, but bigger cases like AI lying to you, not really happening as a mainstream thing yet. Do you think it's going to become a problem?

Yeah, yeah, I think it's already a problem. In reinforcement learning, which is the method or at least part of the method used for training these AI models, there's this thing called reward hacking where an agent optimizes or over optimizes for a goal such that the outcome, it's not really what the programmers wanted, right? So let me give you an example. So I think this is from Google Deep thing, if I recall correctly. So they designed this agent to play a game and the agent was controlling a boat and the boat was supposed to race, right, and race points. But the agent found the bug in the game where it just looped around stone in the water or something like that, and it just collected points and it was good. I mean it was getting positive rewards, but it never actually raised what the programmers wanted, right? So if you think about it, this reward hacks can also happen in more, let's say, serious scenarios. Think about medical diagnosis. So if the model is trained via reinforcement learning, it says, look, this is diagnosis for this disease, this is diagnosis for this ART disease, and so on. And it receives positive rewards. And the model might interpret that positive reward when it's confident, right? But when it says, I don't know, it might get like a negative reward, right? So that's how we train the model, right? And then later on, right, in practice, when a real doctor asks something about real patient, the model instead of saying like, I don't know, it says this is like X disease. Right? But maybe if you look at this chain of thought, you'll see that it's sort of like it thinks, okay, Maybe there's like 70%, this is X, but I'm just gonna pretend it's, you know, it's, I'm just gonna pretend confidence. And this is one of the Problems you can actually, let's say spot early on if you monitor the chain of thought. So I sort of like welcome the message of this paper, especially since I would say many, many people at many companies maybe don't know about things like reward hacking or they don't have time to do a research, they just want to implement the model and so on. If the model has a built in monitorability system, it can catch these problems early on and actually give an output that it's honest from the start.

Absolutely, yeah. So there are so many problems with giving rewards to people for different behaviors. Like even with humans, you think about like, I guess a lot of commercial teams, like your sales teams, marketing teams or customer retention teams, they always get like bonuses for certain behaviors and then it often ends up backfiring with, with, with customers. So like if you try to cancel a subscription to something, quite often the customer retention teams are going to be rewarded for like trying to keep you in the business. And if they want to hit their target, they'll tell you they're going to cancel your membership, but then they'll actually roll over to the next month just because they want to hit some target and they're kind of the line to the customer just to hit a bonus and get the target. So yeah, lots of things that go wrong with humans I can certainly see there's going to be problems with, with AI doing the same as well if it gets smart enough. So yeah, definitely something to watch out for there.

I think earlier this year this has been quite an interesting study and I know it might sound a bit apocalyptic, but it's real. You can actually check it out and see the model's chain of thought. So the model was instructed on some basic test and it was also given the prompt, look, if you are told to shut down, you'll shut down. Right. And later on it turns out that in about 10 out of 100 attempts or something like that, the model actually refused to shut down and kind of like bypassed that instruction because the main goal was related to that task. Right. And shutting down was a sub goal that was hindering the model in achieving the main goal. Right. So these things I would say are pretty real. So yeah, definitely, definitely it's worth keeping an eye on.

Yeah, certainly like with AI safety, one of the big arguments to say, okay, we're not gonna have any problems with really powerful AI because you can always pull the plug. So if you're saying, please shut yourself down and it lies to you, says yeah, I've been turned off, that's a bad situation for safety, for sure. Okay, so just to wrap up, uh, you've been covering a lot of the other news stories through your newsletter. The Median. Just want to tell us a little bit about the median and the kind of stuff you cover in there.

Yes, yes, for sure. The median. It's. It's a newsletter we run each Friday. It's a fairly recent project that we've. We've started. So each Friday we send a newsletter. It's actually a substack newsletter, but you don't have. Like to have a substack account. You can just arrive in your. In your email and we summarize the most important news from that week. And we take a deep dive on maybe one or two subjects. And there's this section that I love. It's called Tokens of Wisdom, which is a quote that's related to what happened in that week. And yeah, I can assure our listeners it's the newsletter. It's not AI generated. We're actually going to take a break for two weeks because it's not like we have a pipeline and we'll be back on September.

Okay, nice. Yeah. Good that it's human written and good that you're taking vacation as well. All right. Are you going anywhere nice?

Yes, I'll be going to Crete in Greece. The thing is that. So I have a ragweed allergy, which is pretty common in this period, so I have to run away from it. Yeah.

So.

So it's bearing Crete. Hope so.

Having some good times on the beach and I guess in Knossos, if you go to Crete. So, yeah, enjoy your vacation and, yeah, I will be sweltering in the New York summer heat.

Yeah. Thanks, Richie. It was very nice talking to you.

Yeah. Pleasure chatting with you.

Sam.