Ep 22: Levelling Up: Exploring AI at Project Rising

Background for listeners. This is effectively a Secret Squirrel test program that we're running to evaluate the psychological capability of these AI models to be World War II spies for the British Army. But just seeing how it performs and how it thinks, to come back with responses, has been brilliant.

I think as humans, it's important for us to try and be familiar with what the capabilities are so that we can make sure that we're always delivering something on top of that. So if you look at the current state, lots of people are using the, well, AI tools to automate the boring, the less important parts of their job, and then that gives them more time to deliver value on top of that. So I think as the capabilities continue to improve, we need to keep looking at what else can we do?

Hi, I'm Freya. Welcome to the Skynet Chronicles. Business in the AI Age, the podcast that dives deep into the world where artificial intelligence meets because business. Join us as we explore the latest trends, challenges and opportunities in this ever evolving landscape. From innovative startups to established enterprises, we'll uncover the secrets behind the successful integration of AI technologies. So grab a cup of coffee, sit back, and get ready to unlock the mysteries of business in the AI Age.

Well, good morning, Alfie, and welcome to the Skynet Chronicles.

Morning, Tim. Good to be here.

So, Alfie, you are now on day seven with Project Rising, which sounds like you're brand new to us, but that's not really true because we've been working together for what, 18 months, is it? On and off?

Thereabouts. It feels like maybe even closer to two years, but two years, 18 months, it's all kind of the same ballpark, right?

Crazy, isn't it? So, seven days in, what have you learned with Project Rising?

Lots of new things. So I got to play around with the 01 preview, or I guess Strawberry, as it's being called. So that was fun. Otherwise probably wouldn't have been able to get my hands on that. Learning about make make.com, messing around with some automations, got to play around with some typescript, sort of making some changes to something that Leo worked on. Never played with typescript before, so that was a fun experience. And now I'm set up with Copilot Enterprise. So I've messed around with Copilot before. This is the coding assistant, not the office assistant. So having to play around with that, so lots of new things to play with has been the focus, I think.

Excellent. So learning curve like a wall, but lots of toys to help you get there.

Yeah, absolutely. I mean, everyone's very supportive and welcoming. So it doesn't feel like it's going to be too hard to catch up? No.

So let's talk about O1 preview. Because it is a crazy smart model, the reasoning tests that we've done with it have been fantastic. Not managed to top it out on maths or anything yet so far. You talk me through the particular play and sessions and experiments you were doing with it.

So for me, I was playing around with the verbal reasoning side, didn't notice, like a significant change over other models. But I know it's still in the preview stage and I think we need to play around with the prompts a little bit because I think these ones in particular are designed to sort of catch you out on the wordplay, like little simple problems.

Oh, this is the 1941 test that we haven't published yet.

Yeah, yeah, that's correct. Like very, very sort of nuanced tricks in the text sort of catch people off guard, which I think as a human, if you're naturally. You've seen these sort of problems before, you're kind of distrustful and you're looking for the trick. Yeah, you do a little bit better, which I think the models aren't. So I think maybe we could play around with the prompting a little, maybe give it a bit of a hint.

I'm quite keen that we don't. So for background, for listeners, this is effectively a Secret Squirrel test program that we're running to evaluate the psychological capability of these AI models to be World War II spies for the British Army. It's fascinating, but you're right, the questions and the way they're written, they are tricksy. And obviously with O1 preview, currently it's not Motimoldu, so we can't feed in the image side. We can only do the text evaluation. But just seeing how it performs and how it thinks to come back with responses has been brilliant. And we played through an example test that I'd done with it where I. I'd use GPT4 to create this ridiculously comprehensive prompt around model railways and specific types of track. Fed it into O Preview and it thought for 40 seconds before it answered every aspect, including calculating the force on the track, whether it could get up the slope, the exact design, and came back with a concept for the design of a track that met all of the requirements of the prompt within 40 seconds. Just breathtaking reasoning and maths combined in a single response to a prompt. But very much doesn't deal with it's crap in, crap out. You give it a weak prompt, it's not going to do much with it. If you ask for something super comprehensive, it gives it the opportunity to do more. And the reasoning it's demonstrating is breathtaking to me. I think on those little questions, there isn't really the ability to demonstrate that it's effectively still just using GPT4 smarts, but where we're giving it a challenge where it can think about something and come back with an opinion and use that opinion and that chain of thought which is somewhat hidden, as in we can't see how it's going through those steps, we just get the representation of it is giving it the room to answer challenges which the previous models haven't been able to.

Yeah, I mean, I think it's very impressive. Like, I had a look at the output of the test that you ran. I think just looking at the test itself, like, I've got a background, I've done mechanical engineering, and I think I would have still had to sit there for a good while to work through the steps and I probably would have made a lot of mistakes. So the fact that it came out of a strong output so quickly was very impressive, I think for the 1941 test when they add. Because I think in the preview they haven't yet added the ability to also work with images which should be coming out soon. Maybe by the time this, this, this episode makes it out, it will be there.

Yeah.

So looking at how it looks at the sort of mechanical systems diagrams and figures out how if you move this, what will happen to this part, I think it'd be very good to see how it does with that, because I know that's something that the other models were struggling with.

Yeah, yeah, I think we're going to see. I mean, it is just a preview model and it's incredibly limited on the amount of requests we can send. If you're using ChatGPT, it's like three or four a day. If you're on the plus package, if you're using the API and you've got the. I think they call it Level 5 Usage Tier, I. E. You've been hitting their API an awful lot for an awful long time, which we're privileged and have paid for, then we get this access where we can ask 20 requests a minute, which is great, but in an application is completely insufficient. So it's still just for prototyping. But yeah, when they release the full model that's multimodal, I'm very excited to see what can happen. The bit that's really intriguing me is where they are effectively evaluating the chain of thought internally within OpenAI. So they don't publish that chain of thought that the model's having. They're not restricting the model in what it does in any way, shape or form in terms of its chain of thought, but they're then able to look for signs of power grabbing or self replication or steps that would be alarming in terms of AI safety within that chain of thought data that they're going to have on their systems. So in terms of interpretability and safety of models, that's a step change and it's what comes next with that understanding and how much of a paradigm shift is this approach to reasoning versus all of the other large language and foundational models that are out there?

Yeah, yeah, I think it's gonna be a big improvement. So I was reading that even though it's still in the preview, they ran some tests against it. I think there's a thing called the simple QA test which showed that there was a reduction in hallucinations compared to GPT4. And there's another thing called the barbecue test, which is a new barbecue. I've never heard of it before, but apparently the idea is that if you're having a, a beer or a sausage with your friends and you talk about where you work, it's how proud are you to say I work at this place, based on what the perception is of how they treat consumers and customers from an ethical perspective.

Wow.

So this test sort of asks some questions and it looks for typical like stereo stereotypes in the response. So it's kind of a test of how, how well it sort of. I know there's all this problem with like biases of models and ethical considerations, so it's just testing that. And yeah, 01 scored better on that compared to GPT4 as well, I think by a considerable amount actually. So nice to see that those sort of features are getting integrated within the models themselves.

Absolutely. And that interpretation, interpretability of what's going on within a model is fascinating. Danny, a developer who works with us, was sharing yesterday that for his master's dissertation he'd been working on an LLM which he built himself with, with a training data set which he created and then looking at the different layers within the model to assess an understanding of trying to evaluate what that model is doing and how it's thinking. And it just popped my mind. It reminded me how stupid I am. Just because that level of understanding he's got and the depth of detail he can go into it just blows my mind. But having those conversations working on how these models can be understood is key to how we then scale and understand what the impact's likely to be in the future. It's all very well having ever increasing intelligence and capability, but we need to know we can trust it.

Yeah, absolutely. I think when I was working more on machine learning before, we always favoured models where we could really interpret what was happening and properly understand from start to finish how we got to an output. And now I think, yeah, explainable AI is so important. I think as well we're seeing like tightening laws around copyright and stuff. So again, it's going to be very important to have the ability to be able to say this is what was used to create this output to ensure that you're not sort of infringing on any copyright laws.

Yeah, fascinating. And if those copyright laws, in terms of where the data came from initially, if it's open source, public domain information, that's one thing. If it's entirely synthetically created data that's come out of a model in the first place that you're then using as your training data, it just. The potential is endless, isn't it? Because then you can create entire test data sets, you can create entire all of the material that a model is then trained on. You can take these giant models like GPT4 and go, okay, well can you distill down everything you understand in science about this subject, this topic, this particular item and summarize it, give the findings, give the equations, give the main people involved in it, repeat that for every topic across every science, produce just this document that is then being fed in as the source of truth for science? Your training data set can be that much smaller, but the capabilities that are embedded within it can far outperform these giant models.

Yeah, yeah, absolutely. And I think like you said, it's just creating much more high quality data set as well, which is very important. And I think just having the ability to fine tune more what goes into a model is going to be very helpful because you can start mingling in your own proprietary data. A lot of companies have just got terabytes and terabytes of data just sitting there and they're not using the full value of that. I think a lot of them, they're not even using maybe 5% of that data. So the ability to be able to mingle that in and really sort of upgrade what comes out at the other end, I think will give people a big edge.

So we're at danger of going into A massive geek hole at the moment. So let's step back and tell us about Elfie, what you've been doing, what your history is, what you enjoy doing and what you're hoping to do with Project Rising.

Okay, so I think we'll start with one step back. So before I was working for a large video game publisher looking at using machine learning, specifically within marketing, so looking at how we could use player data to better understand what the interests of that player are, what that player is, what sort of content they're likely to.

Resonate with, what extras you can sell them.

Yeah, essentially. But I think it goes both ways because we're using machine learning to look at how to sort of sell people more stuff. Yes. But also looking at what players aren't interested at all. Because if a player is not interested, then why are we bothering them? Some players are quite interested in buying content. I think some, some players especially become deeply invested with a certain game and they want to spend a lot of time on that game and they don't want to play over games, so they're not spending money on other games as a regular gamer was. So they're quite happy to invest money in their hobby. So just looking at what sort of stuff they'd be interested in that's going to make their gaming experience better. So it's not all.

It's not all, no, no, it's absolutely mutual.

Yeah. So. So that was the, the focus of my last job. I was doing that for a few years. Before that I was doing some various bits, freelancing, working within data. I think that's how I sort of came across machine learning. So one of my very first freelance gigs was looking at sort of using just some, doing some simple Python tasks and one of those was machine learning and it was just a simple like predict if someone would like this car based on what cars they've driven before. And I just thought it was really interesting and I sort of fell down the rabbit hole and ended up becoming a data science. Data scientist.

And I think it's worth sharing that your journey isn't the typical one into code and data science. You didn't train, you didn't go through a degree path in order to get to this. You went mech enge and then you've sort of pivoted towards this through the wonder that is the platform upwork, where you've effectively done these jobs for various companies, including Project Rising, getting an understanding and a feel of what's possible, building your experience that's then led you down this path. And I think it's brilliant because it shows that there are multiple ways to get into a career and it's never too late. You can choose and you can influence what you're going to be doing. Tell us a bit about what motivated you down that path, or was it just a series of happy accidents?

I think it's a bit of both. So when I, when I started off, it was in lockdown, so I had loads of free time. I was working in hospitality at the time and I just kind of wanted something to do that was fun. So that's how I came across coding. There was no intention of I want to become a coder by trade. It was just, this looks like a cool way to solve problems. And at that time, there was no coding assistance or anything like that. There was no chat GPT to say, how do I do this? So I kind of had to do it the old school way, but it was still really easy. There's so many resources out there for people to learn how to write code.

Yeah.

And I think again, I just thought this would be a cool skill to pick up. And I ended up getting really into it, actually. And then that's why I started doing the freelancing. Because I think the best way to improve your coding skills or your tech skills in general, is to sort of work on real projects where you apply those skills. So freelancing was perfect because it was just a way to get paid to learn, essentially. And that's kind of how I got got where I am now. But I think I just really love the problem solving elements and I think that's what we're, we're doing at Project Rising every day. Right. People come with their problems and we find cool ways to solve them. Yeah.

I've got an idea. Great. Okay, well, first of all, let's see whether this idea has got legs. But if it's got legs, then how do we go about solving it? And it inspires me every day that we can help so many businesses understand the potential. What the art of possible with AI is where we can help a company understand whether their idea or even an individual, whether their idea could actually become a product and if it became a product, how they'd monetize it and how they would scale it. And if that actually has merit. And some people, when I'm talking to them, they're like, are you this nasty to everybody? I'm not being nasty. I'm just checking that you actually want to go through this and you really understand what or this product needs. To be in order to be successful. I was having a conversation with one guy who, he's a good friend and his company has already had work from us previously and he was talking through this concept and over the course of an hour long conversation, we completely pivoted what the product would be to something that. From something that probably would have worked to something that's actually amazing. And I'm really excited to start working and building on. Although I probably shouldn't do any of that code, I should hand that over to you guys because it'll be better written. But it's just how we can help people and then how we can take that understanding for a specific problem and apply it to the toolkit for SMEs where we're effectively building out these capabilities so that any business can benefit from the potential of AI. It's fascinating times.

Yeah, yeah, absolutely. And I think with that toolkit especially, it's just like you said, once we've got a better understanding of what the real problems are, then people come along and there's already something off the shelf that they can use. I think it makes the whole process a lot smoother for them and I guess it's easier for us too.

Well, the customers have effectively validated a concept already. They've come to us with a problem and we've solved a problem. And if that problem then exists in other businesses like how do I get paid? How do I create a high converting proposal, how do I make sure it's been signed? These are repeatable, regular problems, but if you then fit them in in an overarching service where they're effectively little. We were speaking with Tom earlier, he's talking about everything as robots, but effectively these micro pieces of code, be them agents or microservices that are able to complete an action, you get a group of them working together on a problem, they can be used in different ways. It simplifies the process of developing applications because we can pick the bits that do the bits that we need from a broad spectrum of different solutions. But for the end customer, they can get a tool set that means that they can perform as a business that's 10 times their size without having to spend the money on it. They've got the tools, they've got the resources that they've never had before. I love it.

Yeah, I mean, I think it's win, win because, well, we'll have the solution ready to go. And in cases where we haven't solved that problem yet, that means that we've got the bandwidth to focus on those new problems and find new ways to sort of work through these challenges.

So what are you hoping the next 18 months in technology brings us?

That's a very tough question because we.

Can'T forecast what's coming. Is that what you say?

Yeah, I think it's very hard to predict. I think once you say something on the Internet, it stays there forever. And I feel like whatever prediction I try to make will probably be horribly wrong.

We won't call it a prediction then.

I think in the short term we're going to see a very vast improvement in the quality of language models. And especially looking at like the multimodal ones, I think that the quality is only going to keep getting better and better. Hopefully we'll start to see more widespread adoption as they become a little bit easier to work with as well. But I think there's still going to be a long time before the adoption rate catches up with the rate at which these things are increasing.

Yeah, I mean technology is going to keep racing ahead and some companies will adopt quickly, others will fast follow, some will be laggards and they will be left behind. I think the bits that are really interesting me outside of sort of the foundational models and LLMs and what we can do with agentification, the likes of Video JEPA and UI JEPA out of Meta and Apple respectively, where they're building an understanding of effectively the world through masking videos and, and the intent of a user through masking parts of the interaction journey on an iPhone, to then build a model that understands well what's going to happen next. And not just a stochastic parrot that's spitting out tokens, but a general world model that understands how things work, truly understands physics, understands object placement like we've seen the sort of the signs of with Sora, but we haven't actually had a chance to play with that in anger yet. So I think we're going to go through just a huge release of new types of technology that are going to be as game changing as ChatGPT has been for just improving access to language skills and words, but across multiple verticals and how we then bring in those capabilities to make solutions for our customers. It's going to be a roller coaster, isn't it? I can't wait.

Yeah, I think it's going to be interesting. I'm excited to see as the sort of the AI systems that are out there as they get an increasing ability to integrate with other systems without having a human sit there and manually connect up. So one of the things that we do at the moment is we look at what technologies are available to solve problems rather than building things from scratch and then we're connecting them up. Right. So soon I think these models and systems are going to have the ability to connect themselves without us needing to be so, well, so explicitly do it. Maybe at first it will start with we want you to integrate with this and this and eventually it'll get to the point where they can find the best way to do it themselves.

Yes. Which is a little scary. We need to give them guardrails, we need to make sure that those controls and constraints are in place. But I think you're absolutely right. These sort of self solving, self connecting systems will produce huge breakthroughs. But even in the meantime, just breaking things down to we've got a series of agents that have a task and as a task planning authority, a central mind if you like, knows problem X needs to be solved. Which of the tasks do we use in which order? Order them through a message queue that tracks whether something's been completed or not, has a state memory so that it's able to go, okay, well these things have happened. I've got this input data process through the message queue, get the responses back from the individual agents on their tasks, pull that information together, beat it down to sequential agents and get that completed action list. Fundamentally that's a project. A project is a list of tasks delivered by different resources with a project manager making sure those things are completed. And it's really, that's what we built with the original version of the Project Rising project management platform was that task management and allocation to resources. And at that time that was a combination of freelancers on upwork and API calls out to specific AI services. But it's now a next level of detail down where we'll be able to complete a project across a suite of agents that are operating for our customers effectively across the board spectrum of their businesses. What you can then do in terms of a plan is like don't just develop the marketing plan, create the assets, issue the assets, report on how they're doing, come back with an updated plan, feed that into a management board which is also AI that's giving the feedback on how these tasks have performed to the person that's gone, okay, how are we doing this month? What are we going to do next month in terms of our marketing strategy? And it can be a self fulfilling prophecy where it's just feeding back and improving itself. The potential is just fantastic.

Yeah, absolutely. I think it's very cool. And as you said, it is very scary. I think as humans, it's important for us to try and be familiar with what the capabilities are so that we can make sure that we're always delivering something on top of that. So if you look at now the current state, lots of people are using the, well, AI tools to automate the boring, the less important parts of their job, and then that gives them more time to deliver value on top of that. So I think as the capabilities continue to improve, we need to keep looking at what else can we do and how can we improve that and what.

Are the things that we actually specialize in and we enjoy doing. And if we're able to have a conversation with somebody and deeply understand their wants and needs and then communicate that in such a way to a suite of AI tools that produces the output, then we're still adding value. And it's how humans continue to do the peopling, have that interaction and have the experiences that make us who we are while being supported by automation to make us more efficient.

Yeah, absolutely. I think in all the businesses that I've worked in, there's always been too much to do and not enough time. So just looking at how you can do more in less time.

So you mentioned gaming.

Yes.

What sort of games are you into?

Oh, I play all sorts. So I'm really big into, like, RPG games, sort of like adventure games where I can really get lost, get lost in the worlds that are created. I think one of the things I really love about gaming is I think it's a great blend between art and technology. Yeah, I just think it's cool.

And so as we're now seeing the capabilities to generate entire worlds, entire characters, the backstory for characters, the narrative and interaction to a far higher level through AI, what impact do you think that's going to have on future RPGs and these immense virtual worlds?

So I think one thing that we'll start to see is much bigger worlds, much richer worlds. So, as you mentioned, like, one of the things you can do is just get a great starting point for characters. If you look at some of the best RPG games, for example, some of the characters are really well written, and of course that takes a lot of time, but perhaps some of the side characters and characters in the background don't need to have a huge, rich backstory, and that's something that could be quickly thrown together, maybe have someone sort of go through and freshen up a little bit, but all of a sudden you've got all of these characters in the world around you that you maybe won't interact much with, but if you did, you see that they've got their own stories.

So the guy operating the weapon shop, who only exists on the moment you entering the weapon shop, actually having a life outside and a backstory that's real and changes, rather than just having the same narrative every time you go to buy a new weapon.

Yeah, yeah, absolutely. I think it just makes the world feel a lot more real. Like, if you consider in most games, the world is of course built around the path that the player is going to take, which makes sense to do that because you want to make sure that's where the excitement happens, but that's not real life. In real life, you're just part of a world. I think having more of that in games is going to sort of really add to the realism.

So do you think we're going to get to Ready Player One?

I think the Metaverse is a different topic, but I don't see why not. Because if you look at Ready Player One, I think for the most part, it's like a procedurally generated world. And of course, having AI tools for content generation is going to make that a lot more feasible.

It's a series of mini games that are games in their own right, isn't it? People look at Metaverse as being this big monster. Fundamentally, it's just worlds to get lost in, isn't it? It's just another game.

Yeah, yeah, absolutely. Just a bunch of worlds all in one place, sort of loosely connected. So I say, why not?

How soon?

How soon? I think I originally saw that the Metaverse was predicted for the. Well, for 2030. Who knows, it could be a few years before that.

That was while Meta was still investing in it, though, before they pivoted to AI. Was.

Yeah. It's hard to say, though, because the rate at which all of these tools are advancing is phenomenal. So I think it's hard to accurately predict what the future is going to look like three years from now, let alone by. By 2030. But I think if you look at those sort of worlds, if that's the direction that these tools do take, it does go towards a Metaverse, then why not sooner rather than later, I'd say.

And you only need to go to any of these venues where you've got VR capabilities already. All they really need is decent movement floors so that you can walk around without headbutting a wall or smacking your arms on the entrance. As soon as that's solved, then that technology can be Anywhere, can't it?

Yeah.

And you could have a setup in your small room where you're able to run a marathon across parts of Greece and never hit a wall. Why you'd want to do that versus actually getting outside and running, that's a different question. Depends on the weather.

Yeah, absolutely. I think one of the other big things is like the price point as well. If you look at like personal VR devices, the high quality ones aren't the cheapest, but another area that's booming at the moment is cloud computing. Everyone knows about how cloud technology has sort of exploded over the last five to 10 years. So as the power of cloud computing increases, I think what we'll see is a lot of the companies making these devices putting a lot more of that computing power into the cloud so that all you need is a cheap device that doesn't have a lot of hardware. All you need is a good Internet connection. And I think that will sort of bring a lot more mass market appeal. So I think that's something that's going to be required for these virtual worlds to really take off. But it's something that's happening.

Yeah. And we're already seeing Xbox go down that path where you had the super high end device that you needed in order to have all of the high quality graphics locally. And then bit by bit, bit by bit, they're moving more of that to the cloud so that you can have any device and still have that level of graphic and interaction. Maybe not with the same latency just yet, but it's getting close. Where you can game anywhere using this incredible compute in the cloud. Not worried about the device that you're effectively tethered to. So it's coming. You can see these sort of the different lines joining where you've got this AI ability to create new things, you've got the hardware in the cloud and the ability to effectively compute everywhere. The headsets for VR improving the rolling roads, if you want to call it allowing these various capabilities. We're not really that far off, are we?

No, no. The pieces are all there and they're all coming together. So I don't think we're far off at all.

So do you think we'll see people lost in these worlds?

Yeah, yeah, absolutely. I think there's a pretty big danger of that. Like when you look at Ready Player One, especially as the world around them has sort of deteriorated and you've got everyone that gets lost in this amazing virtual world and of course that's probably not great for society. That being said, I think there are some great things that it could do for society. Like you look at, how could you also factor in education systems into these worlds so that people across the world can get access to high quality education that they otherwise wouldn't have? But I think, as with all of the AI stuff, it's all very cool. It can be a bit scary. So there does need to be some regulation in place and some. Some guard rods to make sure that we're not destroying the world that we live in.

I saw a fascinating use case where it was a video of people in hospital beds who can't get out of their beds kayaking down a river with approaching rapids where they were just using tiny little hand movements and that was controlling this virtual experience where they were wearing a headset, they were outside, they could hear it, they could see everything. Obviously they couldn't smell the water, but it was giving them an experience that they couldn't possibly have had otherwise. And that for me is breathtaking. It's how do we enable things that aren't normally possible, not just support the needs of people to disappear into games and how you have that balance where we're able to use technology for good to improve the experiences of companies and individuals so they can get more out of life? That for me is the ideal.

Yeah, I agree. I think it's incredible. And you can't really put a price on that. Giving people a quality of life back that they otherwise might not have been able to guess. That's what it's all about, right?

Absolutely. Well, thank you very much for your time, Alfie. I'm sure the audience will meet you on many an occasion when we're talking with customers. It's been an absolute pleasure.

Thanks, Tim. And thanks for inviting me to join the team at Project Rising. I'm looking forward to seeing what we'll achieve together.

Wait to build some cool stuff.

Sounds good.