FDE Episode 7 : Software engineering benchmarks SWE-bench actually matter | Weekly Tech Update

Happy New year everyone. I hope the year is starting so well. So welcome to the code Ninja spaces. This is our seventh episode. So news one of the day is really interesting today. AI manages a tomato plant. So what happened is an experiment claimed that Claude managed a real world tomato plants by monitoring sensors, lightning warping level and checking the humidity temperature as well. And this wasn't a robot with arms, it was basically a software connected with real sensors. And so what basically a real question here is Omar, we really talk about this as well. That is it the beginning of AI operators into the real world and even with real systems as well.

So this experiment is still an experiment, but yeah, I mean, but still think of it. Yeah, I mean think it was like smart gardens. So they are trying to build those smart gardens. There are two main important points here. One is as you said, these are not the robots. These are basically software agents who are basically operating in the physical world. That's number one. So when you talk about software agents it means you don't need that level of dexterity that's expected from robot. So that's one thing that these are just software agents who are operating in the physical world using Claude. And the second part is which I think is more important from the standpoint of having that macro view of the overall thing, that's basically a truly closed loops system. So the acting, sensing data through sensors, then acting on it and then deciding all through software agents. That's a real breakthrough here. But still I would say it's still, I've read the news, still an experiment, but yeah, I mean looks cool. Yep.

So second News is about a 40 billion coding model has 81.4% accuracy on three benchmarks and it's verified. Our second update is more about software engineering agents. So a Chinese basically what happened at Chinese research group iQuest I guess claims that they have 40 billion coding model which is 81.4% on three benchmarks. And these benchmark uses are real GitHub issues they get and passes the test test cases as well. So does this mean smaller models are actually getting better than the giant model?

So there are lots of things here that we need to discuss and that's much exciting thing to discuss. Number one is with just 40 billion parameters the model has gained 81.5% performance on sweep benchmark. Sweep benchmark we discuss a lot about. It's a software engineering benchmark that this showcases like how good the model is in performing software engineering tasks, fixing bugs, writing software and things like that. With just 40 billion parameters this model has achieved 81.5%. That's a huge thing. But that's one thing. But there's another thing that I really want to discuss here. So there are two wars going on currently. I won't say wars, I mean one, I mean you know, in Europe, it's a conference we discussed about Europe's. I think one of our previous episodes, 38% of the work was coming from China. The most common language that were being spoken in that conference was Mandarin. So in terms of research, there's a very important thing here is like it's Chinese now whenever you see. So China is performing very good in reducing the size of the models in open source models and in that kind of thing because one thing that Eric Shmit told like America has a lead when it comes to AGI or it comes to real AI breakthroughs. China knows that they have, I mean they probably are maybe two years behind but they know that they, their stack is not that much complete. Although Jensen says like chips are basically a manufacturing endeavor and if someone says China can't manufacture that's probably not, that's probably he's not grounded. But anyways what I was saying is like as like America has a lead on these big models and China knows it. So China is not contesting that space altogether. They're focusing entirely on open source models, on making models smaller. And what happens with this is like the adaption that is coming by reducing the model size by opening source model is it just outweighs the penetration of American language models. That's the thing to watch out here. And I think that's one thing that our goal you know is like with American businesses, how to help make sure that we have America should have its own open source stack and that's a place that we need to contest. So that's the news here. So yes, I mean at one end I just wanted to, of course we didn't ask but I just wanted to give the complete perspective why another Chinese companies now has achieved this benchmark which is 40 billion parameters mark. And this is about this. Like all the people who are applying on a 16 in speedrun is basically using open source Chinese models. So imagine America's core AI problems like healthcare, logistics, supply chain are being solved on an open source stack that's being written in China. So yeah, that's what it is. That's a complete picture.

Great. So Moving on the news 3 is AI lens concept as fluid scripted, not as a fixed label. So this is quite deeper idea as well. What happened basically Adobe Research published a work framing concept as fluid structures, not fixed labor. They use the example of planets. Our definition that change when Pluto was declassified. So main thing here is that why should a Pakistani audience care about fluid concepts?

So I mean I'll take us. I'll take a step back from just what's the signal here? What's the news is so AI is. I mean I talk about that. This is a. Think of it like a dashboard. The human brain is a dashboard. It's designed for abstraction, it's designed for. To make sense of the world. Because we are limited by our training data, we're limited by number of neurons, the things we can perform right. Now imagine if I have. Just imagine if I have 100 billion neurons, what would happen if I have 100 trillion neurons, I can process, think about what can I process. I probably can see photon coming in. When you talk about structures, the complex structures, one way to look at is what physics tells us, right? And of course that's one way to look at it. So think about, I mean Pluto example. For a very long time scientists thought that it's a, it's a planet. And then later they reclassified it is as a dwarf body, which is not planet. Right. In its core sense. So basically the idea is like can I mean imagine if we had an AI at the time and we were with 100 trillion neurons. That's it. So AI would have been figured out that this is not a planet. So understanding structure of the bodies, understanding structure of the planet, understanding the physics engines of the word. That's an area we need to think about. So you. That's one thing that I want to say like why AI is good in, in understanding the physical world, understanding the structures of the world, understanding quantum gravity and things like that. But think about it like there's a different thing. I want to mention it. You talk about Pakistan. I would specifically talk about for software engineers, Word is a giant code. I mean historically software engineers focused on, focused on automating tasks that generally humans used to perform. Right. I mean the whole idea of computers, computers was those people who used to do those math calculations. And then there was. Then that's. That's coming on. So basically we were replacing humans ever since we've also so much task to the machines. So that was the idea. But now since human brain is entirely automated. So what's next for the software engineers? Let's say I've been talking about this like we need to think about world as a computation engine. I mean one of the guy who inspired me a lot was Stephen Wolfram. How to combine the concepts of computer science and computation with physics. So for example, weather has its own computer, right? Space has its own computation, fluid dynamics has its own computation. So I think in the years to come, software engineers, Pakistani software engineers, this part they would have to interact a lot with physics based engines. So what does it mean to use physics based engines to understand the natural world? That's the kind of computation that we need to think about now that how so the raw automation for example, what can be automated like for example building automating a logistics workflow or supply chain workflow, things like that that probably in next few years would be entirely outsourced to the machines. They're good enough currently of course machines are very non deterministic in nature. However we bring the infrastructure on top of it that would bring that determinism into those systems. However, I think the next endeavor we should focus on is like how to understand the natural world by applying the same computer science lens. That's the area that I'm passionate about. And that's what you, when you talk about like would AI be able to understand the physical world, the structures that we have. That's something that I think we should be thinking about.

Amazing. So News 4 is more about space manufacturing. A startup reportedly run our fundraise in orbit reaching around minus thousand Celsius. And they claim that. The claim is that microgravity and vacuums can produce far more greater semiconductor.

It's not minus thousand like it's 1000 degrees, it's extremely hot.

Okay, got it, got it. And they claim that microgravity and vacuum can produce far more clear semiconductor crystals. So this means that why manufacturing anything in space?

So there are a couple of things here. So number one, think about like orbital manufacturing is that I think we've been dreaming from 2003 onwards. We talked about orbital compute, but there's orbital manufacturing. So why someone has to be so like think about like so crazy to start manufacturing plant in orbits. So this company, I mean the business case is like there are two things. Like achieving 1000 degrees Celsius in orbit is not an easy thing to do. So that we've already achieved that. But what does it imply? So the goal is, I mean think about manufacturing a semiconductor chip in space is far more efficient because the impact of gravity is not there. There are far less defects when there is very little gravity. What they call microgravity. So think about the scenario where gravity in on earth is not letting us build high quality pure Semiconductor crystals or wafers in space because the impact of gravity is low. So those microgravity environments are suited for manufacturing the purest of the materials like semiconductor crystals. And to achieve that, the one of the major breakthrough that was required was achieving extraordinary hot temperatures in space. And that's what this company has achieved, at least from the perspective of the news.

So the news 4 is more towards the BCI interface. Brain computer interfaces move towards production. So but update is basically from the neural links. Elon Musk says that Neuralink is targeting high volume production in 2026 with more automated implantations. So is this the start of mass market production of brain interfaces now?

That's what they are planning to do in 2026. But just take a step back from here also. I mean what's the implication of this? So Neuralink is so Elan spoke a lot about it. Like currently if I have to communicate with the machine my bit, my data transfer is limited by the bandwidth which is like I can transfer 10kb per second to the machine through these meat sticks as Elon says. Right. And that's, I mean, so we are limited by, I mean think about like we talk a lot about why the system, why someone is not learning fast enough. These are things that for example, we talk about a lot. So Neuralink has a capability of at some place 300 MB per second. It's very high. So that's one thing that the bitrate with machine we're going to be very, very high Also for example, you know I use a lot of voice AI so when you talk to machine using voice, I mean, I know, I mean how much faster typing speed you have. You can't in 10 seconds. You are limited with both in terms of your quality of thoughts and in terms of how you write, limited by the, I mean you can't write probably a paragraph in 10:20 seconds. But I have seen that I can write a complete paper using voice in 10:20 seconds. So it's the same goes to think about if somebody's directly implanted into the brain. So one thing is like our connection with machines will be much faster. Second, the core part currently is like treating some form of disease. For example, if someone think about, if someone is paralyzed they're not able to use anything like not able to use computer and things like that. So that's I think a major breakthrough. But yeah, I mean I doubt that we probably will achieve in 2026 in mass scale production. It's not that easy. It's extremely thin. I mean it's really Visible to a naked eye, that small list of the wafers that we implant. So there are lots of challenges currently in deploying vci. However, let's see where it goes.

So the last news of the day is more like a philosophical closure. So the question is just growth reduces the existential crisis or risk maybe technology growth can take. Us can resolve the challenges like that.

So there's a recent paper by Philip Tremor and Leo Aschenberger. They are trying to understand this philosophical question. What does it imply would AI for example would solve the problems or would cause risks? So they discuss both things. I mean, yes, we'll be able to solve some very important problems, but there are some real existential, I want to say existential, but there are some very important risks that we need to think about. One of the things that I talk a lot about is I think is like job displacement. It's very real and it is coming currently. What I see is like especially for example, think about software engineering. If your job is just to, I mean if you're just coding, that's, that's not something I, I'm finding it very hard in the next two years someone who's just doing. It's. It's just like, I mean that job won't exist anymore. Probably it would be Software engineering was a made up role. I mean the name wasn't made up. I mean what other role someone would have thought about like what I also think like what the new role would look like. If we have to give a name to a role that, to a guy who does lots of things, one of which is coding, what name would we give to that role? That role is yet to be invented, yet to be discovered. But I think it's very real. Job displacements I think is Israel. And it's. The thing is like in historically there was. There were major revolutions and job got displaced, but cognitive labor replacement that we need to seriously think about it, how to, how to go about it if you're not doing 10, 12, 15, 20 things at once. So I talked to software engineers. If not for example, learning, not writing software, not doing, creating content, not doing sales, not doing marketing, if one guy is not doing all of that stuff, it's hard. Something we need to think about is like, I mean I know if you're doing let's say 8, 10 things in parallel, for example, you're doing product engineering, you are creating content, you're doing sales, you're doing a lot of different stuff. This level of generalization, probably it's harder for AI to beat this level of generalization and tool use, but on a specific task, we've just discussed it. With just 40 billion, 40 billion parameters, a small model has achieved 81.5%3 benchmark. I can guarantee asking software engineer to achieve 81.5% benchmark in a general task on the same task. So, yeah, I mean, but I think I'm more hopeful, more optimist, I'm more. I have more optimism for the future, probably. I think we'll figure it out.