AI vs software devs

Foreign. Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI related tech is changing the world, this is the show for you. Thank you to our partners at Fly IO, the home of changelog.com Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents so you can launch your app near your users. Learn more@fly IO. Hello, Jared Santo here, Practical AI's producer and managing editor of all the shows here at Changelog. Daniel and Chris took this week off, but we didn't want to leave you hanging without anything to listen to. So today's episode is going to be a little different than our usual fare. AI is permeating the entire software industry, so we found ourselves talking about its impact, sometimes in practical ways, other times in less practical ways on many of our pods. So today we are serving you a sampler platter. You'll hear a segment from this week's JS Party podcast where me and my two co hosts Kabal and Nick Nisi discuss the recently announced Devin project, which is making waves in developer land. You'll see hear a segment from a recent GoTime episode called How Long until I Lose My Job to AI? Where Johnny Borsico and his experienced panel of friends discuss using CodeGenai to augment your dev skills instead of replacing you. And finally, you'll hear a segment from the Changelog, where my co host Adam Stokoviak and I talk to Jose Valim. Jose is the creator of the Elixir programming language and he's been a guest here on Practical AI in the past, talking about Elixir AI tooling. Today you'll hear us question him regarding Elixir's platform place in a world increasingly influenced by large language models, and how he thinks about it as a language author and promoter. Hopefully there's a little something for everyone on this episode, and if it's not approaching AI from a perspective that's compelling to you, don't worry. Your regularly scheduled programming with Chris and Daniel will be back next week. Okay, first up, it's jsparty and Devin. There's some news that's good. There's also some news that's maybe bad, maybe good. I don't know. The thing that everybody's talking about this week at least as we record, and last week as well, is Devin. Devin D E V I N Devin, the first AI software engineer, according to the makers of Devin, which is Cognition Labs, a new company which raised a Series A led by Founders Fund headed up By Scott Wu, who seems to be a very intelligent person even from a young age. If you watch that video of him doing math very quickly at ages when it seems like you shouldn't know math very quickly. And they got a demo out there of this new AI software engineer. So I could say more. I'll stop right there. You all have probably seen the demo, Cable and Nick, or at least heard about what's going on. This is a new tool which can start from scratch and do some cool stuff. I'll just leave it there for now.

We can talk about the details if you're excited. You too can pay for the right to have a software engineer that can only fix 1 in 7 of your tickets and spin up lots of new ways for AWS to charge you money without your oversight.

Sounds like an intern. No, just kidding. Sounds nice. What are you referring to? Or is this some specific things that Devin's been up to so high level?

There's a couple of things that I'm referring to here. So one is like they're pumping up the marketing on that. This is a standalone software. Why get a coding assistant, get something that can go and do your software and they publish some data on it. And it does do better than the state of the art in terms of tackling going from a GitHub issue to okay, I'm going to actually solve this, implement a change and get it to happen. But the number they published I think was 13.86% of issues unresolved. So that's about one in seven. So you pointed out a list of issues and it can independently go and solve one in seven. And first off to me I'm like, that is not an independent software developer. And furthermore I find myself asking if its success rate is 1 in 7, how do you know which one? Right? Like are the other six those it just got stuck or is it submitted something broken? Right, because if it sets up something broken that doesn't actually solve the issue. Not only do you have it only actually solving 1 in 7, but you've added load because you have to go and debug and figure out which things are broken and you have a whole bunch of additional load. So I think the marketing stance there is a little over the top relative to what's being delivered. The other thing, and this is around, I think a part of what they do is, oh, it can spin up resources for you and they show this cool demo of he pointed at this thing and it allocates a bunch of different production resources for you and the person who's handled DevOps in me before and now the engineering leader who has to sign off on our DigitalOcean or AWS or Google Cloud or whatever expenditures you might have, looks at that and is terrified by. I'm going to give an LLM, which is known for hallucination, which is these things are not. You have to design applications and I'm building applications with LLM, but you have to design around their unpredictability and their willingness to lie. And I'm going to give that raw access to spinning up resources in my cloud. It sounds like something I would not sign up for. I'll say that.

Okay, Kay Ball, let he whose success rate at issues that is greater than one in seven cast the first stone.

Yes, I was wondering how what Nick's ratio is over there. One in seven sounds about the way I would do. I'd pull off the easiest one first. Does Devin know what the easy tickets look like? Because that's the skill right there.

I'm over here counting on my fingers, trying to see if I'm within that ratio.

But do you know when you fail or do you just throw out broken code and you're like, here you go.

It's more of a question of do I know when I succeed? I guess.

Right.

Which is. Yeah, same thing.

You think you succeeded until you find out later that actually you failed. That's been my experience. Or you succeeded under the constraints that you put yourself under, or that was actually specified in the ticket itself, but you actually failed at some other unn unlisted constraints that were unknown at the time but are obviously clearly there in production. And so in that context, you failed. It's not easy. It's not easy to succeed in this world. Well, what about cable? What if you can't? You point Devin at like A$5 a month DigitalOcean and say deploy to this. And can't you cap your risk? I guess on the DevOps side, probably you probably can.

And I do want. So I'm taking a hard skeptic stance on particularly the claim that this is an AI software engineer. Like don't hire a person, use this thing.

And this is their claim. So I think it's fair for you to be that harsh on them because they say, meet Devin, the world's first fully autonomous AI software engineer. That's a very bold claim. So I think it's fair that you're being that harsh. Go ahead.

They're showing some cool stuff. It looks like a pretty interesting tool to put in the hands of Someone who knows what they're doing and is able to validate it, and is able to say, okay, go and solve this relatively well constrained problem where I can easily validate the correctness of your output. Go at this sandbox where I know that you're not spinning up massive amounts of resources in a way that I'm going to regret or even go at this non sandbox situation. But I have the knowledge to check what you did, look at the logs and be like, yeah, that's okay. Those are really cool things that could be really valuable, that could dramatically increase somebody's productivity. And those are so far from being something that I would trust independently to replace a software developer that they're not even in the same country. Like maybe not even in the same world. Like these are just completely different claims.

Yeah, I think that the sensationalism of this comes from not what it can do now, but what it represents and the progress that it's made when comparing to other things, like whatever it was, comparing that 13% to, to other AI chat things that can do things, it's way better than all of those. It still sucks compared to a human, but it's made like monumental progress in terms of AI. And I guess the question is, does that continue? Can it get further than that or will it reach some kind of limit? And then the other piece of it, I think just from a marketing thing, and I'll be honest, the only thing I've seen on it really is a fireship video, is that it's already doing some work on upwork. So in a way that's a marketing claim, that it competes against real humans for jobs.

Truth, according to them I haven't confirmed. But what you said is true, that they say that? Yes.

So this is the struggle with all of the LLM world right now and all of the AI world because on the one hand we have been in a place where we're in the rapid part of an S curve. There have been some very rapid advancements in the core capabilities of these things and they are super freaking cool, like really cool. And also they have a lot of limitations. A lot of those limitations are baked into the architecture that's being used. And so you get kind of a situation where like there's a bunch of people doing really cool stuff with this and like figuring, trying to figure out what it's good for. But it demos way better than it does anything reliably in production because you can get a really cool outcome. You know, 40% of the time, some situations, 70% of the time. And like, you show that and people are like, oh my gosh, this is going to take over the world. And I would not trust a, for example, AI software engineer that even that could handle 70% of my tickets. But 30% of the time spins up millions of dollars of costs for me. Right? Or like other things. And once again, like, I'm not trying to take away from the technology, but I don't think these hyperbolic claims actually serve anyone except for getting attention. They get attention. Okay, great. And you're going to get a whole bunch of people who buy this thing are disappointed. If it costs them a bunch of money, they'll sue your ass off. And like, why would you do that to yourself?

It's somewhat similar to Generative AI in the image. Let's just stick with like static image world where everywhere you see is impressive results. And they'll be like this new, like Midjourney 7 is off the charts. Amazing. Here's nine examples that'll blow your mind, right? If you click through on that, they're all going to be very impressive. Like, those are amazing things. But then you have to stop and think, well, midjourney didn't create nine examples that blew my mind. Mid journey probably created 40, 50, maybe 500 examples. And then you, human decided which ones were amazing and you cherry pick those out as the examples. And like, that's a great team work, guys, right? Computers plus humans equals better results. And so there's the cherry pick. And that's the. I mean, that's what code review on these things will be. That's what happens when you tell copilot, no, I do not want that function. Right? It's all as Hipster Brown calls it in the chat room, human in the loop. And that's exactly what is necessary. And I think the reason why you call them hyperbolic claims K Ball is because they're saying it's a fully autonomous AI software engineer, human out of the loop, let it rip. And maybe fans of the bear will like to say let it rip. But those of us who aren't fans of Devin are thinking, let's not let it rip too much because it might just tear the whole thing down. Now I'm being hyperbolic. Nick, you're not in the long. Do you agree with me somewhat?

Yeah, I think, like, yeah, it's humans who are deciding what is good out of that and kind of helping to train that going forward. But in a way, like I was trying to think and trying to relate this to another article I saw that wasn't about Devin specifically, but was about, like, prompt engineering as a profession being taken over already by AI, because an AI can iterate and more quickly come up with a way to answer the questions that you want by appending exactly what it wants to hear at the end of a string. And I think the example that I heard from that was like, we want you to answer this question, and the AI is incentivized to answer it a little bit better if you put it into a scenario that it likes. So the AI is Captain Kirk on the Enterprise, and it has to answer this question to save a planet from whatever. And the question could be, what's two plus two? Or something really simple. And by putting in all of these extra prompt words that the AI is coming up with on its own, it's making better results overall. And I'm just wondering how that marries to the idea of humans being the ones who curate the good ideas that come out of it.

Well, prompt engineering, I've been convinced by Swix that it's a code smell. Yeah, like we. I was at first, I was convinced, like, this is the new thing that everybody needs to learn. And I think it's just the leaky abstraction that we're currently dealing with as humans because the tooling's not good enough so that we have to engineer the prompts. I mean, Google's search box is prompt engineering.

Right?

Like, knowing how to Google, it's the exact same thing. It's just way harder and it's like way more magical now to tell it the magical incantations to get the best results back out. And so the fact that it knows what results are better to me, is not intelligence or anything. It's just like, we just need that to go away. And I think Devin's actually an example of where they've productized and hidden a lot of the innards that we've currently been exposed to in order to make the tool work better than it would for an inexperienced user to use it. Like, they've actually turned it into a product. And I think that's great. I think it's one step on a long line of iterative improvements that will make it. So the prompt engineering, I mean, you're just going to basically talk to it in layman's terms and it will know how to feed itself the correct prompt, so to speak, in order to get the. To get the goodness out. But I don't know. Cable back to you.

Yeah, I mean, I think so. The high level on all this AI stuff is like, there's really cool stuff there. We're figuring out how to use it, and the current state is clearly intermediate. However, the thing I want to keep coming back to with this is there are things that it's like, okay, this technology is immature and we're going to evolve around it and figuring out how we handle prompts and managing prompts and what's generating them and whatever that fits well in that bucket. And there are things that are fundamental pieces of the way the technology is designed. Right. LLMs, machine learning models in general are statistical, probabilistic. They're very different than most things you think about in software, where you're trying to make something that is logical, consistent, like you could put A in, you get B out, and that is not there with these things. And so you can design applications around that. And there are things that you can do to sort of pin that down, to add validation that is outside of the LLM and do other things. And maybe Devin is doing that. But I think the more we start looking at these sort of places that require judgment, places that require precision, places that if you just make some random up, it can cause a lot of problems. Those are not actually. There's a fundamental thing about what the technology does that means it's not necessarily going to be a good building block for that. And so making hyperbolic promises about where it's going to develop that depend on it being a fundamentally different technology than what it is, feel like they are setting yourself up for a lot of heartbreak.

What about the job market? Do you think it's fundamentally affected by tools like Devon as they progress over the next three to five years? Because we're not talking about humans out of the loop. I think we're all in agreement here that that's not feasible or smart, at least in today's technology plateau of LLMs. But less humans in the loop, you know, that seems like it's very feasible if these tools continue to iterate and even just not have revolution, but evolutionary advancements from here.

Yeah, if it makes me three to five times faster, do we need three to five times fewer engineers?

Yeah, I mean, I think there is. So this is a technology that has the potential to dramatically impact the productivity of software engineers. And I think there's a couple different things around that, as we think so short term that can create some disruption. Right. Short term, that means that a company that had been running on, say, five engineers and might have needed to hire and expand to 15, now they don't have to expand nearly as soon as and things like that. So I think there is the potential for relatively short term disruption. I will say both the history of economics broadly and software in particular is that every time we make it easier to code, we discover there are whole worlds now that we can address and build software around that we couldn't before. So if for example, and this actually there's a particular example of this that I think is interesting to dive into. So one of the big economic challenges in the tech industry in the last four or five years is that we had these massive tech companies with incredibly high revenue per employee. Google Meta, Netflix, like the faangs mostly right. And so they were able to set this salary bar that was super high. They were paying ridiculous amounts of money, that's a technical term, ridiculous for software engineers. And then when we had very low interest rates and a ton of VC money flowing in to the industry, there were lots of companies whose fundamental business economics do not support that level of salary per software engineer who were nevertheless paying that amount of salary per software engineer based on VC capital and sort of this thesis that okay, we'll be able to scale out of this and we'll get whatever. And I think that caused a lot of distortions and problems in the field. Now if suddenly software engineers are three to five times more productive. The range of businesses that could use software but previously could not afford to compete with the FAANGs, et cetera of the world. There's a whole set of business models in there that become viable because it's that much cheaper to develop software. And so I could imagine this actually dramatically expanding the number of viable either software businesses or businesses that are non tech but want would like to include software or could have custom software and dramatically expanding the number of those that happen. So I think long term, I don't think it's a negative impact on the software engineering career path. I think that there's what it means to be a software engineer looks a little bit different when you have different types of tooling. That has been true as long as I've been around. Like I just JavaScript land, right? I remember when jQuery was a revelation. Oh my gosh, this made me so much more productive. It did make me so much more productive all these other different things. And now the level of tooling that we have there that supports our productivity building things in the front end is astronomical and has that taken away from the number of people writing JavaScript.

Speaking of astronomical, Astro has a new database.

What's up friends? Is your code getting dragged down by joins and long query times. The problem might be your database. Try simplifying the complex with graphs. A graph database lets you model data the way it looks in the real world instead of forcing it into rows and columns. Stop asking relational databases to do more than what they were made for. Graphs work well for use cases with lots of data connections like supply chain fraud detection, real time analytics, and generative AI. With Neo 4J, you can code in your favorite programming language and against any driver. Plus it's easy to integrate into your tech stack. People are solving some of the world's biggest problems with graphs, and now it's your turn. Visit Neo4jcomm/developer to get started again. Neo4jcomm/developer. That's n e o4j.com developer.

Up next, we have a go time podcast Fireside Chat between longtime programmers Kent Quirk, Sharon Diorio, Stephen Pyle and host Johnny Borsico. Foreign.

It'S one thing to have gen AI, you know, pump out snippets of code that is part of a larger whole, right? Whereby I'm the engineer, I am engineering a solution, not just, I'm not just a code monkey just clacking out, you know, syntax. I'm trying to fix a problem. I'm engineering a solution to a business problem. Now I could go, you know, as high level as I can. I can just open up, you know, copilot in chat mode and say, hey, this is what I'm trying to accomplish. Start spitting out, you know, files right now, maybe today, right? It can build somewhat trivial apps. I've seen YouTube videos and clips and things of it spitting out entire working react apps and all these things. And that's, that's great. And I think over time it's going to come even better at doing those things. But I have a hard time trying to sort of correlate that or trying to replace solution building because to me, solutions aren't static, right? When a business comes to me and says, hey, I need you to build a solution to this problem, I build it, they take it into production, they do stuff with it, and they come back and says, hey, you know what, this is great. Now I need to change it in this way, or I need to account for this exception, or I need to account for this particular use case or this specific customer, where 90% of the time it works like this way for every customer of this type, but for this customer of that type, on ultimate.

Thursdays during the full moon, it does this completely different thing.

Exactly.

So now what am I Like, how are we supposed to treat those entirely made up solutions that am I just feeding that back into the system and saying, hey, so now account for these alternative approaches.

Is it going to be like it was when the first generated code frameworks started hitting the scene and you'd go in and there'd be all this code. It was like, yes, super fast. If you had to do an orm, it wrote all the code for you, et cetera, and then you needed to change something and all of a sudden it was like, you know, change management in some of those days was yeah.

Or regenerate the whole thing from scratch. And oh, sorry about all your customizations.

Yeah, so that'll be another big test that AI has not yet proven it can do well.

So let's talk about art for a second though, because this is again a similar thing. Like, everybody's really excited. Look at the images I can generate with, you know, Mid Journey or whatever, stolen art. Well, right, but, but the point is again, it's going out and giving you the average solution. It's going out and going here, here are the things that look most like what you described that are somebody else has created already and kind of going to cobble pieces that together.

Or here's an opinion formed by the loudest voices out there that I sucked up as source data.

Hell yeah. But like, I sat in a, a meeting today where an artist went over her design, basically her design process for a big design project. Like, here's the resources I looked at, here's the, you know, the feeling I was going for. Here are the things I considered. I looked at these typefaces, this typeface reminded me of this, you know, building architecture which is relevant to the site, you know, and then that artist proceeded to churn out over the course of a couple of months, 200 pieces of support art for an event that was a brilliant design exercise by somebody deeply steeped in art and creation, who then studied the event and what the event needed and integrated all that. And yes, some random person could have sat down with midjourney and said, make me this stuff, and it would have been much less good. But people who don't know the difference would have been, sure, it looks fine. You know, I mean, we've all seen that, right? You know, my, my document has 37 fonts and 12 colors, but it looks fine to me. But like, there's a big difference between, you know, something crafted and something just slapped together. And, and yeah, I guess I think that AI is going to make it easier to slap Together.

But for most people though, would you argue. So here's what I'm not saying. I'm not saying that these things generated by AI, like if you're a konoiseur of a particular art, you're an architect of a particular kind of application or solution or thing, whatever it is. You can critique the output of Gen AI as it stands today. Again, arguably it's going to get better at what it does, but you can critique the output today and be like, this is subpar.

Right.

This is not as good good as what I could have come up with, but for most people it's good enough.

It depends on what they're using it for. And so again, if you just doing something for yourself, who the hell cares? I mean, yeah, I'll slap something together out of two by fours. If I'm building it for my garage, I don't care. But if I'm going to sell it, if I'm going to make a business around it. That's the part where I'm saying I don't think the AI stuff is there. If you're just doing a hacky project for personal use. Yeah. I mean, maybe you would have had to pay somebody to come in and slap that shelf together in your garage if you didn't have the skills to do it yourself. And so now there's this kind of, you know, yes, there's a few things that I couldn't do before that now I can do today for myself. Design that invitation for my kid's birthday party. Hell yeah. I can't draw, but I can use an AI. That's. There's nothing wrong with that. And yet, you know, so yeah, there's probably some, you know, the kid next door, you're not paying 20 bucks to, to do that for you.

But that's now what happens in, you know, in the future as AI evolves and improves or you know, we get to this uncanny, uncanny valley level of like, oh, now it's not just good.

Enough, it's like, it's like the standard now it's building your whole kitchen.

Right.

See, do we need to worry about.

That and how much coding out there is most of what's out. You know, how many cruds have we ever created in our lives? How many cruds are still being created every day? Yeah. Okay, so that's a problem that's largely been solved greater or lesser degree, but.

Yeah, I mean, it's white box, Right. You make white box easy to do. Yeah.

And tech has always been making things that had gates around them or real. Or created limited availability or. And making them more available, like artisan things that used to be only certain artisans could do that. Only musical artists had a studio and an audio engineer. And now they can go and, you know, create their own tracks with one app at home. For me, that's the part that's like, oh, I can't complain. I can't be the curmudgeon complaining about this latest thing that might make something that I do more accessible to other people. Like, I've benefited from these other things that came along. It's time to share the wealth. I don't want to. I'm really rooting against it.

Yeah, I don't think it's about gatekeeping. I mean, I'm not like, I feel like a big chunk of my career has been spent trying to help people learn to program. And so I'm not thinking that. The reason I'm skeptical is because I don't want other people to do what I do. I think it's more because I feel like the hype and the reality are distinct, that what the reality is producing is mostly devoid of creativity. People are confusing knowing what to look up with, being creative. And I think knowing what to look up is a skill and a lot of us have it. And the better engineers, I think are better at it. And so yes, AI helps to ease that problem.

But.

But knowing what to even ask about, you know, like, or looking at a new solution to a problem, that's something that I think is well beyond what's. What AIs are capable of now or in the reasonably like the LLM model, I think is fundamentally non creative. That's my take on it.

Spicy. We're not at the unpopular opinions yet.

No.

So one thing you mentioned, like the whole teaching, like y' all remember when maybe it was during maybe first or second Obama term or something, but there was this giant push to teach everybody how to code, right? Like it was, it was everywhere. It was in the media, it was in newspapers, it was like, you know, we need to teach our young how to program. Now I'm looking at a clip from, you know, Nvidia CEO like three or four days ago or something saying, hey, people shouldn't learn how to learn how to program. You should now let you know the new programming languages, human language. And I'm thinking, man, you are sitting here, you stand to gain billions a bajillion dollars, right? If your wish comes through, you know, because you're producing, you know, chips and stuff for these Things. Of course you're gonna say that.

Right.

So this, this.

So.

But I mean, what. I'm definitely not on the don't teach people how to program camp.

No, no, I'm gonna, I'm gonna take a slightly spicy take here. I don't think he's. Now, in the time we've all been engineers, we've seen waves of different things that are going to come and take our jobs. Offshoring and code generation. Now AI, and they haven't. And my theory is that the key thing that an engineer has is the ability to communicate. And even when you're supposed to be communicating to people on the other side, product, the business, whatever affectionate term you use for them, aren't always as good at that, although it should be part of their job. But having somebody who can think back and forth, there will, I think, always be a need for those people. Because every, every CEO thinks they have the answer to every question.

That's what they hate to do.

Right.

But they really shouldn't. If they have a business that's big enough to grow, their biggest skill is finding the people and put in the right place. So if your job right now is like doing creds for a company that can't even explain what they want, I wouldn't worry because they're not going to be able to explain what they want to AI.

Right.

Yeah. No, it's the thinking logic. Yeah, there's the think about. Break something down into steps and think logically. Like, I once did have a client very early in my career who was a pretty good business person who really wanted to automate his business. And he was able to sit down and explain it to me. Like, if he had had the tools to program, he could have written his own code because he thought about it really logically and, and it was just my job to basically take dictation and turn it into Pascal for him back in the day. But, but that was. That's few and far between. You know, quite honestly, most people who specialize in business aren't specializing in thinking logically. They're specialized in thinking about people. And like you said about communications.

So does AI then make you more what you already are? If you're a logical thinker, you'll benefit, and if you're not, you still struggle.

And who gets to train the agent?

I want to be in the training side. I want to be the one doing the building of the things that you use.

Right, yeah, it's a good question.

I think at the point, when it comes to Like a personal AI where it's just like, it's tuned to you, your data doesn't get shared, it's just, you know, then it becomes like a superpower.

Right.

You can just, it's like your copilot.

Right, but how would that work if it's only got your data? It's basically replicating to a point.

You, there's a generic, it's trained on the universe and then specialized for you.

Right.

Is the way that we're seeing all.

This, it's like creating your own GPT, but it's based on the larger model, right?

Yeah, exactly.

My company, we have a query engine that looks like SQL and then we built an AI where we train that AI as part of the prompt. We basically can go out and get your data and all the names of your fields and the data types of your fields and we can plug them into the AI query so that when you say, show me my slowest service, it can go, all right, what are the fields that are named according to time duration and what are the things that look like a service name? Now I can write a query for you so it knows how to query honeycomb and then it can write that query for you from your inept prompt because it's been specialized for that particular type of application. And I think that's a really cool use of AI.

That, that is, that is a productivity boost for, for people who are already technical. Like, you know, like, full disclosure, I, I am, my startup is a honeycomb customer. So, you know, I, I, I will go in the dashboard and I can formulate those queries, but I'm already a highly technical user who knows how to use these kinds of tools to get to and know exactly what kind of data I'm looking for and when I've found it. Now for the layperson, right. Who doesn't know, like the layperson, like, the more I think about it, the more I'm thinking, okay, if I'm a, on one end of the spectrum, you have the complete layperson who is using perhaps, you know, Chad GPT or something like it to maybe generate copy and not hiring a copyright person like you might traditionally do back in the day. Right. I'm sure the copywriters of the world are suffering right now because content creators, you know, like, content is like, yeah, content creators are suffering because this stuff is now being generated.

So are all our Google searches.

Right, Right. So if, if that was your job, absolutely. You're impacted.

Right.

And the layperson can now bypass you and get to something that again, the good enough. Right. They can get something good enough to achieve some means, Right. On the. The complete opposite end of the spectrum, you have people who engineer software. Right. That again, given the context of the conversation we're talking about, like, how safe is our jobs, right? And so when I'm asking this question, I'm not asking is the layperson going to find ways of reducing.

Right.

Their reliance on sort of, I don't want to say lower skill, just the different kind of skill. Right. I'm thinking, like, for people like us as software engineers who presumably will be impacted by this to some degree. Right? And we already are, right. For us, there's also the micro spectrum, whereby if you're on the sort of the lower end of that spectrum and if the only thing we are doing is generating crud, well, I'm sorry, your job is indeed in jeopardy if that's the only thing you've been doing right. With your career. On the opposite side of it is the highly specialized person who understands a business problem has to debug and troubleshoot and talk to people and integrate different.

Things and institutional knowledge.

Right. All that stuff. I mean, I don't see that skill, Right. I don't see that being replaced by AI anytime soon. Am I wrong here?

I don't think so, personally.

How many of those people do we need?

And that's the thing, right?

Not musical chairs. We should be looking for our chair now.

No, Any business is thinking, do I need a thousand engineers? Right. When 500 will do?

I mean, I think, as we've all said, right. It makes us more productive today. So I'm writing more lines of code per day than I was five years ago.

Right? Right, right.

So that's good. But we all kind of expect productivity to continue to rise. So this is a productivity tool, not a replacement tool. You know, we're also using, like, using languages that are more expressive than they were. You know, like the code I write in GO is probably one third the length of the same code I write in. In C or used to write in C back in the day. So that's also a productivity boost at some level, at least if you believe the old metrics that it's basically, you can write the same number of lines of code per day no matter what language you write it in.

But I think actually knowing how to use it is a skill to put on your resume. But not before too long or if not now. I mean, even if I didn't want to use it, I would because it's becoming of like it's going to be a point where like, oh, you know, I use Copilot all the time. Oh good. You have the point for you to.

Get the job is prompt engineering on already and you'll make it profile.

It's going to be soon.

No, but if you put it in your interests as AI, it shows up in the keyword searches.

There you go. Right, Nice.

I mean that's a good point. But it, but to me it's like back to my woodworking thing. It's like I know how to use a power saw, I know how to use a drill press, I know how to use lathe. You know, those are, those are kind of expected today. If I'm gonna do woodworking and say I only use hand tools, people are gonna look at me like I don't have time for you.

Same thing. Yeah.

And the same is true of like if you're not using Copilot, what am I paying you per hour?

What are you doing?

You know, why are you not, why.

Are you not as productive as you could be?

Yeah.

I think at this point the only people who really aren't using, using it are people who are doing like very arcane languages or people who, their businesses, they don't allow it, their company doesn't allow it. I think everybody else has at least tried it.

I mean if you're a company that doesn't, that doesn't allow such things, like I understand not bringing sort of open source code into your organization, that might be the wrong licensing model for you or something like that.

Right.

You don't want to be in some hot water. All you have to do is look at, you know, Oracle and Google over the whole Java thing. I think those were the companies involved. But if you allow your engineers to use a model where you can control the kinds of things that were used in the model. Right. For the training and you can have, maybe you can run your own internal gen AI for code generation, whatever it is.

Right.

I think if you're an organization that is afraid of these things, you should at least follow that route as opposed to saying, hey, nobody can use any gen AI coding tools whatsoever. Because I think you're going to lose people if you do that. Because, because I'm going to look at my peers that are, that get to use these things and are they, they're learning those skills.

Right.

And then now I'm falling behind because everybody's using, you know, some sort of co generation tool and I'm not. Right.

I mean this is where I hope somebody Reaches out. Somebody who hears this and reaches out and can answer that question of like, can you have a copy of Copilot that you train on a specified set of repos and only those repos, private repos. Well, I mean, I would, you know, as we joked, I mean, if I'm a go developer, I'm training it on Johnny's code because if I don't like it, I can go with Johnny, but nice, you know what I mean?

There's people out there, one neck to choke. I get it, I get it.

It doesn't matter what language it is. There's people out there that you respect their code and you'd be like, yes, I would like my code to be more like this. I wish it would learn that. That's what I was thinking. Or that's the way this problem should be thought about. I mean, yes, there's precious few of those people and those people will probably never lose their jobs, but for the rest of them, you're mortals that are going to have to work with the tools that are out there. And I would love to, if this is a possibility now, that you could train Copilot on what you'd say to train it on. And not all of GitHub, I think.

There are companies that are working on that product. I feel like I've even seen a product announcement like it. But yeah, I mean, you know, the thing about it is you can take one of these LLMs and you can essentially subset and you can make a tiny compact LLM that will run in a box that you can actually stand on your desktop and then you can further train that with new information. So that's exactly what you want to do here. You want to take a coding centric LLM like a copilot and create the mini version of it and then train it on your repositories. And now it knows how to write your code. And it's also not talking out to the cloud while you're doing it. So there's got to be businesses like that, if there are.

And this is what we redact all of this and put it into our business plan.

What's up, friends? There's a new book out there called the Hacker Mindset. This is a productivity cheat code to unlock new levels of success in your career, in your creative pursuits, and in your personal growth. This book is about leveraging the principles of white hat hacking and applying those skills to the broader world. It's available for pre order right now and it's not your typical productivity guide. This is written by Garrett G. A seasoned white hat hacker with over 20 years of experience. This book reveals the secrets of hacking and how you can apply those skills to overcome obstacles and achieve your goals. So don't miss your chance to get ahead and get this book the Hacker Mindset. You can preorder your copy today at the hacker mindset.com be among the first of many to tap into this power of hacking for your success. Join the movement and embrace a new way of thinking. Again, that's the hacker mindset.com.

Last up we have Jose Valim, creator of the Elixir programming language on Changelogging friends. That is our talk show flavor of the Changelog podcast.

To Jared's point earlier AI, as it's known today, GPT, ChatGPT and others are not that good with assisting with elixir programming. And so I guess the question is what does it take to make it good? You mentioned embeddings earlier. You mentioned documentation being more readily available. What does it take from a, I guess a leader in the elixir world to enable LLMs to be better? Like what role do you play in that journey for them to better consume the documentation and better know how to programming elixir to help folks like Jared and myself or our team or others to really become better and more proficient elixir versus just like anytime Jared asks ChatGPT for assistance, it's just like no, it's not good, so just quit.

So I think if I got the question right, I think we did our work correctly in the sense that at least from the language point of view, in the sense that documentation was always first class. So documentation is very easy to access. So if what you want to do is to configure an LLM, it's actually very easy to access that programmatically, send that, extract information. And we talked about one of the things that you also have to do is try to get understanding from the source code so you can find oh, this code is using those modules, is importing those things and those are things that you can do relatively easily in Elixir. We can most likely improve that. So I feel like we have the knife and the cheese. It's just a matter of somebody going and cutting the cheese. I feel like the foundation is there in terms of having this information structured, but somebody needs to feed it somewhere. But again we can go back. Maybe it's a Corpus size, maybe ChatGPT indexed hexPM already. Not sure. Right. Maybe it has done that. I don't know if I can Send a letter to somebody, hey, please index my website. Or maybe it's a matter of. So one of the things is that redmonk, they release twice a year kind of a graph plotting GitHub against Stack Overflow and I think they're having the most popular languages according to GitHub and Stack Overflow. And then there's a linear thing in the middle. And it's very funny because Elixir is high on the GitHub side but quite low on the Stack Overflow side. One of the reasons for that is because we have always had the Elixir forum. So that may be one of the things.

Where's the knowledge? Where's the back and forth from the community?

Yeah, the knowledge is in the forum is that thing being indexed? Because we know the Stack Overflow is Right.

Right.

And ironically that's one of the reasons I think I may be misquoting that redmonk, they are considering removing Stack Overflow from their plots because it's prone. I think it has been losing relevance in the last years. But maybe in the effort of trying to have a closer community where everybody can engage with each other, where I'm active in the forum and I'll probably not have this patience if I was dealing with Stack Overflow. We created our community a special place, but it's not known.

Yeah.

So I think it's still too many unknowns. But I think at the core, at the core we unwillingly did a good job because we were worried about documentation being accessible, documentation being first class. So we did that and that can be. And we promote people to write documentation, lots of documentation. There is a lot there. And maybe the RAG is going to be the thing that is going to be enough. That's one of the hopes, right. Going back to. We want everybody to be able to use this. If RAG is good enough, then a lot of people would be able to augment their ecosystems without depending on OpenAI or whatever. But we are still evaluating when we.

Talked about sort of the long term future of Elixir artificial intelligence and that sort of larger topic of, you know, how long will be relevant and you know, can I generate it? Well, that whole conversation, this makes me think of this, you know, necessity to not have a black box. That is whatever AI is because just like you said, who do I send a letter to to index my stuff so that my very relevant language today remains relevant tomorrow? Because tomorrow says AI will continue to be more and more relevant to developers in their journey to develop.

Right.

So who do we send the letter to how do we know? Well, currently the status quo of AI is for the most part a black box. Obviously open source LLMs and indexes have become more and more pushed because of this challenge. But I think this illustrates and highlights really the long term challenge because even you can't say for sure why what wasn't indexed was indexed for the elixir corpus. Whether that's the forums, whether that's the documentation through hex documentation or whatever. It's unclear to someone like you how to enable ChatGPT or the likes to better support elixir assistance for developers using those things to use this tooling. And that's just not cool because long term we need to have inroads into those places so that we can be part of the future. If AI is predicting how we'll get to the future.

Yeah, and I think, yeah, it's too early. I think we're going to improve a lot. I was listening to a podcast today where Sam Altman, he was saying like they improved ChatGPT3 about 40 in the orders of magnitude in terms of size, performance and things like that since they started, I think 10 times for ChatGPT three and a half. And I think open source is going to catch up, I think and I think that's the hope. But yeah, it's also like we go back to this when we are thinking about LiveBook because what I want is for open source to win, right? But when I'm building a feature for LiveBook, right, I need to build the best feature for the users, right? And when I can use ChatGPT4 and I can immediately see the results and they are really, really good, I can use other tools off the shelf. They are not as good. We are a small company, we are doing open source. So my options, if I have to choose for my users is going to be ChatGPT4 because it gives me the best result for the least amount of effort.

It's there.

And this is like. So we were backing about my indecision about investing this stuff because I want open source. I want should be open source. But right now the quickest return of investment is GPT and then I am in this contradiction space, right? But yeah, and it's just, I think it's just patience. We have to be patient. And you know, I think probably in one year and the whole thing is like, it's crazy to think about is that this thing has been happening for a year only, right? It appears that this thing has been out for so long, but it's a year. And I think, like, if I'm back on the show in a year, we may potentially be having a very different conversation. So, yeah, we'll see.

Do you have any fear about this? Like, even as you respond to that, you sort of had some, I wouldn't say like trepidation in your voice, but you sort of had some uncertainty. Do you have any, like fear and uncertainty and doubt, the fud that people sort of pass around? Do you have any fear about this?

No, not really in the sense that I consider myself like very lucky, very fortunate or whatever, or blessed, whatever you want to say. It's. I think maybe it's a. I'm not being overconfident here, but more like thankful that I think whatever happens to me, it's going to be fine. I truly believe that what's going to make Elixir survive is the community more than whatever technological changes, unless there is something very drastic. I talked to my father about this, about investments, right? So when Bitcoin was in a crazy, right? And then my father is like, oh, have you heard about this thing that if you put your money there, people got this huge return. And then I always told him, father, if we got to know about it, it's because it's too late. Or if something happens, it's like, oh, father, if something happens, if something goes this bad, it's because it's going to be bad for everybody. So don't try to fight it. So again, unless there is a very major change, I think I will be fine, right? So I'm not worried about me in the sense I always think more about. It's more about ideals, you know, again, like I like to say, well, me, 10 years ago, that's where my trepidation is. If things go like closed source, you know, and those things, they happen by, we don't see the results. Like, I think another polemic topic about this, it's like, hey, I use Chrome as soon as Chrome came out, I immediately, today I don't use Chrome anymore. But as soon as Chrome came out, I immediately swapped to Chrome, right? And if I had known that this would lead to a point where Google is in this position where it has a lot of control over the browser, over the web and over how we use the Internet, like 10 years ago, I would probably not have used Chrome if I could have seen it or so I think that's where my trepidation comes from of things being closed source, like the developer experience. So today, another example today, Elixir was the first programming language that GitHub had, the new navigation code, navigation things that were provided by the community. There were some programming languages and there still are where they have very good navigation and exploration on GitHub UI. The path for that to get that feature, to get that behavior was. And I'm very welcome. I'm very thankful that the GitHub team, they discussed with us and allow us to do that. But that's closed source. GitHub plays a major role over how developers use. It all comes back to this idea of if you want to provide a good experience for your users, how much of that is behind something closed source that you have no control and you are depending on somebody paying attention to you or you having a contact or me having a name because I was very active in the rails community that GitHub uses 10 years ago. Those are the things. But I feel lucky. But it worries me how much is being closed, right? How much is. Is going to be out of our control. And then the. The trepidation I guess is like what does that matter for the small Jose out there, right, that wants to start building his thing today and they won't be able to.

Well, you killed the vibe there, Jose.

Oh, thank you. Well, you just. That's me at parties, you know, so you just.

Not invited.

Just kidding.

Oh, funny. All right, well, let's. Should we try to close on an up note? On a high note? On a.

On an upper.

Was that.

Wow, I had no idea.

I think we should end it right there. Adam, don't you think we ended on a high.

This cheese and knife tactic there, I love it.

Do you want higher? I don't think it will be good for the listeners.

I think that that was plenty high enough for me. Adam, were you satisfied with that?

That was a high note. Literally, yes. And I dig it.

Thanks for listening to this very special episode of Practical AI. These were just extracted segments of much longer conversations. If you want to hear more, there's a link in your show notes to each of the episodes featured here. Thanks again to our partners Fly IO, to our Beat Freakin Residents Breakmaster Cylinder and to our friends at Sentry. Save yourself 100 bucks on the team plan when you use Code Changelog while signing up. That's all for now. Chris and Daniel return on the next one. Sat.