GenAI Ghost Stories

One of my core beliefs is that AI is incredible and we need to have the right tools to actually make it work. Like, I don't think there's any, like, free lunch or silver bullet. I know, similar to good old software development, we need to put in the work and write good code and test and deploy it and work with users. So that's what we want to help developers do.

You're listening to Off the VPN.

A show where two work besties get real about working in tech.

What's up, Allison? How you doing?

I'm doing great. How are you?

You know, I'm living the dream. I love spooky season. Like, it's spooky season and we're here.

We're here.

Spooky season has arrived. That's correct. On a non-spooky note, we have a guest today and I'm so excited to introduce her. Giulia, welcome to the podcast. Welcome to Off the VPN. For our listeners, Giulia is the CEO and co-founder of a hot AI startup, Quotient AI. Giulia, you want to introduce yourself?

Thanks so much for having me here, Allison. Jasmine, good to see you again twice in the same week, I think, which is exciting. Yeah, my name is Giulia, CEO and co-founder of Quotient. We're a startup based in Boston in the AI space, building developer tools for developers, and we're focused— I'm thinking a lot about testing AI, building AI, and helping those who are trying to make an impact. So it's great to be here and excited to chat with you. And yeah, spooky season where at least two of us are in Boston, which is the spookiest town, or almost the spookiest town.

We're very close to Salem, which is like Halloween capital of the, of the world, probably. Maybe someone might disagree with that. I don't actually—

you can come at us if that's wrong.

Yes, actually, please do inform me.

Um, but thank you so much.

We're so excited to have you here on just like this really special episode. I know you're just doing like tons of exciting stuff, and like Julia— like, I've had an opportunity to talk to Julia in like several other Evacuations, and just like her take on just Gen AI and what's going on and like all the innovation that's happening and just like the future of like honestly engineering as we know it, if you think about it, right? Or just like technology, like we're on a frontier and like Julia is helping carve the way for a lot of engineering teams. So I think that's like really exciting and we're really fortunate to have you to come chat with us.

Yeah, super excited to be here. I'm doing other podcasts and I'm recording my own podcast also, but Not, not many podcasts with two awesome hosts, so, uh, very excited to be here.

And did anybody else on any of the podcasts you've been on dress up as Steve Jobs for Halloween?

Not inadvertently. So like, it's more like there's a uniform in tech and Steve Jobs is like core tech uniform.

That is true. I will say I initially was wearing a gray turtleneck, so I just like completely borked the uniform because I was like, yeah, you He wore gray, right? And then I had to Google it. So, um, Jasmine, what are you dressed up as?

Can you guess?

I mean, I know.

Not, not typed. I'm not typed. That's—

I'm Python. I'm Python, guys. I'm Python.

Yeah, that's—

if people are listening and not watching, Jasmine is wearing a snake shirt.

I am. It's pretty snazzy.

Yes.

Julia, you got like the colors going on as you—

Costume memo. I think it was probably an email I didn't read, which is fine. I have a funny story about this though, which was I've been complaining about not having a CEO outfit for myself. So I had to go shopping and fun fact, if you Google clothes for CEO, tech CEOs or tech founders. Not very women clothes out there. Oh, Google is like, we're very sure you want male clothing.

Very sure you want it.

It's like completely context switching from the ads it usually shows me.

Well, I think we found a niche. We're going to have to like create something all around this. We should do an episode just on, on clothing.

We'll call you.

Again, I have thoughts.

I'm glad.

I'm glad.

I'm too.

Writing it down.

We're scheduling it. When can you do this?

Yes.

Go to downtown Boston. We try to find something. Speedo and premium.

Before and after a funding round outfit makeover.

Yeah.

I have a fun little game for us to start off today. So if you're down for it. So the game that I have in mind is called Trick or Tech. So what I have here is a list of different statements about things that are happening in GenAI. So as we know, right, like there's tons of, you know, different startups and different technologies that are like, it's like every day in the news there's something a little bit new or different or like unbelievable. So we wanted to do like a little game based off of that to play off of that idea. So I'm gonna give a statement and then you can say, is it a trick? Meaning there's no way that's true, I'm making this up. Or is it real? Is it tech? So trick or tech.

What if it is real as in someone claimed it, but it isn't actually a trick in that it's not real, real. Oh, what's your—

you're suggesting like vaporware? Like, is it vaporware?

I don't think you can talk about this space without addressing vaporware. That's— that's—

so we should have an app, a third option: trick or tech or ghost.

Oh, I like this. Okay, trick or—

I don't, I don't know if I have any vaporware. I didn't think of this until now, so I don't know if any of these Julia found immediately in this game.

It's perfect.

Okay, we will start with— this seems like an easy one. Okay, so AI identifying the best time for a team to deploy code based on historical data and potential issues.

Is it just like not Fridays?

That's a really good question.

Monday through Thursday, optimal time. I mean, Trick, but I could see someone putting this out there. Trick could be.

Do you think it's Trick?

Allison, you think it's Trick?

I don't know, I'll go Tech just for fun.

The answer is Tech. LaunchDarkly has data to optimize deployment times and reduce risks. Apparently today I lagged. Unless it's a ghost. We don't know.

That we don't know.

AI that can detect emotions in cows based on their mooing patterns. Trick or tech?

Now I'm actually like questioning whether I heard this because I did hear someone like doing some like voice analysis of some animals, but I don't know if it was cows or someone else. Okay, I may say tech. I'm going to be really bad at this game. Because I think I like to work when I second-guess myself. Tech. Allison?

Yeah, I feel like that's plausible. Let's go tech.

Yeah, the answer is tech. Yeah, there are researchers that have deployed or developed AI systems that analyze cows' vocalizations to detect emotions like stress or happiness.

It's—

I guess it's intended to help improve animal welfare on, on like farms.

I love it.

Wild.

I want to work on that. Yes, it's the facts. I'm quitting. I'm putting in my 2 weeks immediately.

I want to go work on some cows.

I mean, that sounds like fun.

Is that not unbelievable? I'm surprised you both got that. I thought I was going to trick you.

No, because I already knew there was an app that tells you what your cat's saying.

So like, it's not that far away.

It's like, okay, we already have the technology, we can just deploy it to other animals.

My thing is, how do you test something like this? Like, you test, like, if you're valid, like, you don't—

it's not like you can ask the cow to label data, you know. The best products are— the best claims are the untestable ones. Then you can say whatever you want.

Very true.

All right, let's see. An AI that can help to control weather to ensure perfect conditions for outdoor events.

The AI controls the weather?

Yeah, helps to control the weather.

Trick? What?

So I know there is a company that helps to, helps gives you a bit of sunlight when you want. Like it can give you Like, I don't know if your neighbor is pissing you off, you can like, like flash a spotlight on his house in the middle of the night if you pay this company money. That's what they're saying of their product. But I don't, I haven't heard any weather ones. I gotta say trick, but probably definitely within the realm of possibility.

The answer was trick. Yeah, that doesn't exist.

It doesn't exist.

I'm like picturing, I don't know, like AI somehow clearing like rain clouds. And I'm like, what? What is this?

But what a world we live in that we would even be like, I don't know, is there, does, is there someone who's doing that? That's insane that we had to even like think about it. Or the fact, Julia, you had something that immediately came to mind is pretty incredible.

Yeah, I know. What you're describing is more the plot of Snowpiercer, right? Like they split up stuff in the clouds and then they mess up climate forever.

That's the one where they're stuck on the train, right?

And they have to eat like some post-apocalyptic, like, snow situation.

Wild. And it just keeps going. The train.

Yeah. You cannot stop the train. I forget why.

All right. I have, I have one last one. Okay. A platform that speeds up AI product development by running quick, Iterative experiments. I heard a good tag.

There's a nice easter egg to this one.

Yes, the answer is tech. Julie, you want to tell us why?

Yeah, that's, that's actually a cool shit. I'm happy I picked up my own product in the lineup. Sometimes I'm like, there are so many cool products out there.

Yeah, we would have cut this out if you didn't, if you didn't recognize.

I'm going to market this whenever the website Um, no, yeah, that's, that's what we're, we're very focused on. Like one of my core beliefs is that AI is incredible and we need to have the right tools to actually make it work. Like, I don't think there's any like free lunch or silver bullet. I know similar to good old software development, we need to put in the work and write the code and test and deploy it and work with users. So that's what we want to help developers do. And I hope then it's going to be, I don't know, everyone who is able to work with, with these new technology and not just a small group of people at top tech companies.

I love that last part you said, like, everyone can kind of have access to these things and that would have insane, insane impact. Everyone is able to do that and there's not all these like giant boulders in the way of people being able to create new technology like that.

All right, so I have a story for the two of you, a spooky, frightening horror story, maybe, if you will. I mean, to me it was pretty horrifying, but I'm very curious to what the reactions will be. So the title of this post is, "Am I the asshole for giving my developers a pay cut?" As I'm sure you are aware, software devs are super expensive. I am not a software dev, but employ two of them for my mobile game company. Long story short, I recently did two things. I bought my devs ChatGPT Plus, and I also did a pay cut of around 20%. I demonstrated that I could generate the same code they were writing in their commits with GPT-4 with only very minor differences. They threatened to quit and told me I was free to generate all the code myself, but told them I still need someone to integrate the code. My core argument is that integration does not cost $100K a year. I also pointed out that artists and sound designers don't get nearly as much as software devs. And software devs have been living in a salary inflated bubble for a while now. They are painting me as the bad guy, but the writing is clearly on the wall. GPT-4 can generate code just as good as, or even better than, a human. All software devs are needed for now is to integrate the code into the codebase and fix any hallucinations. Thoughts?

This reminds me, there was this meme circling Twitter a couple of days ago, which was, oh, you're right, like, Cursor can, or like, Copilot can generate code and replace all engineers. Look what I built with it. And they're like, oh, show me. And the person shares their localhost address. So I think that, that's my initial reaction.

Oh yeah, can you imagine like someone being like, yeah, I'm gonna take 20% off your salary, like, and this, and then the second part of just that alone, I'd be like, what do you mean? What do you mean inflation up? Like, what do you, what are you talking about? We're supposed to go up, not down. But then like being like, oh, because you use Gen AI, like, what? I I can't even like wrap my head around this logic that you think like you don't need developers. I love that the developer said you could do it yourself. That's actually really funny.

I mean, it's not like, it's kind of the right response. Like, okay, if that's what you think is involved here, then why do you, why do you need anybody? Like, why, why a pay cut? Why not just go to zero and do it yourself? Right? Like, if you take that, like, logical argument all the way through, which, like, obviously I don't think it's logical. I, I also love that they were like, bet, go ahead.

Do you think that this is like a common misconception? Like, I think people do. I mean, we got— actually, Julie, I don't know if we said this to you, but we got our first hate comment, Allison and I, for OptiVPN, and it actually— on our YouTube we have a hate comment.

And honestly, I was really excited because, oh, because haters, and there's like, oh, there's drama, right?

Like, yeah, I was like proud of us. I was like, oh my God, someone cared enough to say something horrible to us. But they were like, basically they were like, AI is gonna take, is gonna like take all the developer jobs anyway, so I don't know why you, why you guys are doing this, basically.

So there's also like, why test, right?

Like, we could just drop this in a Google Doc and call it Uh, yes. Right. That's a really good point. Exactly.

So I experienced a version of this personally. So more than half of my job is going around and talking to potential users and customers. And it's almost never developers actually who, who say what I'm about to say, but if it's like a manager or a director or VP or almost never founders actually. Like if it's, if it's like middle management, let's say in tech, there is this very strong, strong feeling that things can be built in-house. Like there's this like core conversation about, are we going to build this in-house or are we going to buy it? And people are very convinced they can build something in-house until they try and build it. And then they're like, yeah, like this. Thing that I thought was easy, uh, took me like a month and I thought it would be like a 2-day conversation. So this is, this is like a fundamental, like, I know what this person's up to, like the, the one in the post, but it's a very fundamental, and it's a fair question. Like a lot of times, you know, some things you should build in-house. My take is you should build things that are core to your product and your business and everything else you should outsource. This idea of like, I can do it myself, I don't need other engineers, is a very core, core, I don't know, conundrum, let's say.

Do you also think like this person, they're saying like, they're pretty much saying like, you're not doing like the full breadth of like engineering work now because the robots are doing it basically. Do you think that that's like what's happening now? Engineers just are like on autopilot and are doing 20% less?

I think that person should just try to operate without Linear and see what happens. I don't think that's the case. I think it's just what's happening and what's going to keep happening is the job is just going to change. Like people will just continue doing, like start doing different tasks and have more time for different tasks. And yeah, there are always outliers. Like there are people who are super like overemployed and they have like 5 jobs they put in 2 or 4 hours a day. But those are outliers. That is not the norm. I think we should be talking about like the average person, the average software engineer, the average product, the average company. And I think that job is there to stay. I think it's going to change how that time is being spent. And yeah, it may change I don't know how many people you end up hiring. Like maybe you do see the same output from like a smaller number of people. Like that can happen.

Yeah, maybe. I love— no, I don't love, I hate that he reduced software development down to just integrating code, like basically putting code in the right place and then fixing hallucinations. Like, that is just a wild— to me, a wild, like, misunderstanding of what your developers are actually doing. But I do agree that I, I think, like, at the end of the day, like, there's a— there's new tools in the space, and so what that will do is just change the way that people work and possibly open up room for developers to do, like, different types of work, like more like higher value types of work. If you're sort of making the, the coding aspect a bit easier, like does it actually like, I don't know, widen the possibilities of where devs can contribute beyond coding? Like I think that's one potential, like really interesting thing that we could see of like, I don't know, could a dev become more of a PM a little bit? Like have more of like a product angle. I mean, that's just one example, but I do think that it is wild to me that this guy was like, I just need you guys to integrate the code and therefore I can pay you 20% less.

It's okay. He's gonna, or sorry, they're gonna pay a developer when they need to pay someone to fix whatever's gonna go wrong with this.

Yeah. Yeah, I mean, ultimately this person is just super cheap when they should be looking at, like, just hearing what you guys are saying, like, this person's very cheap when they should be looking at this as opportunity of like, okay, what does this unlock for me now that my developers are faster or have these tools, rather than cheaping out on like, yeah, you guys are overpaid. Like, excuse me. Right?

Like, could they actually be doing more faster because this is enabling them to work differently, as opposed to just saying like, yeah, they just integrate code and so I'm cutting their pay and we'll work at the same pace given that they have these tools, but they deserve to be paid less. Like, completely agree. It's just like the, the whole framing of it is wrong.

So asshole, do we think asshole, or are they just—

I mean, I don't think they're a good manager. I think that's definitely facts. And there is definitely a sense of entitlement. I don't know if like, I feel like I also need to be like, I think it's like, I like to reserve that label for like the truest of truest assholes. They can run the company the way they want. People are free to leave. We'll see what happens when he's surrounded by, I don't know, what were they using? ChatGPT? Yeah. Mm-hmm.

Facts. So what, what is the final verdict then? So you're saying not necessarily an asshole, they're just—

I think I'm, I think I'm gonna go with your definition of not an asshole, but it is soft. Okay.

Yeah, I hard agree. Just really seem misinformed.

I'm gonna just go off—

is this—

you guys don't want to call him an asshole? I will call them an asshole. So, um, because like, it's not even about the Gen AI and like, oh, they're thinking about it wrong. Like, okay, I can maybe give some like, yeah, they need guidance and stuff like that. It's the fact that they're like, yeah, all you people have been working for me, I'm taking 20% of your pay, is crazy. So for that reason alone, it's not because, oh, the business is losing money, they're like underwater, they need to perform. Like, there's no like real reason. They're just like They're just tight that they have to pay all these engineers like a, the salary that is appropriate in the industry, and they're trying to find a reason to pay them less. So for that reason, I think they're cheap and I think they're an asshole. And that's, that's my final, that's how I feel.

One corner case for you, Jasmine. What if this was written by like a FAANG CEO and the salaries they're getting 20% cut are like very cushy, would you still call them an asshole?

I mean, you, you should— like, I don't know what you say you're gonna pay somebody something and then you're like, come on. Like, I don't know, I just think that like anytime you're taking bread out of someone's pocket, I can't get behind it. Like, I just can't.

That's fair. That's— yeah, but I, I see what you're saying.

Like, if they were already— but then they, they said how much they were making. They said what, what was it again? Didn't they say what the I heard 100K, but I forgot what I heard. It was 100K. Like, why do you— 100K plus.

They didn't specify. It was 100K plus.

Oh, but that probably means like, not like 300K.

Yeah, that's my guess.

Yeah.

Yeah.

Okay. Fair.

But, but I mean, I don't know, to like maybe make this a bit better, I would've I think like to see some logic behind the 20%, 'cause that, that number is completely out of nowhere.

Yes.

Like no argument was even made for like how you are deriving the fact that your developers are doing 20% less work because of the use of ChatGPT. Like it just seems like pick a number out of a hat that feels right. I don't know, like not that I agree with it to your point, Jasmine, but If you are going to do that, at least like, what's the logic behind it?

Yeah.

As opposed to like, pick your favorite number between 1 and 30. I don't know.

I don't know. You're seeing glass half full and I, I go the opposite.

We definitely agree that they suck.

Uh, yeah, we all agree this is a horrible idea and it's not gonna work out.

Yeah.

I hope those devs quit. I hope they did. I hope they found better opportunities elsewhere.

I feel like there's so many unlimited topics. We could keep going all day. I love that we got to talk about the stories and talk about like all the crazy stuff that's happening that we get to witness. And I think that we keep revolving around the topic of like how specifically engineering, you know, just itself is going, is changing, like just how we develop, uh, the tools that we're gonna need that Julia really focused on how to help developers succeed in that. What, what do you kind of think engineering is going to look like now? Like, how do our workflows impact? How do we work with other people? Like, what, what, what does it even look like? What do you imagine for that?

That's a good question. I think there are a couple of things that I see happening as AI tools become more incorporated in the developer workflow. One, I see engineers starting to work more as let's say, guides of artificial intelligence. Like you get some, you have some, some, something else do the grunt work and something else do the repetitive work. Oh, you show up at the end of that process and you say, okay, I received what I expected I would receive, or I need to refine that process and try again with the goal of getting like reusable processes that next time around we need to develop something new, you can get something semi-automatically. And this is that like, I don't know, productivity multiplier process that's promised by everything. But then this also means that for even jobs as like straight out of school for engineers are going to change fundamentally because right now most of the job of entry-level engineer is that repetitive work. And I think what we need to be careful about here is how do we need to adjust, like, software education in a way that people can get to being a senior staff principal engineer and have, like, core engineering fundamentals while maybe, like, skipping this, like, hey, you have to you have to put in your dues and you have to like code up an entire backend system from scratch kind of thing, which I expect is going to become maybe as rare as people doing their own like pen and paper calculations. So I think things are going to change there. Uh, the other thing is I expect everything to be a lot more collaborative. Something I spend a lot of time thinking about and seeing is because as developers are incorporating, for example, LLMs into their systems, they're tackling problems in which they have little less and less expertise in. Like you have an engineer building like a custom or like a legal assistant bot and they're not a lawyer and they need to make decisions at coding time, at development time about what they need to put into things like the prompt for the LLM such that the legal assistant works. And that's like a core pain point that I see for developers right now where they don't know, like they don't have access to the testing capabilities or the feedback at development time to know what is good and what is not. And I think this is going to be solved by, say, like closer collaboration. I think there's going to be tools that show up that improve collaboration between developers and subject matter expertise, such that the loop between, oh, I'm a developer, is the answer to this legal question that I'm getting out of my code correct? And someone having an opinion on the accuracy of that is as tight as possible. Right now, that is not the case, but I see everything becoming very collaborative because, again, the problem space that these— this technology allows developers to tackle is becoming bigger and bigger every day.

And I love how, I don't know, like realistic it is, right? Because I think speaking of horror stories and I don't know, spooky posts on the internet, like I, I feel like that is a lot of the conversation happening, which is like, are junior devs a thing of the past? Like, should I even bother studying computer science and learning the foundations anymore? Or, you know, like these, these things that are just causing, I think, more panic as opposed to like your take, which is this is how workplaces operate. And given these new tools, like these are some potential avenues for change, but that doesn't have this like apocalyptic outcome. Of like no junior devs, pay cuts for everybody, just full jobs, like, you know, out the door, like anybody can be a lawyer. Like, no, you still have to have some subject matter expert, like giving input on building, you know, tools around like legal documents and lawyer things, whatever lawyers do, you know. So I really love how realistic that take is.

What do lawyers do?

I'm like, I don't know what they do. Go to court. No, I know what lawyers do.

It's okay.

All I know about lawyers comes from Suits, so I feel like that's enough.

Legally Blonde.

Legally Blonde is definitely my reference point.

Yeah.

Like, it's hard.

Yeah, exactly.

But yeah, also the jerk boy thing. About like, I know our entry-level devs gonna go away. I think the devs as they exist today will, will probably go away. I think we just need to adapt. Like, I don't think you can say, yeah, someone with an entry-level developer skill set today is a persona that's going to exist in the market in 10 years. I actually don't think that's the case. I think it's still going to be a college grad with a computer science degree, and I think the skills are going to be wildly different.

That is a really good way, good way to put it. And I think we even talked about like, we have our own study. We did a jellyfish like impact of Copilot specifically, and you just using like AI coding assistants, and we're finding like junior developers specifically are like 4 times faster, which is exciting and mostly scary. You wanna talk about scary? That is a horror story, I'm sure on its own, of having developers 3 to 5 times faster. But it's very exciting 'cause it's like, imagine in, it's exciting and scary. I was just thinking of like having like a bunch of junior developers reporting to me that are that fast would, is just really scary. But it's also very exciting, right? Like I think, I think from the conversations, Julia, I'm having with you, it's like it changes from being like apocalyptic and scary, you know, into more of like a hopeful, like I feel like I'm more feeling like I'm a part of like this moment in like, you know, just history and technology, right? And like, this is like a really big moment that we get to live and be a part of and, and advance. And so I think, uh, we get to set the patterns and decide how AI is going to fold into our lives. You know, it's happening whether we like it or not. We just have to kind of like guide it, which is why I really appreciate all the work that you're doing at Quotient, um, Quotient AI and, um, Also, I also wanted to say, like, don't you— if anyone's interested and you're listening and you want to know more about this stuff and like just hearing all of Julia's hot takes, like, you have a podcast. Am I right about that?

Podcast? And like, where I have you on my podcast.

Look, hello.

Yeah, I just started recording this a couple of weeks ago. It's called Two Shot. It's a prompting joke. It's only for the, only for the people who are deep into the AI weeds. No, my take is, um, I think everyone's making the field sound super vaporwavey and complicated and exclusive. And I'm not about that. Like, I think this is like cool technology. I think it has easy, straightforward applications. I think people are engaged. Keeping using terminology, and I want to do away with all that. So I bring people who are enthusiasts, researchers, experts in the field to just talk about things in a normal human way that's accessible to everyone, uh, and also dive deep in some things, poke holes at different stuff that's out there. And you can find Two Shot anywhere you get your podcasts. It's out now and excited for more episodes.

I can't wait to listen.

I—

me too.

I'm— I want to subscribe. Also, Quotient AI has some of the coolest— like, I got to see—

we have—

I got a t-shirt recently. Julia gave me a t-shirt. Yeah, I know, Allison. I will be borrowing it.

Get hip.

Uh, no, but fire logo. Go check them out. Like, if you're doing anything with AI, like These are some great people.

They're doing really cool stuff.

So, and Julia, again, I want to just thank you so much for being here and just chatting with us. This has been such a fun conversation. We had so much fun just learning from you and talking about just, just everything in between.

So it was super awesome. I'm excited to continue hanging out in Boston. I also need to come up here and see you guys soon. Otherwise, I'll see you on the interwebs. Thanks for having me.

See you on the interwebs.

Bye, ma.

Bye. Thanks for listening to Off the VPN.

If you have any juicy tech stories or some burning questions that need answering, visit jellyfish.co/offthevpn to share them with us.

We want to spill them on Off the VPN. And don't worry, we'll keep it on the DL.