OpenAI and Ex-Shopify Product Lead Reveals How He Builds AI Products | Miqdad Jaffer

So when you build normal consumer products, you have goals, you have a path, you have some requirements, you have an estimate, and then you kind of work throughout that, and then you work on the edge cases in case there are any to be able to narrow in on. But there is a deterministic path toward doing it. In a non determining system, it's very hard to predict what the error cases are going to be and it's very hard to plan for every edge case along the way to try to defend against it. So the first thing that we had to do was assume, assume that everything that we possibly thought would go wrong will go wrong. We don't want the AI to do something without the merchant knowing that thing is happening. How do we push everything in front of the user in a way that they can review, they can look over, and they maintain full control so that this is augmentation, not automation. And that's kind of where we started with a lot of our thinking on this. And what made that unique is the way that we build is not through like one use case at a time. In some cases, it's like a data set at a time of like, this is what we need to be able to like understand. This is the problem space that we need to understand. And there are different ways to approach it. You can fine tune, you can do something from retrieval, augmented generation, and then you can do something just on general prompting. But each approach has pros and cons to it. So like being able to balance and evaluate on an ongoing basis while knowing that your evaluation is never going to be comprehensive and there's going to be errors. And so having that full corpus of unknown means that we build from our principles and stick to our principles as we build versus having, having something linear that we know will work.

My guest today is Mikdad Jaffer, ex AI lead at Shopify and current product lead at OpenAI. I've always admired Shopify's craft centric and meeting averse culture, so I was eager to explore how they applied it to building AI products. We spoke about how Shopify uses its get shit done process to accelerate its AI efforts, how Shopify built Sidekick, its AI assistant for merchants, and how Shopify prioritized quality and craft across all of its AI products. Keep listening if you want to learn how generative AI products are actually built. If you enjoy our conversation, please like and subscribe to support this podcast. All right, Miqdad, welcome. So let's start with a broad question. Shopify's mission is to make commerce better for everyone. I'm curious how you got Shopify's generative.

AI efforts started very so annual planning wise. We do where a situation where Toby writes his themes and he writes them in the form of a letter. And the letter is from a merchant to Shopify. And I love Shopify because. And one of the themes on there was Shopify keeps me on the cutting edge. And he talks a lot about how AI is like a big part of that. But that letter happened towards the end of the year. And what happened was November is when ChatGPT kind of came out and did a like breakaway party of here's generative AI. And the question that was asked was, can we do something with this? And that question comes from like, this is the cutting edge. How do we get our merchants to get on this cutting edge? And the simple answer was, yes, there are a lot of things that we can do with this. And I think we can start with something that's pretty straightforward of product description. And why do we pick product descriptions? There is some N billion products in Shopify and many of them didn't have a product description. And we thought there's no better use for generative AI than trying to fill in something that is such a regular task and happens so many times for merchants. But this is a good place to start. So we put together a theme, we put together a goal of let's get something out in front of our merchants so that we can see how they'll use this, whether they'll use this and what we can learn from it. And we launched something with our winter edition in February at this point. And there was a lot of popularity to it. There was a lot of merchants using it, there was a lot of interest in it. And our goal was can we keep this net neutral? Like net neutral to positive was our goal of how do we make sure that at least this doesn't hurt anybody? And at best this provides something valuable. But like, let's get the technology out there to test. And feedback was great. Everyone started asking for more features, more capabilities, but ultimately the thing that they wanted was, can you put this everywhere? And so that's where the whole thing started.

Oh, that's great. So you actually just found like a small specific use case and just ship chipped it and got the reaction as opposed to writing a big strategy document or something.

Yeah, I think it's important to get something in the hands of the users quickly. And this was like, we could have wrote a strategy doc and we could have said, like, this is our overall path for Theme. We didn't know whether this was going to be used or not, so we figured we'd just find out.

Awesome. So can you talk a little bit about Shopify's Geshe dance system and how did you shepherd the AI efforts through that?

Yeah. So our GSD process is a few phases. There's a proposal where you put together like a one pager to say that this is the thing that I want to do, this is the problem that I hope you to solve. Here are the principles I want to kind of operate by and this is what it's going to look like. Then that goes through a review process and then we go into a prototype phase and then the prototype phase we're testing. So a lot of it is sometimes we'll do UX things, sometimes we'll do technology things. But what we're trying to do is can the solution that we're proposing actually be a solve for what it is that we the problem that we want to solve for? And in this case it was, the proposal was a symbol of like, hey, we want to put product descriptions in. This is how many merchants are affected by it, this is how many blank product descriptions there are, which is what people write. And the people that do have product descriptions do tend to perform better. So we think this is a good place and we think we can do that with generative AI and we think we can do that with a little bit of ux. So we tried a bunch of different UX patterns. There was like interruptive ones, non interruptive ones. And we landed on a set of principles. One principle being the user has ultimate decision making power. They are the ones that get to decide what the message is going to be to their buyers. And we don't want to interrupt that flow, we want to be augmenting that flow. And so all of our UX was around that. And we went through our proposal, many different ideas we went through trying to prompt the AI and realized, okay, we need more information. So we started to build some embedding sources. We started to build some like rag based approaches of like what do we add in additional context wise to make this good? And then we built our own like test method of like, this is what a good product description looks like, this is what makes it that way. And then ran it through a few thousand times to see whether it was going to do it or not. And we felt good about it. And then we move into a phase called build where we present. This is what we want to build, this is what we've learned along the way. This is why we think this is the right approach and let's go ahead and build it. And we did. And then it's a matter and then the final phase is released. And in the release phase it's like this is the working version, this is how it works across some multiple sets of shops. And here are the caveats, here are the gotchas, here's what we think is right and this is what we want to release too. And our initial release was just English speaking merchant and we thought that would get us to market fastest. And it did. And we followed with multiple languages shortly thereafter.

I heard that sometimes you guys don't even have a meeting for some of these steps. So you just like maybe you make a short video or something, I don't know.

This project was a very fast cadence. So what we tried to do was we had a channel with all of our stakeholders in Slack and a lot of the conversation goes back and forth of here's the work, here's what's happening. And we let them be in on the public channel so like they could just watch the work as it goes. And if they want to interrupt, they can interrupt and ask questions. And then every couple days we would just make a demo video and just post it in the channel and just ask for feedback. And a lot of like our vp, especially Glenn, my boss, he makes a friction log and his friction log is like walking through the video or walking through the product and giving feedback on what he thinks is good or bad. And we usually like turn it around in a couple days. But yeah, it's a very, very like fast cycle. I don't think there was many if any synchronous reviews on this. It was all Async as we went.

Okay. And when were merchants involved in this process? Like did you get feedback during the prototype stage or.

Yeah, in the prototype phase. We took mock ups to them, we had a couple conversations with them, we said, what do you expect when this type of thing happens is this. And we had taken a couple of their products previously and generated descriptions and said, what do you think about this? Like does this make sense? Gave them a few samples of it and we got really positive examples. And we even asked the open ended question of like, what do you know about AI? Like what do you think about it? What are your considerations for it? And so we had a good idea of the messaging that we wanted to go forward with. We had a good idea of like, what good would look like and we had good feedback for even future Things that we weren't thinking about from our merchants of like, hey, can you also solve this problem? Can you solve this problem? Our merchant started to enumerate a bunch of different ideas and so it's. It was a really good collaborative process.

That's awesome. So after this initial success with the descriptions feature, like you ship Shopify Magic, which is like a whole suite of products, right? So like, how did you go beyond this initial feature to that whole thing, like, how did you prioritize the jobs to be done? You know, all that kind of stuff?

Yeah, for us it's the culture at Shopify is such that, like, everyone is aligned around that mission of do whatever it takes to make the merchant successful. And when a feature like this drops, every kind of team comes forward and says, hey, are there other things that we can do to leverage this technology as well? Because we see a lot of different places where it could pick that. And for us, we saw generative AI as this is a good place to be able to augment what the user already does. How do I solve for cold start? How do I deal with some of the more repetitive tasks that are going on? How do I save the merchant time? So we started to look at where those places were and where the best fit for this technology could be. And what we found was there is a lot of effort on the marketing side of things and getting users in is a challenge. And then the merchandising part of it is helping them convert because they understand the product and can close at that point. So a lot of the efforts that we made were around the marketing, the SEO side of things and all of the adjacent pieces there of like, okay, how do I get them merchant in, or how do I get a buyer in, or how do I make sure that they can walk through the funnel in a way that's going to be meaningful for them so such that they'll convert. So we went with email as a good place for subject line generation. Like, if we can generate you a better subject line, then maybe you'll get better open rate, or maybe you'll be more likely to send the email at all. And then maybe we can help you write the body of the email and then maybe we can help you with some of the answers. When the merchant is on the page of your online storefront and they have questions, maybe you don't know what questions they're going to ask, but based on your conversations, we can tell you and we can help you answer those questions as well in a way that's going to Be impactful. Then how do we help you customize your storefront so that whenever you have to write about a product or whenever you have to write about yourself, you can get those things going and get started. And a part of it is for our new merchant, some of these things can be a challenge. And simply having a button to push to give you an idea is all it takes to get you started as an entrepreneur. And because people come with high intent of wanting to do that, and as much of the friction as we can remove along the way, the better off entrepreneurship is going to be and the better off our merchants are going to be.

Got it. So kind of thought through the merchant journey and like the biggest problems, like around growth and like, yeah, kind of progress that way, right? Yeah, got it. What about like, did you have like an OKR for this thing? Or you guys not use okrs? What kind of goals did you set?

So we did a general like heart framework that we walked through of. This is what we want to be able to measure. This is what we want to be able to do. The one thing that we principled ourselves against was trying to be too growth hackery about this. Like, we didn't want people to have to use AI for the sake of using AI. We wanted it to be part of the process of what they were doing and it'd be available. So we don't have a. You need to have. We need to have this much adoption or this much usage of this capability or feature. Part of it was if we introduce this feature, it cannot do any harm. And when it works, it has to at least be neutral or positive in terms of its effect. That's kind of the initial goals that we set for ourselves. And the rest of it was a matter of like, how do we this into a place where the users can engage, how do we stay strong to our principles, and how do we make this an augmentation feature, not a replacement feature?

Okay, so when you say neutral and positive, is that like from an experiment or just like for the merchant experience?

Is that the merchant experience and the experiment result when we're running a B test on it? Got it.

Okay. All right, so let's talk about another fee feature like that. Got a lot of news. Shopify is Sidekick, right? So can you just maybe start by describing what it can do?

So Sidekick can answer important questions, can help you understand Shopify. It can make analytical queries and navigate to the right places. So if you don't know where something is, how do I set up my Canadian tax settings? Where do I go for that, how do I set up inventory numbers, or where do I process a return? It can help you create different objects. So if you need to create a product, if you need to create a discount, if you need to create any kind of draft order, all of those things are possible with Sidekick. And it also lets you edit themes and dictate to the what the storefront needs to look like and what changes you want to make from a semantic understanding perspective. And that's a starting place. But we wanted to do everything.

How is like, what are some challenges or learnings from building Sidekick? Like how is this different from just building a regular consumer product?

So when you build normal consumer products, in my past experience and a lot of my stuff has been in data, so it flows a little bit similarly. But even in like a general product development cycle, you have goals, you have a path, you have some requirements, you have an estimate and then you kind of work throughout that. And then you work on the edge cases in case there are any to be able to narrow in on. But there is a deterministic path towards doing it. In a non determining system, it's very hard to predict what the error cases are going to be and it's very hard to plan for every edge case along the way to try to defend against it. So the first thing that we had to do was assume that everything that we possibly thought would go wrong will go wrong. So we started from the error cases first. And in this situation it's going to make mistakes and how do we plan for those mistakes? So if Sidekick doesn't know how to do a thing, which ultimately someone will ask it something that it doesn't know how to do, how do we plan for that scenario? There are going to be cases where the LLM wants to answer, where it's not necessarily the correct answer. What's the thing that we want to teach it? Not easier to be able to. So all those things have to be built up front. In addition to it, you also have to work with when the system goes down, when there isn't enough gpu. Like these are normal error situations, but when an LLM makes them, they can be much more magnified, especially when it becomes part of a regular flow. So there has to be ways to maintain the personality of Sidekick while still making sure that it's still helpful. And then you work around all the safety measures that you need to build around the AI. And so for us it was we don't want the AI to do something without the merchant knowing that thing is happening. And some of that has to happen from. You can't have the AI write to the server because if that happens, then some type of transaction took place and was written to dis that the user wouldn't know about. So we couldn't do that. So we switched it and said, how do we push everything into the client state? How do we push everything in front of the user in a way that they can review, they can look over and they maintain full control? So that this is augmentation, not automation. And that's kind of where we started with a lot of our thinking on this. And what made that unique is the way that we build is not through like one use case at a time. In some cases it's like a data set at a time of like, this is what we need to be able to understand. This is the problem space that we need to understand. And there are different ways to approach it. You can fine tune, you can do something from retrieval, augmented generation, and then you can do something just on general prompting. But each approach has pros and cons to it. So, like being able to balance and evaluate on an ongoing basis while knowing that your evaluation is never going to be comprehensive and there's going to be errors. And so having that full corpus of unknown means that we build from our principles and stick to our principles as we build versus having something linear that we know will work.

Yeah, I've been working on an AI product too, and that's a struggle. The entire product is like an edge case, no matter what's going to happen.

Yeah, exactly. Yeah, exactly.

Can you talk about the different layers of AI product cycling? Like, you know, the data, the model, like, can you briefly describe it for readers?

Sure. So let's go from the bottom to the top and we'll kind of work through that. So at the very bottom is the model itself. Then on top of that is the adapters that you might want to build on top of it. So any kind of fine tuning would kind of apply an additional adapter. Then on top of that is the API through which you will access and any kind of additional changes that you want to do in between. So you can access the model directly, but you can also build an API in between that handles a few different scenarios for you. Some of that could be history management, some of that could be parser, some of that could be grammar enforcement, some of that could be streaming, some of that could be error handling. So some of that is an API layer that we've kind of built in additionally. And then in Addition to that, then you go above and there is the actual UI layer to it. And in the UI layer, some of it is the actual components by which it will show up and then the components by which it will interact, the components by which it will give optionality, the components by which it will collect feedback, and the components by which it will allow for error management along the way in a safe way. Then there is the full integration with the rest of the admin. So it's one thing for it to like interact in its own little box, but it's a whole other to interact with the associated system. So it's what is the API between it and the system. And again, because we're not doing things that the server side and we're doing things on the client state, we want to make sure that the there is an API clean between Sidekick and the admin that's going to be consistent. And those are some of the main key part. The data part of it is constant. I think that the data part of it is either in the form of embeddings or in the form of, in the form of data for fine tuning and instruction. But all of it is something that kind of comes from the entire machine and then the pipelines that build all of that as initial parts of it.

Got it. So there's a lot of companies building these like chatbots, right. And I think the way to one way to differentiate from like, you know, ChatGPT is like it has more personal data about your business or about what you want. So like for Sidekick, what kind of Shopify data is it trained on? And like, how do you avoid privacy issues?

So we don't train Sidekick on data that is personally identifiable. Some of it is like our own created instructions of how we want to use the admin help documentation that we've kind of collected and curated over time. And some of it kind of comes in the form of rad, which is retrieval augmented generation, which is at the time of the query, rather than necessarily training on it. So we try to differentiate accordingly. So we try to keep the merchant's context within the merchant's context and then we keep the general understanding of Shopify as things that we can potentially fine tune the model on.

Got it. Okay. So when the user submits a query, maybe you add a bunch of information, additional information.

Right?

Like in.

Yeah, we'll add like into the prompt, into the context window, specific things about what the merchant is doing, things about the related products or whatever it might be. But it's all in the context of the session and never something that's like trained against the model.

That makes sense. Okay, so let's talk about the prompt. So you know, there's all this like breakthrough as about prompt engineering and stuff and there's also just like eval process right that like can you like just talk a little bit more about that for people who don't know what they are?

So the, the prompt side of it is obviously something that kind of continually develops and I think there's a lot of different method methodologies and ways to do it. I think what we're doing is that we start from the evaluation scenarios and work backwards. And what we do there is like we'll set up a number of integration tests of when this question gets asked, this is the answer we expect and do a few hundred of those for each different scenario. And then you try to run it in an automated fashion. And in the automated fashion there are ways to evaluate and assertions just as you would any Hana unit test. And some of those assertions are equivalency ones where it has to say something very specific because it has to match a specific type or in other cases they're semantically correct. And in the semantically correct version you want to do many examples of those because you are likely using an LLM on top of that to do the evaluation between semantic correctness between two responses. And so that's an additional part of the evaluation side of it. But what we do is we start with the integration test. So I think for us it was every new scenario. We built a few hundred of them and then evaluated against that. But again, because we knew that there were going to be cases where there were error, we already had that built in from the start with the initial.

First few hundred cases. Do you do that manually or like entirely? Okay, so you had like an eval team or did you do it yourself?

Some of it we did ourselves. Some of it was like we have, we don't have an eval team so to speak. Like we have people that were subject matter experts in the area coming forward with here are the a hundred different cases or 200, 300, 700 different cases that are supposed to be passing and we'll hand write them. Like that has to be human done whether it has to be done by subject matter experts. Otherwise you won't really hit the mark of what the things should be. There are ways to be able to like synthetically generate more of them from the initial tests, from the initial hand curated one. But what we wanted to do was create the scenario in which like we were handpicking the things that were most important and that sidekick had to do properly before we were okay with it.

Yeah, I went through something similar. You know there's a way to get the AI to evaluate AI's answers. Right.

But like local way for the AI to generate more like potential and score it.

Yeah, but I think we went through something similar where we went through the synthetic process and then we got a human to look at it and it turned out like a lot of eval was wrong. You got to do the human part first. Yeah.

The other part that's hard about this is a lot of the things that we're used to in the world of unit tests at least is it's like single turn. So it's input output and I'm like black box for the the processing. But as long as the input and the output are correct, then I'm okay. But in the case of a chat, it's multi turn. So you don't have a test that's just on one thing. It's like it could go down a different path along the way of like, let's say you ask the question of like, what kind of discounts should I set up? So in that first scenario it should be like, well, these are the different types of discounts that are available. And then the next question that the user is going to say is like, okay, well, forget that. I want to ask about analytics. Like, what's my store actually doing before I go ahead and do this? And it's not even on the same subject anymore, but now we have to switch. And it's like, well, your store did this much in the last year. Okay, cool. Can you make me a bogo discount? I was like, okay. And so here's how you do it. So you have to plan for these type of scenarios too in the manual curation. And so that multi turn is just as important as a single turn, because this is not a bot that is going to have one subject matter as its expertise has everything.

Do you think it's worthwhile to try to train the user to submit better prompts? Because a lot of times these very short prompts aren't as good as a few shots and a bunch of paragraphs of stuff. Right?

Yeah. I think that people will learn how to do prompting over time. And I think that this is going to become one of those things like the web, where everyone didn't know it to begin with. And the more you learn, the more you learn how to search in Google, the better your results would be. But at the end of the day, or like the job of the user is to be an entrepreneur and not to have to worry about how the technology works or how to make the most of the technology. Our job is to do that. And so we don't view teaching our merchants how to use this type of technology as the key. For us, for us it's about making it feel like magic. And that has to be simple. That has to be like, bring what you have and what you want and let's see how much we can get you there.

Got it. How about when it comes to the underlying model, right? Like do you guys have a lot? I mean maybe this is private, but do you guys have a lot of debate between GPT 3 and 4 and maybe some other model? There's different costs and different capabilities.

So part of it is that's kind of an always on question. And I think it's about running it through the evaluation and figuring out what the best case scenario for the user is. For us it's always the scenario is secondary. The primary question is what's going to solve the problem in the best way and how do we make sure that our merchants are successful? And then the secondary cost, the secondary question is like, how do we make sure that this isn't going to be something that is cost prohibitive, but if you make it work, then let's make it work.

Got it. And I guess one of the flaws of this thing is that sometimes it tends to make up bull and shit. Right? So what is acceptable hallucination and what is not acceptable hallucination for this stuff? Yeah.

So acceptable hallucination is moving people along the path that's on the correct subject. And unacceptable hallucination is like issuing commands or things that are going to be harmful. And what we're trying to do is like add layer along the way to validate and verify that the, that what gets generated is useful. But ultimately we're accepting the fact that it's going to hallucinate. That's why every scenario is one in which the merchant will have full control of what goes out there. And Got it like that. Because that's like principle one. We don't feel like a hallucination will be harmful because the merchant will be the ultimate decider on whether this is good enough for them or not.

Yeah, there was a lot of like auto GPD and this kind of stuff before, but I don't think this thing is good enough to just like run by itself. I need a human person to evaluate, like, to review.

And the thing is that even if it gets good enough, like, that's a decision the merchant gets to make. Like, well, for us it's like they're on Shopify because they get to represent who they are and their brand and they talk to their customers. And even with an AI in place, like, it should still be them.

Yeah, exactly. Do you have any thoughts on, like, it costs money per token to use this thing.

Right?

To use the model. So like, and, and I think you put material out there saying, like, we want to roll out psychic to all merchants for free. Like, how do you prove the business model of this thing? It could just burn money.

Like, do everything possible to make our merchants successful and then make money doing it so we can do more of it. But we never want to reverse the order. So it's always about like solving the problem first. And if we believe this is one of those things that can be transformative to our merchants as we do, then it's important that we invest in it. It's important that we give it, give them all the power that we possibly can.

Got it?

Yeah.

Because if the merchant makes more money, then, yeah, Shopify helps the benefits too. Yeah. So let's put it all together. Like, you know what would like be a really AI powered merchant experience? Let's say the merchant, like I'm managing my store during holiday shopping season or something like that, and all this AI stuff is already shipped. So what will my experience look like as a merchant merchant?

So I think that there's, I think Sidekick will help you a lot. Even before the holiday sale begins. I think it's about like that planning of that holiday Sal and being able to have an assistant with you along the ways of making decisions. I think by the time the sale happens, ideally what would have happened is the products that you wanted to create were created in a way that were going to be compelling and were going to be friendly. For SEO, the discounts or any kind of promotions that you wanted to create had all of the appropriate choices being made in terms of when and why to choose the discount. And then you knew everything about your business that was going to give put you in the best position possible to be successful. And after the sale and promotion happens, I think you can ask for like, how did I do? And to get, be able to get that answer and to be able to walk through, like, what do we learn? How do we do different? What could we do differently? I think a lot of that is in the hands of the merchant. One of the things that we realized is that especially early on when you're by yourself, entrepreneurship is hard and is lonely and having someone to be able to bounce ideas off of, having just some stress testing on it and even being able to get help on like how to do the best possible thing, that's what's critical here. And that's why we built Sidekick. And that I think is the game changer for people is before the sale even happens so that the sale can be successful.

Makes sense. Yeah, I don't sell physical merchandise, but like, you know, as an online creator, like I, I just want to like create content. Right. I don't want to deal with like you know, running ads, managing my business, like all this other stuff. So if there's like AI that can help me with all this other stuff, that'd be ideal. Yeah, yeah. So like, before we move on to the last section, like, I'm just curious, you mentioned a few principles. One principle is like the merchant is always in charge. Do you mind just sharing all the main principles for other people building AI products?

I think the idea behind principles is that you put fewer, but the ones that you put are true principle. And when I say principle I mean this is the non breaking one. The one thing that when you are forced to make a decision you go back to these principles and this keeps you kind of grounded. So for us it's always that first principle is key and that's one of the big ones I think the other ones that we think about are around the experience and like how we want merchants to interact. But it all comes from that first one. So they're all like the choice is in the merchant's hand. So it's not, when we think about like whether we generate options or not, what's going to be better for the merchant and what's better for them is to be able to see multiple options and to be able to choose. And the, I guess that's the main one that we've kind of built for the AI side of things. Others come for like sub products and things and ways that we're considering that. The other one that we've built in from a UX perspective is that the merchant needs to know that the thing is working. And so we've opted for streaming experiences at all times wherever possible. We're showing that this work is actually happening because it needs to feel like it's live and they're not waiting for it to think. So a lot of it is about like making that experience feel like magic. So we have A ton of UX principles around that. But fundamental to the whole thing is AI will not do harm.

Yeah, I think principles are often like very underrated for product development. Like, I feel like a really good set of principles actually has a much bigger impact on a product than like some big strategy document or whatever, you know, because it actually affects the decisions that you make.

Yeah, yeah. And I think it makes it easier to make decisions because you have a thing that you're lined up against. So it's a different methodology, but it's the thing that we probably spend the most time on is making sure these principles are sound and then stress testing them on all directions.

Awesome. So let's wrap up like two questions, right? So like, I think Toby came out and said he wants to make Shopify a crafter's paradise. Can you give me an example of that from your product? Or like, do you just have like a. No meetings on your counter or like what can you give me an example of. Yeah, craft.

I mean, I can, I can give you multiple examples off of the team. And I think it's not, it's not just about the meetings, although it's a big part of it. I think it's about where the decision making comes from. And a lot of that is about being able to do things in the open, being able to bring forward the best ideas. And you can see it from the way that we even built product descriptions as a product we went through and it was the individual crafters that were bringing forward what the best solution was going to be. It was the individual crafters that made the majority of the decisions. Management is there to course correct, advise, bring forward anything that people might be missing. But the crafters that are largely dictating what the flow is and the idea of like working against a goal and setting your own timelines, that comes from the crafters themselves. Like, this is the thing that we want to do, this is how we want to build and this is how we're going to make the biggest impact. And I think empowering the team with principles and empowering the team with a clear user that you're trying to hit and a clear mission that is resonates with everybody. It makes it a lot easier as a crafter to get the noise out of the way and just focus on what's important. And if you get those things done early, you don't have to waste time on the small things because you know what the big things are.

Is there like, is there pressure in Shopify internally where, like, you know, the Crafters are like, oh, you know, like, I gotta get promoting to management to make it. Or like, you know, I gotta get more. Is it happy?

Absolutely not. I think there are two tracks. There are entirely two tracks and they go in whatever direction you want to and they go as high as you want to. So we have crafter tracks and we have manager tracks and there are people that want to manage and there's people that need to manage. But for the most part, as a crafter, there are people that are equivalent to directors and VPs and beyond, and you can go as high as you need to go.

That's awesome. You guys should write more blog posts about that so more companies can copy those two, two track approach.

Yeah.

Last question, like, how are you guys using AI internally for employees at Shopify? Like, is there some internal tools or.

Everywhere is the better answer. So I think that we've built our own kind of internal version of ChatGPT, which just goes through the A. We have a search bot for our internal wiki. We have Slack bots that are AI driven. We have AI tools from like, image generation for people to play with to access to just whatever API is out there for people to play with. It's highly encouraged. We've had like hack days with streams of like, AI work, but like, it's everywhere. And the idea is for everyone to be able to use it to solve personal problem, company problem, but the idea is to get to use it and to become more familiar with it.

So it's not like. It's not like there's some internal team only working on internal tools. It's like anybody can just build.

Anybody can build, and everybody does build.

That's great, man.

Yeah.

All right, thanks so much, Big dad. Yeah, I think that's all the questions I had.

We got to go into a little bit more depth.