Working directly with LLMs, code, and your Amazon data | Amazon X AI Working Sessions

Yeah, so, um, I, my name is Jack Atlasoff. I've been in the e-commerce and growth marketing space for about 12 years now. I've pretty much done every type of e-commerce from D2C to marketplace, owned my own businesses, uh, worked inside of startups, inside of large, uh, companies. And everything in between. And I've been on the agency side and on the brand side. So I've been following just a little bit of like AI context. I've been following AI since about 2015, 2016 in terms of like the research and development in the space. And actually we started using what was DaVinci through OpenAI's API back before ChatGPT even launched. I— this must have been in like 2021 or something like that. So I've been using AI daily now for just over, I would say, 3 years, and I'm very excited about what we can do with AI as knowledge workers. Oh, I should mention also that I'm no longer with Bird Rock Brands. Technically, I don't start in this new role until next week, but I'm going to be the director of agentic commerce at Right Side Up. So that's my new role.

Oh, very cool. Yeah, I know you mentioned that BirdRock was acquired, so kind of felt like maybe that time was limited, but that's awesome, man. And we'll give you a little bit of time later on to plug what you're doing next.

Cool. All right, so should I get started?

Yes, please.

All right, cool. So today I'm gonna show you guys how I use AI to build software and analyze data and build software that analyzes data. But, um, the output of the process that I'm going to show you, it's just the result. The real meta here, the real alpha, is the workflow itself. So, um, everything that, um, that I'm going to focus on here in this presentation is to help you understand how you can build a similar workflow or just take this workflow, customize it for your own needs. But really the workflow is the product that I really want to focus on in this talk.

Got it.

All right.

Yuko, Yuko, I noticed you have your hand raised. Do you have a question before we get started? Okay, cool, cool. Gonna put it down. Go on, Jack.

Yeah, and I, I will try to go through this so that I leave about 15, 10-15 minutes at the end for questions. All right, so let's—

by the way, we're getting, we're getting your full screen, not just the, uh, the slide.

Yeah, I think that's okay because I'm actually going to be sharing a bunch of— I can do this for now, but I'll be sharing a bunch of different screens here in a moment. Yeah, but for now I guess I can just expand this. Um, okay, cool. So let's just introduce a problem as a way of entering into this workflow. So I'm sure that a lot of Amazon sellers, a lot of business owners in general, can can relate to this pain point, which is how do you grow total revenue while still maintaining profitability, right? So specifically with Amazon and with Amazon Advertising, you want to scale top line, you want to spend more on Amazon Ads, but you're not seeing that total contribution margin dollar growth that you need to continue scaling profitably. So your ROAS might look healthy, but every time you try to scale, your P&L starts to suffer. And you're looking at all this data from your Amazon Advertising dashboards and it's giving you a snapshot of what happened, but it's not really telling you what to do next in order to solve this problem. Also, I see it a lot. I'm sure we all see it that oftentimes ROAS is not the greatest of metrics because it can look really healthy, but it's not really creating— it's not really as a result of you creating much incrementality. Or net new customer growth in your business. For example, if you're spending on branded defense or just branded campaigns, the ROAS there can look really good, but it's not really leading to more total contribution margin dollar growth for your Amazon business. So before we dive into the workflow, one more just context setting here. I just want to introduce a kind of broad history of Amazon advertising. So the way I think about it is there's been 3 phases, or there's really been 2 phases of Amazon Advertising, and I'm gonna introduce a third. There's Amazon Advertising 1.0, which has been around for years, which you can think of broadly as you have 2 axes that you can scale your ad spend on. You can scale on the targeting axes. That's like, you know, keyword targeting, product attribute targeting, ASIN coverage, and you can also scale on the bid aggression axes, which would be like your cost per click essentially, right? And as you scale across these two axes, your spend will grow. So that's Amazon Advertising 1.0. That's been around since I started selling on Amazon over 10 years ago. Then there's— now for the last couple of years, there's been an expansion into what I will call Amazon Advertising 2.0. Which you can think of as just more a kind of surface area expansion across those two axes, right? So now instead of just keywords and products, we have categories, we have audiences, we have different destinations that we can send advertised traffic to. We can bid aggressively on top of search versus detail pages versus rest of search. So we've really still— we're playing in those two axes, but we've really expanded the— surface area of those axes. So that's what I would call Amazon Advertising 2.0, and you can definitely scale a lot more spend through, through this new framework. But then I'd like to propose a new axis of scaling, and this has actually been around the whole time, but quite honestly, it requires a slightly higher level of sophistication to implement, and that's why I think it doesn't get talked about that much. Also, if you have a small catalog, this type of optimization and scaling on this axis is not going to be quite as beneficial to you. But if you have even just 10 SKUs or more, you're going to probably find some benefit by implementing this framework. And so what this is, is your ability to allocate your dollars in an optimal way across your campaigns. So what is, uh, what does spend allocation really mean? It can be helpful to think of it in, uh, kind of metaphorically in the context of the way that, um, you would allocate money to different investments. So let's say you have the equal weighted S&P 500 and you allocate the same amount of dollars to every equity, every stock within that S&P 500 portfolio. Or the other scenario might be that you allocate your dollars based on the market cap of every stock in that portfolio. I think we're all pretty familiar with the fact that these two allocation strategies, even if the, the total dollar amounts are the same, even if you don't allocate a, you know, different amount to each strategy, These two strategies will yield different return profiles, right? So you can think, you can apply the same metaphor to your Amazon Advertising campaign allocation of spend across your campaigns. And remember, this is not a set it and forget it type of exercise. If you allocate a certain amount of dollars across your different campaigns, you will, you know, get data from the market and you will most likely rebalance. Just like you would with investments. And my kind of argument here and the way that this optimization works is over time, they, you know, those sellers, those advertisers that can rebalance their spend allocation most optimally and scale their spend allocation most optimally will generate outsized returns in the form of total contribution margin dollar growth across their entire Amazon business. So we talked about the pain, we've talked about the context behind the pain, and we've talked about a potential, you know, solution to this, which is implementing this new spend allocation optimization framework. So what does this look like in practice? Well, it's quite complex, unfortunately. I wish it was as simple as just finding new keywords or bidding more. This new axis is a little bit more complicated, but the good news is you can implement it with software. So I built an app, it's called Next Dollar. Uh, it's not publicly available, but it does exist. I will show it to you today. Um, and the goal of Next Dollar is to use a little bit of sophisticated analysis to help you figure out as the advertiser, um, where the next best dollar of spend should be allocated, right? So what is that optimal place to put that next dollar of spend? So the goal of this application is to try to get you out of a reactive mindset and into a more prediction and proactive mindset. After all, I believe that there's a tremendous amount of value in being able to predict what's gonna happen more accurately than your competitors. There's a lot of alpha in having an, uh, some kind of a feedback loop or an accurate system of prediction that improves ideally over time. So that's what Next Dollar attempts to solve. So what does it look like, uh, when you're inside of Next Dollar? So what it looks like is it's continuously evaluating the, um, the relationship between your spend and your contribution margin dollar margin dollars, specifically your incremental marginal contribution dollar for every dollar of spend. I know that's kind of a mouthful, but we'll kind of walk through a little bit more details here in a second. So it's looking at, okay, what happens if that— what happens if I take this next dollar and I put it in, you know, A campaign versus B campaign versus C campaign? What is the result in terms of— what is the predicted result in terms of the total amount of marginal contribution dollars? That, that decision might make. And obviously you need a little bit of data in order to build the initial model, but then after, after you build the initial model, you just kind of run it as you get more data and it improves over time. And so what this looks like is spend reallocation suggestions from one campaign to another. So again, you don't necessarily need to introduce new dollars into the system. You can just reallocate more efficiently. You can also introduce new dollars into the system to continue to grow, But even if you don't introduce new dollars in the system and you've never done this before, uh, my claim is that you will uncover more total contribution margin dollars simply by applying this new level of optimization to your existing campaign infrastructure and your existing spend allocation strategies than if you hadn't done it at all. So, um, graphically, visually, what does this look like? Okay, I'm not gonna get into the math, but fundamentally what this looks like is that every product, every campaign in your Ad Console has a relationship between spend and sales. And more importantly, every product in your Ad Console and every campaign has a relationship between the next dollar of spend and the predicted sales that come from that next dollar of spend. And so the idea is we identify where you currently are on this curve, what we call a spend/sale saturation curve. And then we use a little bit of— or the app uses a little bit of sophisticated mathematics to figure out where you should be on the curve given your margin requirements, right? That you have to, you know, be profitable on that sale. But you, uh, given the fact that you don't want to have diminishing returns because you have other campaigns where that next dollar would be better suited, right? So remember, we're trying to optimize total account-wide contribution margin dollars. So we really wanna make sure that the next dollar of spend is being placed in the most optimal place. And that is, where is it gonna generate the most amount of contribution margin? So we might be here and we need to move to this point here, where we're recommended, but we don't necessarily wanna spend here in the curve if we have another campaign where we're over here and we could get up to here, right? So you can see how as you move products from where they currently are, and you might be overspending, right? And then the, the app will recommend that you decrease your spend, right? And reallocate that spend to something like this situation where you can increase your spend. And as you rebalance like this every week or every 2 weeks or every month, if you're not spending a lot or don't have a ton of campaigns, or maybe every couple of days if you are. As you do this more and more, the model gets better and better at predicting these curves, and you become better and better at knowing where to put that next dollar of spend. So what are some use cases for, you know, an application like this? Well, if you've trained a model to do something like this, you might be able to know what to spend on new campaign types better than a competitor who doesn't have something like this. You might be able to know what to spend on new audiences, right? Have a more informed, educated hypothesis on, on what to spend on new audiences. You might be able to figure out what to spend on new landing destinations or new ad types, or how to differentiate the spend across the different child ASINs within a variation family such that the total contribution margin dollars across the entire family is is maximized. How are we doing? I haven't— I'm not looking at, you know, the chat, so I just want to make sure we're— everybody's— we're still okay.

Everybody's following along, no questions so far. I think everybody is waiting for the next, the next slide, which is how— there it is.

Yes. Okay, so how is something like this created, right? How does an app like this go from just an idea or a pain point to, uh, something that exists in the world. So this is the typical way that people build with AI, which, you know, you can think of it as a bell curve. You have low output on the left, high output on the right, and most of the time you're going to get something that's an average output unless you use a deliberate workflow. This is my claim and kind of the core of my talk is unless you use some kind of a workflow to push you into the the right tail there and have a better than average output. Why is this happening? Why do most people get an average output? I think it's because some version of the Dunning-Kruger effect tends to happen when people work with LLMs. You might have an idea and you say, I'm going to make an app, and then the LLM makes the app and it looks pretty and you're just so stoked. That's a very dangerous moment because it feels like you've become a subject matter expert. It can feel like you've become a subject matter expert in that moment, right? But then as you start to use the app, and I'm sure a lot of people on this webinar have probably experienced this if they've pushed their LLMs to build something for them, as you start to use it, you find all these ways that it breaks or doesn't work or produces garbage results. Or maybe you take your thing that you built and you bring it to somebody uh, like a coworker, and they poke holes in it, and it might be a little embarrassing, right? So then you're in the valley of despair, and, and, and that's just, you know, what happens. We've all been there. When I first started coding with LLMs, I totally went through this process. Um, so how do you get out of the valley of despair and move towards data code mastery with LLMs? You, uh, build yourself a workflow. And what I've done here, this is my workflow. This is similar, you know, to some of the stuff you might see out there already. I've come to it naturally. I think that it's, it's pretty solid, but feel free to, you know, take it and customize it for your specific needs. Fundamentally, this workflow is 6 steps, and the 6th one is kind of like a meta workflow. So the first step is to research, and there's a couple of different ways that you can research. The second step is to compress that research. This is a very important step that doesn't get talked about nearly as much as some of these other steps. What do I mean by compress? I mean boost signal to noise. So you pull in a whole bunch of context during the research step, but not all of it is super relevant. And it's your job as the human in the loop of, you know, building all of this or working with LLMs. It's your job to boost signal to noise. You're— for every unit of time that you spend in the compression of the research that you've generated, for every unit of time that you spend in this step, it's going to pay massive dividends as you start to move forward with building. The third step, people are starting to talk about, but I still don't think most people are aware of it. And this is the alignment step. So you have your research and you have your high signal-to-noise research after you've done the compression. The next thing, the next part of the step is very critical You have to make sure that you and the LLM are on the same page about what you're gonna do. So what does this mean? It means, are you using the same language to mean the same things? Literally, like the vocabulary. You know, when you say tacos, does the LLM understand what you mean? When you say incremental contribution margin, does the LLM understand what you mean by that, right? People skip over this step and then are like frustrated with the results that they get and they go, oh, that's not what I meant. Well, did you take the time to even align on a shared vocabulary? So this step is very important. After you've researched, compressed, and aligned, you're ready to plan. Most people either go straight to plan or they go straight to build and verify, and they skip the first 4 steps or the first 3 steps. But it's only after the first 3 steps that I even start to plan. And you definitely want to plan in a way that is structured. You want to, um, generate planning documents. I'll show you what those look like here in a moment. Then you want to basically go back into the align step and what's called grill. There's a skill called Grill with Docs or Grill Me. Basically have the LLM interview you to make sure that everything that you've put into the plan is what you agree with essentially. After you've built the plan, then you will go and you'll start to build. The way that I build is through a build-verify loop. So I don't build something unless there's a test that I can run on the thing that I've built that proves that it was built correctly. So how, how does that work? Well, the test is set up so that it's failing by default. That's very important. When you set up your tests, when you're building something with an LLM, you should assume in the test that the, the feature, the, the piece of UI, you know, code, whatever it is, should assume that it's failing by default. Make the LLM prove to you that it's working, um, before that feature or that test passes. And then the next thing is I always combine my builder agent with a verifier agent and make sure that they're two separate agents. Um, you can do this with Codex, you can do this with Claude Code. I can show you what that looks like here in a moment. Um, and then like I said, uh, The last step of this is kind of an optional step. I really recommend that you do this, and it's kind of a meta step, and that is build some kind of, build some kind of process or use a plugin or a skill to capture your learnings as you go through the first 5 steps of this workflow for every project that you do. Capture your successes, capture your failures, capture the errors, capture the insights, and record them somewhere. I use Obsidian. Um, there's lots of different, you know, tools you can use. Some people use Notion, some people use Google, uh, Workspace. But capture the successes and the failures so that your workflows and your, um, context that the LLM can pull from for future applications or future build projects, uh, improves over time. Okay, so what do all these, um, stages look like? So the research stage can look Uh, I'm switching tabs here. Can everybody, can everybody see this?

Yep, we're in ChatGPT now.

Yeah, now we're in ChatGPT. So the research stage can be as simple as this. Um, this is ChatGPT Pro, which is the, the, you know, highest tier of plan. But, um, you, you can use ChatGPT— I guess it's called Heavy or Heavy, uh, Extra High and High— you still get a decent result. You can also use deep research here in ChatGPT. You can use Gemini, you can use whatever, you know, deep research tool you want. But literally here I just posted in the LinkedIn post for, for this event, right? This is how the whole process for the app started. I posted in the LinkedIn post and then I just provided a little bit of context around what I want to build, right? I think this would be cool to demo in the webinar as a kind of example of this workflow that I— that is the real reason that we're all here. And so—

Okay, Jack, I got to interrupt you really quickly here because I think that's something that a lot of folks, myself included, may have missed, which is, are you telling me that Next Dollar, this app that you've been demoing, is something that you didn't even think about building until I told you about this webinar series a few weeks ago?

Well, I had an application that is a version of this from before.

Okay.

But I wanted to build a complete— I wanted to build one completely from scratch so that I could document the entire workflow process, which is the real kind of— I think the real, you know, value of—

Oh, absolutely. Yeah. I mean, like, yeah. So you didn't— it's not like, you know, you came up with the idea in a vacuum. You have a lot of years as a practitioner in the Amazon space. But really the guts of it is that you didn't actually build Next Dollar from— it didn't exist, to your point. It was from scratch like 3-ish weeks ago, and you got to today while starting a new job, while preparing this presentation, etc. Um, yeah, that's mind-blowing. And I feel like if nothing else, like, that is something that like the— this entire call can internalize, and it's going to change the way that they pay attention to the rest of this presentation.

Okay, good. I built it in 4 days, but including all the research and everything. Including building the presentation about the app that I built. But anyway, yeah, so getting back to it, so yeah, you do a bunch of research and this is what the research looks like, right? And then, so just jumping back in here, once you have this research, right, what does it look like to compress research? Well, I'll show you. I'm just gonna skip to the end here of the presentation and show you a couple of different You know, you can look, you can look at creating visualizations from the research, right? So when you create, you know, the— there's a saying, a picture is worth a thousand words, right? A picture is worth a thousand prompts. And so if you create, if you get that deep research out of your LLM, I would recommend that you ask the LLM to build you a bunch of slides documenting in a visual format, you know, what is the research all about. Another thing that you can do, okay, let me just kind of go back here. Another thing that you can do is you can, let's see here, where are we at? Yeah. Is you can build these overviews that have like these little flowcharts of how things are supposed to, you know, work, like the logic flows in your application, right? So you can ask the LLM to build you a visualization. Can you guys see this okay? I can zoom in, I think.

Please do zoom a little bit. It's, uh, we can see the general sense of it, but yes. And by the way, this is a good time for me to remind everybody here, yes, you will get a copy of the presentation later. So if you're trying rapidly to take screenshots as we go through this information-dense thing, you don't need to do that.

Yeah. So this is all driven through ChatGPT, right? You can drive it through Codex or Cloud Code or Cowork, but in this case, all of this is just generated with ChatGPT. So again, you can just build your, your loops and your workflows and all that, and that all fits into the compression stage. Step 2, right? You're just trying to boost the signal and dampen the noise from your research stage. So then we go to 3, which is align. So what does that look like? Well, now I'm gonna take you into Codex, okay? And I'm going to take you to what a grilling session looks like. Now look here. Can you guys see my screen?

Yep.

This is how, what it looks like to align with an LLM, or in this case, a coding agent. Um, okay, so I brought in the research, right? You can see I brought in all the compression here of all the, the features, um, and I brought in the deep research reports that I did in step 1. And then, um, I used something called Grill with Docs, and I said, hey, let's align on this context, ask me 100 questions. I just chose that number randomly. You can have it ask you as many questions as it wants by not specifying a number, or you can specify some other number. I've had grilling sessions— I don't recommend this, but I've had grilling sessions on complicated full-stack marketplace applications that I've built out. I've had them go for 450 questions until the LLM is totally satisfied. These are multi-day grilling questions.

See, uh, yeah, sorry, real quick, Jack, when you say, uh, you know, use something like Grill with Docs, and when you show that here, uh, can you give folks a specific example of what that means and how they could do that?

Yeah, so that's what exactly what I'm going to do. So the LLM starts to ask you questions. What should the project— where should the project source of truth live, right? And then it gives you a recommendation and a reason, and then you can agree with it or disagree with it, right? What, what should I call this thing? Call it that, right? And then you might agree with it for a bunch of questions, and then at some point you might disagree with it, and then you have an opportunity to correct it before you've built anything. And align on that shared understanding. So here it says, what is the single wow moment for the demo? My recommended answer, the branded ROAS trap scenario. Reason, do you agree? And then I said, I think it's a great start, but we should go one level further or wider and also show the same type of thing for other types of campaigns, blah, blah, blah, blah, blah, right? And then it, it aligns on all of that context. And what does that look like? Well, you generate— it generates a context .md file. And look here, you have all the language that you've agreed on. You have all the definitions of the, the terms of art and the metrics and all of this useful stuff that you need before you start building, so that the thing you get at the end is actually the vision that you had in your mind before you, before you started, or maybe just the vision that you built together with the LLM. Sound good? Everybody with me so far?

Yes. So just to kind of go back to my question, I see how you walked us through how to interact with something like the grill skill, but for those who maybe are a little bit further upstream, is that— how do they even get that grill skill into their LLM or Claude or what have you?

Well, I will send all of the resources that go along with this talk I will send them to you, Andrew, if you want to distribute them out to webinar attendees. But just so people can see, there's a guy named Matt Pocock. He has a skills repository on GitHub and the Grill Me and Grill with Docs skills are in here. So you can literally give this URL to your coding agent, Claude Code or Codex, right? And just tell it in regular English to say, hey, I want all the skills in this repo. Install them for me, or clone the repo and install the skills, and it will do that. And so this guy has created these skills that I love so much, which are the Grill Me and Grill with Doc skills. Those are not my skills, but they're open source, and I use them all the time. They're, they're really great. And if you're really curious, you can look into something called domain-driven design, which is kind of the foundational framework and concept that he used to build all these skills.

Thanks. And if you've never done this for yourself, it really is as easy as taking that GitHub URL and giving it to your LLM and saying like, import this skill, and like, so that you can use it, and it will do all of the hard technical things for you. If you've never done that before, definitely give it a shot, and it'll be a good input to some of those sessions later this week as you walk through how to find interesting skills and whatnot. Cool.

Yeah, and if we have time at the end, I can show you what it looks like to call the GrillMe skill or any skill in Codex. But, um, just to keep moving on here, I just want to touch on where work happens, right? So as you go through this process that I listed above here, this workflow— research, compress, align, uh, plan, build, verify, and compound— that work is going to probably happen in different places, and that's okay. Um, I like to research with ChatGPT Pro I really like it, which means that work, that research work, a lot of it happens inside of ChatGPT Pro, 5.5 Pro. Sometimes though, I like to research with something called SignalSweep, which is a custom research tool that I built, um, and that work happens in Codex, right? So some research work might happen in chat, some research work might happen in Codex. It's just your preference how you like to work with these tools. You can also work with ChatGPT inside of— let me pull this up. Uh, nope, that's the wrong one. Let me pull this up. Yep, you can also work with ChatGPT inside of Excel. I don't know if people are familiar. Uh, by the way, Claude also has an Excel integration. If you switch this here to heavy, it effectively gives you that extended or extra effort thinking. And you can work inside of this chat window and it will take action in your— it will take action in your Excel workbook using formulas just like you would in Excel or using script. Like, I think there's an XML scripting language that it uses as well as some Python. So you can work natively with Claude and ChatGPT inside of Excel. You can work inside of your coding apps, you can work in chat, you can sometimes use MCPs and CLIs depending on your use case, or you can work inside of Cloud Design, for example, if you want to build pretty slides like, like these ones. I don't know, I don't know how pretty these slides are, but I use Cloud Design to build this presentation. So yeah, match the workflow stage to the surface, and I have recommendations that I'll share for where each step is best, uh, where I found it to be best, um, worked on. But you can always tailor that to your own workflow. All right, so let's dig into some of these steps a little bit deeper. So where does that initial research signal come from? There's, like I said, there's ChatGPT 5.5 Pro, which you can do deep research or, or just use out of the box as a kind of researching assistant. There's also a custom tool that I made called Signal Sweep, which I've created an open source version of it's called SignalSweep Open. This one I specifically created because I wanted to have not just deep research, but I wanted to have broad research that's very timely. So what SignalSweep does, it is it only looks at data from the last 60 days. So if something happened before 60 days, it just doesn't, it discards it, but it looks across 150 open and closed source data sources.. The open source version of SignalSweep has about 80 open source data sources that it looks at. Some of which you have to like OAuth into and whatnot, but they're all free. So I like to do that to expand the amount of research that I do. And then I also have this other open source repo that I'll share with you guys after this talk, which I call Scout. And the whole purpose of Scout is to help you identify— the whole purpose of Scout is to help you identify unknown unknowns. Okay. So the goal of Scout is to help you learn and identify things that might help you in your building process go from— go from this average output to a right tail, significantly better than average output. And because a lot of times when we work with LLMs, we are not subject matter experts at the thing that we're trying to do, right? If we're really pushing ourselves to build and create, we'll often find ourselves in a kind of beginner state, right? Because of that, we don't know what we don't know. And so it's really important that we have some way of identifying unknown unknowns and turning them into known knowns. And also it's important that we have a way of turning unknown knowns. What is that? That's tacit knowledge, right? It's knowledge that's somewhere in our ecosystem, but we are not consciously aware of it. We have to have a way of surfacing that as well. And that's another thing that SignalSweep does. It looks externally, but it also looks internally. Okay. And then how do you compress all that information that you get from step 1 there? You can use image generation. ChatGPT ImageGen 2, I think right now is really good. Claude Design is really good. You can use Excalidraw. It's a free tool that builds workflow diagrams. You can use ChatGPT 5.5 Pro to build you, um, condensed and compressed charts and visualizations and things like that. Let's see, I'm going to try to get through the rest of this pretty quickly. Um, then into step 3, we have the Grill Me and Grill with Docs skills for context alignment. We, we already talked about these. For planning, I really like to use Compound Engineering's Every— uh, by Every, which is the company— Compound Engineering's plugin and skills, CE Plan and CE Doc Review. This is the planning skill and this is the plan review skill. I like to use these and then go back and grill on them. So I'll use Compound Engineering Plan and I'll build the plan from all the compressed research that I've done, that I've done. And then I'll grill on the plan and grill with docs on the plan until everything is understood. Then I'll review the plan, right? And if it, if it needs another planning pass, I might plan again and grill again. It just depends on how complicated the thing that I'm trying to do is. But I'm really showing you here and stressing that a lot of effort and time goes into the research, the compression, and the context setting, the alignment, right? And the planning phase of all of this. No lines of code have been written yet. Zero lines of code have been written. All right. Then we build and verify. So again, how do we ensure that our plans are going to be built accurately? Well, we use— let's see here. Let me show you. So we, we might have something like a build brief, right? But honestly, this, this is really hard to enforce accuracy for an LLM if you use something like this. So we turn this, we turn this into JSON. Now look at this. Every feature has very clear evidence for, uh, what would constitute that feature being accurately built, okay? And you, you can use the planning, compound engineering planning, and you can use compound engineering, the plan review skill to build something like this for you. You can also ask Codex and Cloud Code to just build it for you right out of the box. But I find that the skills do a better job. But so the reason these all say pass is because they all got built. Right. But I did not build— I did not write this by hand. But what I did is I took my plan after I had grilled on it and aligned on all the context and reviewed it a bunch, and I turned it into this goals.json. And this is very easy, or I should say LLMs love this kind of stuff. It improves accuracy a ton. And as you can see here, this is a long document. Because every single feature has a test essentially that it has to, um, the LLM has to prove to itself via the verifying agent that, um, that it's correctly built.

Yeah. Did I see 20,000 lines of code in there?

Yeah, well, it's 20,000 lines of JSON, so yeah, fair. I, you know, I think it took maybe an hour to generate that document, and obviously I didn't write it. By hand.

But yeah, yeah, that's the main thing to emphasize throughout the rest of this, everybody, is that like Jack's not writing code here. He's, he's pointing at things to do.

Yes. Your job and my job is to continue to build this workflow and to continue to improve the workflow. People are starting to talk about loops. It's the same thing, right? I was never a software engineer, like a traditional software engineer. But there's software engineers now are saying like the job is changing and a lot of them are starting to say, my new kind of role is to build the workflows and build the ecosystems and the environments within which those workflows operate. And we do that building using English or whatever language, you know, you happen to speak, which is pretty cool. So the— just to close the loop here, We would build and verify using something like that goals spec document. We could then start to build against that long document using something like goal or loop, which are essentially loops like hooks, little bits of script that you, that you can give a regular prompt in natural language to the agent like Codex or Cloud Code, and it will just— it will generate a little script that says, keep looping on this, keep working on this until every single one of, you know, these tests is showing pass. And I've had loops go for 36 hours, or goals go for 36 hours before. As part of this process of building and verifying, sometimes I'll use computer use and browser use where the AI will take over my browser. You'll see it up here as a little pointer, kind of like you see on my screen right now, and it'll move around, it'll click things, it'll, you know, essentially simulate user behavior with mouse and keyboard until it's satisfied that the feature that it's working on is built correctly, and then it'll mark that verification block of JSON. As passing.

All right.

And then the last thing I'm not gonna go too deep on, but basically as this is happening, I have some scripts that I've built also custom to me that you can look this up. It's the concept is pretty popular now with Karpathy's second brain, but I built some script that just capture all the knowledge from successes, failures, learnings, And these are scripts that auto-run. They, they work on a timer essentially. And every couple of minutes they, or, or sometimes every, you know, 5 minutes, sometimes it depends on the script, but sometimes every couple of hours they'll look at what's happened in the coding session and try to figure out what learnings they can take from that. And they dump those learnings into Obsidian, right? So this is my Obsidian vault. You can see as I zoom in here just how much information exists in here. Can you guys see that okay?

I mean, inasmuch as it can be seen clearly, yes.

Yeah. And so the reason, you know, I have this much information in here is because this is a continuous automated process. I'm not doing this manually, right? Every time that I'm capturing something, it gets added and interconnected. Into this, um, I guess you could call it a second frame or a, a context graph. Okay, getting close to the end here, guys. Um, all right, so just to sum up, useful AI starts with context, not clever prompts. You're going to want to focus on research a lot in the beginning. And you're gonna wanna turn or uncover unknown unknowns and turn them into known knowns. And you're gonna want to have a process for doing that. You should be, you should have epistemic humility. You should be able to admit when you don't know something because the LLM is just gonna follow your lead. So if you don't have that humility, it's just gonna say, sure, you're absolutely right. And you're gonna get an average result. So you need to have some kind of process in your workflow for estimating accuracy. And in this situation, with this application specifically, the way accuracy is defined as the delta, the difference, right, the distance between the predicted outcome and the actual outcome. And then you want to align on context so that what you get out after building is what you, you know, intended. And then you want to have a build and verify very tight loop there so that you're not building in a vacuum. And so that all your tests that you're testing what you've built against are default failing. And you have to prove— the LLM has to prove to itself that it's passing every test for every feature that it's built. And this process can be parallelized, which, you know, we can we can take a look at if you want during question time, but I just want to kind of run through the last few slides here real quick. So this is the full, this is the full, um, workflow as, uh, actual steps, right? So this is what I did to build this app: deep research, uncovered unknown unknowns, more research, compressed, aligned, planned, closed gaps with more grilling, uh, wrote the goal prompts, built the product and verified everything continuously, continuous verification as the build was happening. And as far as time goes, guys, you can see here how much time is taken up by research and by compression and by planning, right? So phases 1 and 2, the research, compress, and plan and architect phase took up almost 90% of the time, of the actual time that I'm sitting at the computer interacting with the LLM. Now, I ran the build overnight, actually, and into part of the next day. So technically, like, if you want to think about it time-wise, it did take a bit of time, but I didn't have to interact with the coding agent very much at all. Basically not at all once all of this process was done. Now, I would be willing to I bet that almost 90% time spent on planning and architecting and researching is way more than most people spend on those parts of the process. So by the time you're done with step 1 and step 2, the code almost writes itself. You can then use all of that planning and research and compressed artifacts and all of those goal builds, you can then basically point that to Excel because it's the same model, right? It's the same, it's the same exact, let me see here. Yeah, so it's the same exact model that has taken in all the context of the app that you've built and you can just give it all of that context in an Excel plan, in an Excel build plan, right? You can see here how it's building out the same, logic workflow structures and table structures and everything here. And you can just insert that as an attachment into ChatGPT. 5.5 Pro lives here just like it does inside of Codex and inside of, um, ChatGPT on the web. And it'll build out the entire Excel for you. I had to go through this a couple times just as far as like auditing and fixing a few things, but building the Excel after building the app took probably an hour and a half at the most. Okay. So main takeaways for the app, just really quickly, the main takeaway that I'd like to leave you with, just something new to noodle on, is stop chasing attributed ROAS if you wanna introduce this new scaling vector into your advertising strategy on Amazon. And you wanna grow total contribution margin dollars, I would recommend that you start thinking about spend allocation across your campaigns and how you might optimize that. And for the workflow, you really wanna build yourself, whatever you're working on, a repeatable compounding loop, something that you can improve over time and something where you are taking the time to research and align with the AI and you're, um, you know, verifying what you build and you're compounding your knowledge. So remember, the system is the asset. The machine that builds the machine is where the real value is. All right, thanks. That's everything, Jack.

Holy cow. Uh, I, I don't know about everybody else here, but I feel like I need a drink and some time to process because that was extremely information dense and I really appreciate it. Feel free to leave up your question slide and with all your contact info in case folks want to get in touch with you later on. But I'm going to walk through some of the questions that we got while you were talking. And of course, folks, please feel free to continue submitting questions via Q&A or in chat. So the first and most obvious is everybody wants all the things that you just showed us today. And some of it we know is proprietary, but Just curious, you know, other than the recording and the deck, what else can folks get access to after, after the presentation today?

So I can share all of the recordings, the deck, the skills. So everything that you see here that I, that I shared, that's a skill or a plugin. So Scout, SignalSweep Open, I can share, you know, links to all of these skills like Compound Engineering, and the Matt, uh, Pocock, uh, skills as well as the, uh, builder verifier agent, um, setup that I like to use, which is called Lazy Codex. This is all free. Um, and then the next thing that I can do is for webinar attendees, I can share the optimizer part of the, uh, of the application, right? So this is just some sample data and I can share the link to this right now. It's password protected. So if you grab this URL and you're trying to get access, sorry, you you won't be able to access. But I'll share this after and you can put your own data in here and you can take a look and see what you get out of the optimizer. And if you are motivated, you can try to reverse engineer the application.

That's awesome. So you're gonna give us a whole lot, you know, not just the deck, the plan, the walkthrough, the mental models, the recording, but also a lot of the skills and even some of the sample code and whatnot., so that people can really go end to end with this. Um, that's amazing. Um, somebody earlier when I was marketing this commented like, it's kind of crazy that this is free. And I feel like this is a really good example of like, how is this free? So thank you. Um, another good question that came in from Scott Kennedy, uh, what pushed you over the line to build your own tech instead of buying SaaS?

That's a really good question. Um, I, I honestly think that this technology fits my personality really well because it rewards curiosity and a kind of, um, tinkerer's mindset. And so, um, specifically with this app, I used to manage at Bird Rock a very large catalog of products, and we cared most about total contribution margin dollars. So what pushed me to build my own software in this specific case, um, was not seeing anything that could do something like this for me within the Amazon space to the level that, you know, that I needed it, um, done. So, um, you know, I, I decided to use this as one of the first, um, test cases for building with AI. But in general, you know, I think at this point now I value the, I value the benefits that I get when I build my own tools almost to the point now where it's my default. Because the thing that you'll find, just to, just to quickly summarize here, is that as you get better at using coding agents and LLMs and your success rate goes up on what you do, you will naturally find yourself becoming a more general thinker because the, the tools, the machines, the AI, whatever you want to call it, it becomes really good at the specific. And it— I just love that feeling of becoming a more abstract thinker and building my own workflows, which then can be deployed to build my own tools.

Yeah, that makes a lot of sense. You know, once you— especially once you've achieved a level of not mastery, but something that sure looks like it to those of us who don't know any better. You, you can— the power of being able to just unlock things to yourself, customize, adapt to your own workflows, iterate, etc. is pretty hard to understate. One last one from Donna. How do you control things budget-wise? How do you avoid burning through a zillion tokens? Because we've all heard the horror stories.

Can you all still see my screen?

We sure can.

Alrighty, so Donna, I'm glad you asked. I have 76% of my weekly usage remaining and I have one reset available and I am on the $200 a month subscription plan. So I don't know what this application is worth in the open market 'cause it doesn't have any paying customers, but I built the whole thing plus the webinar plus all the materials for some fraction of $200. And I have that, you know, for the rest of the, for the rest of the week. And then it resets. And yeah, I, I have only run out of my $200 subscription once in the entire time that I work, have worked with Coding Agents. And I've done, like I said, 36-hour coding runs before. So I think it's, yeah, we can get into, that's like a whole nother webinar, how I like manage contacts and stuff like that.

Yeah. That said, what I will say is that Jack sits in like the greatest arbitrage and leverage opportunity when it comes to LLMs because he's an individual person who can access a Claude Max subscription. And you can get roughly, the estimate is something like $2,500 to $3,000 of coding credits through a Claude Max subscription. Subscription per month. So there is a little bit of arbitrage there for those of us who are in corporate or in larger subscriptions. It's gonna be a little bit more because Claude is not as generous with us, but it is still shockingly affordable. I rebuilt our entire website, including all of the underlying architecture, including rewriting everything that I wanted to through processes similar to Jack in about a week using my $100 a week Claude design budget. And, uh, didn't. And I went through, I think, like maybe $100 in token overruns.

So by the way, by the way, Andrew, I'm glad you brought that up. Real quick, just to touch on Claude Design, um, so Claude Design now has the ability to edit slides. If you click this little edit button here, you, you'll see all these, um, can you still see my screen?

Yep.

You'll see all these little, uh, dashed line outline boxes. If you do this edit function inside Claude Design, you don't burn nearly as much tokens as if you continuously prompt for changes on the slides. So my advice is, if you're running out of usage because of Claude Design, try messing around with the edit functionality. They now have pretty much full what-you-see-is-what-you-get drag and drop and full, you know, admin edit functionality like you would get in PowerPoint. Inside of Quad Design.

Jack, thank you so much. This was incredible. Again, for anybody who wants, there will be copies of this presentation and the recording available at a later date. For anybody who is joining us for our other sessions, looking forward to seeing you, but otherwise have a great rest of your day. And thank you, everybody.

See you guys. Bye-bye.

Take care, everyone.