How we scaled 2M+ Amazon content updates without losing control | Amazon X AI Working Sessions

Alright, welcome everyone. For those of you who are joining us for the first time, I'm Andrew Hamada, co-founder and CEO of Reason Automation. This is the fourth in our series of Amazon AI working sessions. Without further ado, let me turn it over to our main leader here, Christian, who I know from many years now of just circling each other in the Amazon ecosystem and doing a lot of joint marketing and co-events.

Perfect. Good morning, Andrew Hamada and everyone on the webinar. I'm Christian, Christian Umbach, one of the co-founders of Autopilot. You'll find us under autopilotbrand.com. And I'm super excited to be here today to share with you a bit of our journey, specifically over the last 3 years of how we started with the idea of fully automating PDP updates, catalog updates, and sort of how that's gone and how it has scaled. We came into the space initially working sort of very deep on the data pipeline side around API integration, event streaming integration, and how to deploy machine learning models, and that led us into the Amazon world when around 2020, 2021, a lot of API integrations were needed. And one of the big learnings for us early on were very much around, I would say, kind of seeing where brand teams, brand operators spent their time and also where maybe they don't spend their time. And listing optimization tended to be in the second category. So that's why we picked it as a problem. We have spent the last 3 years almost exclusively focused on 4 fields: titles, bullet points, description, and backend search terms. That would have been crazy if you had asked me that 3 years ago, but it's, I guess, it turned out that way, and I'm happy to share a bit about our journey here today with you. Let me bring up a couple of slides to kind of guide us along. And yeah, we're going to talk about— we are going to— I hope you can see the slides. We're going to talk about how we scale to more than 2 million Amazon content updates and how that's evolved over time. Maybe briefly on— you see a lot of marketing claims out there around Autopilot. We actually named the company Autopilot, Autopilot Inc., and it goes back to a passion of mine very much around the aviation space. I used to spend a couple of years in, in the aviation industry, and innovation and auto like, or the autopilot in a plane pretty much delivers about 95% of the flight, of every commercial flight that you're, you're taking today. Um, that the pilots are kind of hands on stick and pretty much for, um, the taxing. Um, then, yeah, as you get to the runway, the takeoff and, and the landing. But pretty much everything kind of in-flight is delivered by the autopilot. And we kind of took that same analogy into the Amazon space to, on our end, kind of focus on the problem once the ASINs are up and running. We wanted to build out the perfect optimization suite for listing optimization and catalog optimization. Right now, it starts taking us to kind of the earlier steps of that as well. But for today, what we're going to focus on is kind of that core part of optimizing listings and doing that at scale. Maybe before we dive in and kind of share our learnings, I would just— I'd be super curious to hear a bit from your end. And I think, Andrew, you're gonna help us fire off some polls here to get a sense of like how big are— maybe as a first one, how big is the catalog that you're managing today?

The poll should now be visible to everybody. It's just 3 questions. The first is how many active ASINs are you managing? Second is how often do you refresh the listing content of your top 10%? Selling ASINs? And then the last is sort of the opposite of that. When did you last refresh the listing content of your long tail bottom 50%? I see answers flowing in. Christian, can you see the live poll results?

Um, no.

Okay, well, I will then— I will continue holding this open until I see answers slow down. And then I will let you know once I hit the end poll button, you should be able to see the results along with everybody else here. Perfect.

And then we can, we can pick this up in, in a minute here, then I guess as the answers are flowing in. I assume folks are seeing, seeing the answers to, to the questions. Yep.

All right, it looks like we've got an answer slowed down, so then I just hit end and there's your results.

Oh, let's see. Maybe I have to stop my screen share to, to see the results.

Oh, I don't know. It says, it gave me a button to say share results, and now it says stop sharing. So hopefully folks can see that. Thanks, Matthew.

For some reason I'm not seeing any results, but maybe you can briefly walk us through them if you see them.

Yeah, absolutely. So let me share that since for some reason Zoom makes it difficult to do this. That's annoying. Zoom also doesn't want me to be able to share that screen. Cool. Thanks, Zoom. Really appreciate it, guys. Okay. But what it's looking like is the majority of you from a catalog size perspective are managing catalogs between 1,000 and 10,000 ASINs. With the second subset at 100 to 1,000 ASINs. So everybody here managing some volume, almost nobody managing fewer than 100. So scale is obviously an issue for all of you. For number 2, how often do you refresh the listing content of your top 10% selling ASINs? The overwhelming majority answer was quarterly, with the others coming in with trickles compared to that. And then when did you last refresh the listing content of your long tail bottom 50% of sales? Almost everybody said longer or never or in the last year with a very, very small number saying in the last month.

Got it. Awesome. Yeah, thanks for sharing all of your input on that. Thanks for Andrew Hamada for sharing that. So going back 3 years in time, we, at the point in time, we basically deployed or we saw catalog data about across about 15,000 stores. And I would say across these 15,000 stores, the insight was similar or worse to what you just shared here this morning, which was that Yeah, on average, I would say when we spoke to folks, they said, oh, well, we update every 3 to 6 months, you know, at least the top sellers. I would say the reality was that oftentimes from a copy perspective, they were actually on average updated every 9 months, and pretty much outside of those top 10, and depending on the catalog, sometimes top 20%. There was hardly ever an update after the initial setup. So the way we went into kind of this entire space in the first place was seeing this data and understanding that the ones where refreshes were happening were significantly outperforming others. We thought, oh, there's something there, so let's go down that rabbit hole, or, you know, that rabbit hole, which turned out to be a very fruitful one. And, you know, as part of that, if you think about it strategically, obviously right now, you know, a long tail, 50% of ASINs, if you manage north of 1,000 ASINs, obviously 10,000 ASINs, that's a lot of products which are not getting, you know, attention from like a listing standpoint. And sometimes if you think, you know, there's oftentimes this concentration or Pareto effect around like, oh, 20% of our sales, or the 20% of sales, 80% of the catalog, but 20% of the catalog made up 80% of the sales. Sometimes it's in some ways a self-fulfilling prophecy. If the majority of the catalog does not get the level of attention. So for us, that was kind of a starting point at the very beginning. What we're going to dive in now is, I guess, a couple of, let's say, points of view around how we approached catalog optimization, and then we're going to go through the exact flow, the exact process that we built over time and how it evolved from a sort of automation supporting humans to in some ways human and automation kind of working, working hand in hand and driving this flow. Overall, when it comes to scaling the impact of catalog updates, really what you're weighing kind of against each other is speed. Yeah, let's automate everything. Let's update everything. Let's do it fast versus control. Like, how do we make sure it's compliant? It's on brand. It's actually live. And most importantly, around all of this is that the performance is right, that, you know, an update makes sense. And as moving the ASIN forward, moving the brand forward versus actually something that's hurting performance. The reason why catalog operations usually kind of breaks at scale is really around the complexity that kind of quickly, you know, quickly scales, which is even if you have 100 listings, We used to focus on 4 fields, now item highlights is coming in, so it's 5 fields, it's 10 child ASINs under kind of the core parents, so all of them have their listings. And even if you do just a few refreshes per quarter, that easily then adds up to be more than 10,000 content touchpoints.. And you can argue that's for, you know, one smaller brand. So, you know, think about how that looks at the scale that, that you're operating on your end, where you can kind of add a few, few zeros here to, to the mix. And the, the most, one of the, I guess, important notions for us that was that as we thought about the process of updating the catalog is, you know, it was oftentimes looked at as this kind of chore of like, oh, it's, you know, we have time right now, let's do a listing update. And there's keyword research, new copy being developed, and that's being published. And that was kind of the process. And what we found out basically initially through building an alerting system, which then took a bit of a different, different shape was that the, the reason to update is basically a specific event that, you know, we're able to detect. That event can either be specific Amalgam alerts or issues that they're flagging in your catalog. It could be compliance alerts. It could be different scoring elements that we've built around the The SEO part, maybe from a keyword perspective, the GEO side from a shopper intent perspective, could be specific movements around the keyword bank, then policy or field changes like Amazon saying, hey, here's an item highlights field now, and you have until July 27th to push an update, or otherwise we're gonna start pushing updates. A big one I think that is largely undervalued today is seasonality, and we're going to dive into that a bit later on as well. And then other triggers, basically, like detecting overrides, like did your content just get overwritten maybe by your own processes or by someone else, another seller, another contributor on the ASIN. And then obviously there's kind of the no update window. So, you know, today, kickoff of Prime Day, no real content updates should be made at that point in time. What we tried first, you know, also speaking with brands about this 3 years ago, you know, it was kind of that moment in time where GPT-3 was just coming out. We got early access through, through the API. We started playing around with that. But the, um, there was a lot of hesitancy still out there. Um, and I think some of that still exists today, uh, around, okay, like, let's be cautious on what AI is really doing here, um, in the process.

And just to help ground us, Christian, um, like, roughly what year was that?

This was, um, 2023.

So like, that just cracks me up because, you know, we're talking about, you know, oh, back in the days of GPT-3 and like, you know, when things were so different and yada yada, and really we're talking about like 3 years ago.

Yep, uh, 3 years ago. Um, for context, Rufus, uh, was announced in February 2024. Um, so that came like about a year after. Um, and obviously, I mean, no question that the models are, are a lot better today, but it was basically— if the way we thought about it was, okay, how do we build the rails such that as things get better, um, we're able to kind of plug it into those rails and, uh, and adapt around it. Um, the way we did updates in 2023, you could say, was essentially like Every update to, to every content field got reviewed, like went through human review, went extensively through human updating as well. So it was all, you could say, AI generated at that point, then human reviewed, human kind of intervened. Let's say we would capture all of those interventions as feedback for us. That's one takeaway we're going to get back to later. Log everything. So every feedback, every update gets logged. That was a big part of our development process as well. And at the very beginning, the measurement was quite naive. Like, basically, we were just looking at the traffic side, like, oh, Did it support traffic? Did it, you know, what did we see on the traffic side? But obviously, when it comes to traffic, there's a lot of other factors that are impacting that as well. So over time, we basically moved from this like, oh, let's, you know, take an ACE and let's generate an update and let's publish that update to a much more comprehensive process. And this is This 7-step process, pretty much what we are still using today, or what we've orchestrated and architected our system around today. And we're going to dive into those different areas now. And obviously, feel free to jump in with questions here if you want to go deeper. The first part, we already talked about that a bit, is kind of the trigger side. So every update is not a, you know, let's say not a time-bound event to say, oh, it's time to update the listing, but rather a result of, you know, these different elements here that are at play, which are giving us the reason to provide an update in the first place. Just before we dive into the funnel and this slide, maybe should have Wanted to bring it in a bit earlier, but we'll just add that as context because at the end of the day, we're going to dive into the operational aspects of looking at this particular funnel now. But the reason why this matters at the end of the day is that we see this as a profit-generating function within kind of the, your brand operations. And it's profit generating from sort of two different aspects. One on the revenue side, we have the controls around organic traffic in our hand in terms of making sure that through the right content, the right keywords, you know, the right shopper intent positioning, we can maximize the organic traffic side. I would say that there's a supporting means towards the advertising side. So by the way, here's some advertising opportunity that you're missing out on, or maybe you're overspending in certain areas. Conversion rate is the other kind of big controller for us that we're impacting. And then on the cost side, it's more essentially about bringing down, you could say, the cost of human listing operations and also the cost of returns on the cost of advertising reliance, or in other words, like improving your return on ad spend. So that just as, I guess, one more framework here to think about that. Listing optimizations means you have a number of controllers in your hand, and we're essentially dialing up and down these controllers with the updates that we're doing. So with that, let's jump into this funnel here. Stage 1 is the triggers, and that is kind of the trigger event here now for the different controllers to see what to update. Just looked at some of the trigger examples around compliance policies. But largely, I think, you know, for us, we drive about usually probably on average 2 listing updates per ASIN per month. And a large indicator there is actually kind of the keyword movements, which is usually representing at least one of those updates per month. So this stage really decides, okay, it's time for an update. Now, when it gets to putting together the actual update, think of this as gathering context. I would say the standard approach there, still, if you look at a— it's called VA playbook— is take the existing listing, do keyword research, and then use a tool or kind of build new copy from that. The system that we build on that side is a bit more comprehensive. It basically says, okay, let's take the current listing. We also ingest all of the different, whether it's product sheets that you have, your media asset library. We can kind of pull in some, for example, in the supplement space, maybe there are specific health requirements that are coming from a governmental website. Those for us are inputs around the product. We're building out shopper intent, what we call unique selling points, partially informed by reviews. At the end of the day, this section is really focused on, like, you know, if you look at the facts around a product, the outcome that this fact drives, and the differentiator against the market, and think about the different features of the products that you're selling, those basically make your unique selling points and We have that as a base because we want those USPs to be reflected in the title, in, you know, the item highlights, in the bullet points. Then other inputs are obviously Amazon requirements. There are going to be allowed terms and banned terms. Now, you know, your team will probably have their own little list of these are specific banned terms, or, you know, this is where we sit in the gray zone of using specific terms or not using specific terms. I would say it's not always a, you know, black and white in terms of like what is allowed and what's not allowed. There may be context for an update coming in from the parent itself or kind of across the different parents. Brand requirements, like style requirements. There's extensive demographics data on Amazon for, to better understand your personas and your shoppers as well. We're pulling in Alexa prompt performance. And one of the core areas, sort of last but not least here, is seasonality to understand, like, is, you know, the reason why someone is buying your product in May different from June, different from October, November, December. Um, and like around that, um, it's basically kind of an understanding of like which events are playing a role. Like, that's Mother's Day, Father's Day, back to school, uh, you know, summer vacation, gifting period. Like, how does that play a role? And, um, should impact the listing side. So all of that for us is context. And, you know, from a technical perspective, it's basically wiring up these different parts. And, you know, you can, I mean, if you do it for a handful of ASINs, you can probably have kind of skills within, within cloud, for example, around that if you want to rely on a system like cloud. If I would say you do that at a scale of like thousands of ASINs, you probably want to have more dedicated systems in that sense, or so the kind of components that can reliably perform.

This bridges nicely to our last session where Brett was really talking about how you can leverage other MCPs and whatnot to start to get at this data at scale, some of it anyway, like for example, leveraging an MCP to talk to Keepa's API and get a bunch of product data that way instead of needing to necessarily go scrape listings yourself. But without giving away the store, I would love to hear, Christian, like what are some of your, what are your thoughts on where people could go to get started with some of these data sources?

I think, I mean, certainly in terms of where to get started. So the, I guess, from a source point of view, there's actually a lot of great sources directly from Amazon. Like you're gonna get your current listing content from Amazon, obviously your reviews, you can, you know, the demographic data, the prompt performance data. I think those are more simple inputs kind of into this flow. You know, the product sheets that you have from maybe your media assets or your D2C sites, like those are all also great inputs kind of into this process. And you could say more easily kind of crunchable for the LLM to work. On the seasonality side, the way we built or we think about seasonality is very much informed kind of by keyword trends over time. So we essentially use search query performance data. We go back 2 years in time and look at that, how week over week for the last 2 years specific trends have evolved. And I think for, for that, the, well, the data gathering is a problem that is specifically for that, it requires a bit of a bigger system build out. And then just crunching through, through that amount of data is Yeah, maybe something that you want to have a little bit more control over than just an, uh, sort of an, an LLM going over the data. But I think, you know, if you want to get started, there are— I would say in some ways the biggest challenge is, okay, how do you on, on your end, you know, on your machine or on, uh, um, you know, your drive in, in your organization, How do you find a repeatable means of gathering that data, storing the data, and then actually kind of working against that data? Because at the end of the day, what you're putting in is absolutely determining, like, what is coming out of this. So you want to make sure that, you know, that kind of follows a consistent quality. And If you look at this at scale, and, you know, Andrew, you've built a business around that, and we're touching, I'd say, data quality extensively as well. You realize that ensuring that you have the right data to actually work from is one of the most important parts of maybe avoiding hallucinations and kind of ensuring that you're going the right way with the updates. So let's jump to stage 3, which is, um, yeah, now we're actually generating updates. Woo! Um, you know, that's what folks usually see as like, hey, you are doing listing optimization. Um, which for us basically breaks down into these 5 steps, which is, um, or so if the, the LLM with all this context that we just looked at, spits out a draft. The draft gets evaluated by a number of guardrails to make sure it's policy compliant and, you know, banned words compliant specifically for that ASIN or for the brand. It goes through kind of an auto-scoring mechanism where we score for shopper intent, we score for keyword placement, and mostly for length and so forth. And what this chart is missing, after auto-scoring, it actually goes back to draft if it doesn't pass the auto-scoring. So you can think of this as a, as a loop if it doesn't hit a certain score. And then the final step here is basically a human in the loop. Today, not for all of the content updates anymore, but for certain risk categories, you could say, whether it's title fields or specific industries. And after the human review, it can either also go either back to draft and kind of flow through this process again, or it passes. And then it's, you know, for us gets the status of, okay, this is ready for publishing and, you know, kind of ready to go from that perspective. The learning here along the way was that, I mean, at the very beginning, you kind of have a draft and the human. So yeah, those are kind of the two parts to work with. When we introduced auto-scoring, which is probably around like 1.5 years ago or so, we would see that it always takes a bit of time to kind of calibrate with things like that. And we're going to see— we're seeing this right now actually around introducing kind of shorter titles, introducing item highlight fields. Like, changes like that means there's a lot more extensive kind of iteration to see, okay, what is actually right and what should be passing through. But then this process basically delivers an update that is ready for publish, but it's not published yet. Then it—

So sorry, Chris, real quick. So just to emphasize, you're saying that like all 4 of the first stages here are happening automatically through AI based on the inputs you reviewed on the previous slides, and it's only until that human score adjusts step that a human is actually finally in the loop.

Correct. The one exception I would say is around, we have certain seasonality triggers where it's basically, let's say, an alignment with a brand on whether they want to like play in that seasonality, or sometimes it's done maybe a manual input to say, okay, like the brand has specific campaign that they're running. You know, I can— one of our customers is Warmies, for example. They have these amazing plush toys that you can warm up, but in the summer you can also actually put it into the freezer. So they're running a big coldies campaign. So that was for us more of a manual input that kind of then acted as a trigger. Where we could align with their D2C campaign. But pretty much all of the other steps, yeah, are more kind of happening on an ongoing basis.

Awesome.

And then the next part is, you know, then, and this is now, you know, we have the auto-scored updates, we have the human gave its listing expert gave us some thumb up here, then we run another set of compliance checks. And that is necessary because, you know, sometimes in the process either maybe there was still, you could say, a human change that, you know, could be maybe hasn't passed the first compliance check. But it's also, you could say, just an added safeguard to make sure that, you know, we haven't missed anything in the process. And in this step, we both look at, like, overall Amazon compliance policies, we'll look at industry-specific policies. So if you're selling beauty products, for example, like, you know, is the copy, the wording still Mocra compliant? And it's going through another kind of review of banned terms. Usually this is just a, like, okay, let's, you know, automatically check this again against our guardrails and we're good to go. But it has become an important part of just kind of ensuring compliance. And I would say that the We actually introduced this because what we found, and this goes back to a bit of the human role in the process overall, is that at some point we reached the point in our development where if we had errors or mistakes being introduced into the content, those were rather errors that were in— they were plugged in through human input in this process. Than through machine input. Um, so it kind of became a sort of protection about sort of, I guess, against that, and, uh, an added flag. Um, super interesting. Then, okay, now we, we have all the contents compliance checked. Um, it's from, from our point of view, it's ready to go. Um, the, the next step for us is publishing through API. We're not a big fan and have never been of flat files.

We—

the API, surprisingly, on the vendor side for publishing content updates is actually better than on the seller side. I would say for most other functionality, that's the other way around. But on the vendor side, we do have specific error detection and through the API where it will say, oh, by the way, you need to fill out this specific attribute in order for your content update to go live. On the seller side, it's more like, okay, thank you for updating your SKU. And the part that then where Amazon is a bit more difficult than maybe many other platforms is just because you updated your SKU, successfully does not mean that your content is actually showing on the ASIN page. So that's why we have a few more stages here, which is the next stage for us is to confirm that the update is actually live, live on the detail page. And, you know, we, once we detected live, we can put a timestamp to it. It's an event that we record. But in quite a few cases, let's say about, depends a bit on the brand, sometimes, you know, in the 10 to 20, 10 to 30% range initially when we start working on the catalog, we actually see that updates are not just going live like that. So in those cases, we basically then again have a workflow to run through republishing, do error fixing, and the error fixing on attributes. I guess that's still embarrassingly manual today. That's one of the next areas that we're looking into today. But there's oftentimes, unfortunately, still a need where let's say you go into Seller Central, you already see the attribute pre-filled in the field and you literally just have to hit save and that solves the problem. So it was kind of some communication problem, I guess, within Seller Central. And in the vast scenario of cases, like, that solves the problem and we can actually go ahead and publish. If that also doesn't solve the problem, then if teams are working with SAS or have a vendor manager, we may bring them into a loop or otherwise go through support to get specific updates out there and live. But obviously, that comes with more overhead then. So we have all interest of kind of getting through the republishing, through the fix that we have in our hand, get— getting this out overall. And now I know, you know, we're already in stage 7, but the, the most important stage of this entire listing optimization process actually starts now, which is not necessarily the— all of the content update that's going out, but is building an understanding of whether the update that we just published is actually performing better than the prior version and is kind of really a superior version. And we've built this out in a couple of different steps. I do think that when you start building on this problem, you're building on the initial steps first, like the parts that we just talked through to kind of get your update out. I do think that spending time on this stage 7 that we're going to talk about now is really critically important because, yeah, at some point you maybe want more resources, you want to scale up the process, and whether that's, you know, development resources or whether, you know, paying for tokens, someone is going to ask, okay, like, what are we getting out of all of this, and is it making sense?

And that's where—

we'll dive into that part now to kind of understand how are we measuring that, and how do we detect if things are going wrong with updates as well. In terms of going wrong, there's obviously different time horizons of measurement. Like from the moment of getting an update out, we're starting on our end basically a daily performance check for each update. So each of the 2 million updates has received this automated performance check. Which is essentially looking at page views, sessions, units, conversions, and is basically comparing the actual result versus a kind of a forecast window that we gave for, for the ASIN. So it's probably a little bit small to see here, but you can see on the right side, you know, for example, for sessions, kind of the different fields of how did the sessions evolve before we pushed an update and then, you know, where it gets color-coded and the update started. And we have an understanding of kind of those different stages: green, you know, amber, green, yellow, amber, red. And depending on where the update here lands, like over the first 7 days, over the first 14 days, that may actually trigger an alert for our system to say, okay, like something went off guard, off rails here, performance is really impacted, we need to intervene again. And this is more from, in some ways we still see this as publishing because it's kind of making sure that there's a safe update that's going out and performing. The, the next part around measuring impact is going to look at two different things. First of all, at market shares, and second, then around specifically kind of a bottom-up contribution of added sales, which we're calculating from the search query performance data. The reason why I guess we're going to do a little bit of a deep dive in our understanding here of our take on measuring performance is because a lot of, I would say, the established view of like, oh, we're updating your listing, so what are you doing as a next step? You're kind of starting rank tracking. Maybe you have some third-party tool for that to say, okay, like now for this keyword, we're, you know, position 4, or, you know, we were position 5 yesterday. So it's going in the right direction. We don't believe in rank tracking anymore at this point. So for us, we're not specifically tracking like the details of organic ranks. What we're instead doing is we're tracking the specific market share that your ASIN has on a particular term from a perspective of impression share, click share, add to cart share, and purchase share. And the reason why we're doing that is not because it's kind of more fine-grained along the funnel. I mean, that's a nice benefit as well. But it is really around your market share and the understanding of the size of the market and on your conversion rate of the market is much more powerful of an indicator to say, okay, this is actually the market that you want to play in versus seeing like, oh, you know, we have a keyword like insulated water bottle and you're positioned for this week, and last year, last week you were on a different position. And that is for us, this element of market share understanding is very deeply ingrained in our reporting, also on the one that we're going to look at kind of from a keyword impact point of view then as a next step. And yeah, you can— there's something obviously around the goals that we have. One is around driving traffic, the other one is around driving conversions to that traffic. Part of what this market share level tracking helps us do is understand, you know, is the funnel overall healthy or is it leaky and where does it start to leak? And like this example here on the right side of seeing like, you know, 12% impression share, 11% click share, but then only 5% add to cart share. That's an example of where like we look into those cases to really understand, okay, is it a problem of the content, or do we see some other factors going into this as well, like out-of-stocks, buy box losses, for example, where maybe your impressions and clicks still show up, but you actually don't have a real shot at, you know, winning an add to cart, winning a purchase here. So that's one point of view. And then on the other point of view, if we look at a healthy funnel, we can basically see what are the keywords that align with this healthy funnel. And then you can go one step further and say, okay, are we actually advertising on those keywords? And, you know, are we like, yes or no, do you want to advertise? But overall, I mean, the— an increasing share of clicks add to carts, purchases is kind of the ideal flow that you want and which we want to support with the listing updates as well. And this, you know, the underlying data for this here is not something that you need to, I would say, magically acquire, or someone needs to necessarily invent as a formula. Amazon actually provides that data through, for example, through the search query performance reporting. Maybe the challenging part is more like, okay, how do you really use the data in your flow? How do you make sure it's of the right quality and you have kind of the right point of view of like, do you care about the keyword level? The ASIN, like child ASIN level, parent ASIN level, the product line, and kind of having those different perspectives. And one reason why this is important to understand kind of this relation of market share and the actual sales then driven around market share is because when you have a, like in this case, this is for kids' snow boots here, where you have, I guess, crazy demand for parts of the peak, then there's kind of a seasonal gap that comes back up in the winter later. No one is buying snow boots in the summer. You know, if you look at it from a pure, maybe just a pure sales perspective, a pure traffic perspective, you're going to see those heavy fluctuations. But building a market share-based understanding actually helps kind of track of like, okay, are we making progress here as a brand that is selling this product? And are you growing faster than the market overall? And again, the underlying data here is what you can get through, you know, the search query details, like the Brand Analytics details that Amazon is providing. Now, that was sort of the final performance overall. What we're obviously, you know, even though there's a lot more shopper intent optimization happening now, keywords are still playing an important role because, you know, you'll see that on Amazon, you'll even see that or hear that from OpenAI and others. The reality on how specific queries are treated is kind of this fan out of specific components of a query, which still, you could say, translates into keywords then, or specific searches that these queries are generating. So for us today, tracking market shares on keywords, for example, is an important part of how we look at a listing and how we also track our updates. So, you know, this is here, for example, for a brand where we started driving updates sometime last year. And in this case, we specifically looked at the last 90 days to understand what was the impact from our updates. And the impact basically being comprised of— we opened new keyword markets. There was kind of a core opening of the market, which was which drove $135,000 in sales. We managed to grow market share then over time in that market, which unlocked another $250K in sales. And then the second part is more in some ways a kind of conversion rate focus, which is like, where did we kind of keep keywords in the listing or maybe move them up within the listing?, but we actually through those changes managed to drive additional sales as well. And again, this is like we would start looking at these updates here at these numbers from the moment we know that for— we introduced the keyword kids snow boot into the listing today. So we can, we have the average market share in the last 3 months and we can start tracking the market share now going forward. And put a dollar value to that. And then on the other hand—

Christian, we're coming up on time here. We got about 5 minutes left. And if you don't mind, there are a couple of really good questions that I wanted to make sure that we addressed live while folks are still here. If— do you mind?

Yeah, let's do it.

Awesome. So one I think is about attribution. Somebody asked here, you know, when you're trying to measure the impact of content improvement and keyword tracking and all this kind of good stuff, do you pause, or do your clients pause other optimization levers to do this, or how do you isolate for the impact of things like pricing, ad changes, etc.?

Yep. Um, so for, I guess, different things, we once in a while, we love to run sort of dedicated experiments, like through Amazon's A/B experiments, uh, specifically in so purely more isolated environments where then we're kind of running on the Amazon rails and there's no other pricing or image changes then happening on the ASIN during that time. Otherwise, we usually, we tend to go back in a specific review maybe like once a quarter or was every 6 months and look at specific cohorts of ASINs of like, oh, these, you know, were ASINs that had price changes versus no price changes. And, you know, within the no price changes, those had updates versus no updates. So we kind of try to dissect it from there. Given how dynamic the environment is, it's very hard to do, I would say, perfect controls. And that's in our learning why we had to move away from, you know, let's see what the impact on traffic is for your ASIN, to more of a kind of bottom-up understanding like the one that we're seeing here of like, okay, what this keyword got introduced, what happened to this keyword over time, did it grow market share? And then, you know, you could go a step further and say, okay, by the way, on on this specific ASIN, actually we had, you know, specific changes. So maybe let's remove those from the analysis that we're looking at.

Cool. That makes sense. So a little bit of like cohort testing, a little bit A/B testing, etc. But that it is difficult to completely isolate all factors at all times without going to that kind of scientific experimentation. Awesome. And then one last question that came through not formally here, but that someone asked in the last session was, Thinking about the catalog size question that we polled on early in this session and the fact that folks are mostly working with 100 to 1,000 ASINs, but more often 1,000 to 10,000. Of all the things you've talked about here, like what are the things that break or are the big challenges as you move from like 1,000 to 10,000 ASINs that you're trying to do this for?

I would say the, I mean, if you, I think the jump from 1,000 to 10,000 ASINs is actually pretty easy from an operations perspective if you're actually optimizing 1,000 ASINs. I think the hard part is the jump from, you know, 50 or 100 to 1,000 ASINs. I think that's the really hard part because the, like even if you have 1,000 ASINs today, you know, in reality you're probably just updating 100, um, every 3 or 6 months or less frequent. Um, and so I would think of it more as that step between how do you get from 100 to 1,000, and there, um, I think it's kind of this combination of like figuring out like when to drive an update, the right context windows, and again, finding what works for your team as well and for your management team to get the metrics right of, I guess, reporting on an impact and seeing, okay, these metrics are understood through the organization. If we talk about traffic, you have branded traffic, unbranded traffic, Maybe with broader catalog optimization, you're primarily driving unbranded traffic. How much of a share is that today? How much of a share is that in the future? Those are the type of questions that I would focus on. And because then once you have a smooth process for 1,000, like dialing it up for 10,000 or 100,000 is like, is rather sort of a a technical finesse of maybe getting some of the scaling right. And there you're going to be working then with, you know, more around problems like Amazon's API limitations around getting SGPR data that, you know, that needs a more refined system. But overall, it's, I think it's that 100 to 1,000 mark to focus on.

Awesome. All right, I think we are firmly at time, sir. So I know that we didn't get quite all the way through your presentation, but I do want to thank you very much, Christian, for coming on board and being the first to talk about what scale problems look like here. You know, obviously you're managing updates across a very, very, very large catalog, much larger than any one brand. So your insights here are really appreciated. Thank you everybody for attending. Join us again tomorrow for our next sessions, including my own, as a subtle, like, self-promoting thing. I'll be talking about how to manage context windows and work with really, really big data. And otherwise, thank you so much. Look for recordings and follow-ups by the end of this week. Take care, everyone.