Automating Customer Support with AI Agents (How Gumroad Does It)

Cool. Awesome. Well, yeah, I just wanted to try to get back into the rhythm of hosting classes for small vets. And we have a lot of cool stuff that we've been working on. We're doing an onsite. So last, last week and this week, we have the team here and we've been working even some alumni. One of them, though, does also have a cold, so he's not currently in the office, but you can see our office over here. Different angles. And yeah, one of the big things we've been working on is automating customer support, not just automating it, but hopefully actually improving it as we automate it, making it faster, improve the quality. And so yeah, I just thought it'd be fun to kind of go through it with folks because I think it is really important. I think it's sort of like where code generation was like 2 years ago. Where I think now it's sort of all everyone talks about. But 2 years ago it was like, you know, Clock 3.5, Sonnet, and, you know, we're kind of pretty early, I would say, to that wave. So anyway, let me share my screen and I'll go through some slides and then I would love for people to ask questions and we can kind of share as much as we can about how we work and what we've implemented and the software. That we've built, etc., etc., etc.

Cool.

So let me share my window.

Cool.

Yeah, so the big focus of this is talking about how we basically use AI agents to automate Customer support, 99.8% right now, roughly. We get a lot of tickets, like 363 tickets a day. And kind of honestly to my own surprise as well, we've been able to do a pretty good job automating quite a lot of it and making the user experience of doing it like pretty awesome, I think. Doing customer support has like never been so fun in my opinion. Unfortunately, things are fun and then they're completely automated, so you're not even doing the fun thing anymore, but so it goes. But anyway, yeah, so this is the kind of the before picture, pretty typical support experience, right? Like every, everyone has sort of probably seen this before if you have a business. People email you and then you have to answer their questions and it can be in different languages. It can be about payments and refunds and Data deletion requests is becoming more and more popular. Now you have like AI tools that like help people, you know, kind of do that. You have like Stripe payout issues or someone changes countries or more bugs and you have to diagnose if it's like an actual bug and all of these kinds of things. And each one of these tickets like could be 15 minutes, you know, 30 minutes, like they're pretty complicated. And with support you have like a long tail, so it's not like you have 99 tickets that are all exactly the same. There is sort of like this long tail, um, of tickets. So that's the before. This is like the after. This is like what happens now today, uh, in the Gumroad universe, which is these tickets come in through Gmail. We have support@gumroad.com. Anyone can, can email us, partly why we get a lot of support. Anyone can send us an email.

And it's great.

I think it's way better than having like a live chat or anything like that because anyone can email from any device and you have, you know, track record of what you've sent and all sorts of awesome stuff like that. But typically people don't like to offer email because it's almost like too easy, right? To create, right? You have to go through some flow before they allow you to like email them and, you know, things like that. You may want to collect more information from the user. That you would do in a form, right? These kinds of things. But now with our setup, it's just way better to like let anyone email us and then Gumclaw like does all the hard work that normally would happen, you know, by the, by the person creating the ticket or, or the human who's like, okay, fine, like the customer didn't give us the information, but it's fine. I can like figure it out, right? And, and solve it. So yeah, just, just like a sample of them. But basically these tickets, like even 3 months ago, we would have had a really hard time with solving them. There are typically things where it has to kind of like start an action, wait for the action to finish, listen for the action to finish. It could take 3 days, you know, and it could, it needs to do like one of the, one of the most complex tickets that I really thought we would need humans for a long time is like account verification. But actually account verification is like actually very good for AI to automate because it's like, it's quite annoying as a human because like basically the more stuff you ask for, the more time it takes to review this stuff. Versus with AI, you can sort of ask for a lot of stuff and have AI review all of it, right? And get to a much higher degree of confidence. For example, if someone's bought a bunch of products on Gumroad and they have access to like the dates in which they bought these products, they can send it to us. And even if they're not perfectly accurate, we can get really like, you know, yes, they, they, they know the, the dates that they bought all these products, or they don't. Versus before, a lot of it, a lot of what we'd focus on would be just more like the actual identity of the person, like their name and, and, and address. But that's not that hard to, you know, get someone's information. So I think this is actually like way better. And so yeah, that's kind of the after, and this is how it works. Setup for us where basically tickets are coming to Gmail. Actually show you, I'll go down and we'll come back. So basically every, every, every Gmail helper is our internal SaaS that we built that listens to Gmail. Every, you know, ticket that comes in gets an immediate webhook. Kind of like you've set up something with GitHub or Slack webhooks, it's similar to that. Helper basically pings, pings Gumclaw through this webhook. Gumclaw starts acting on this ticket.

It's—

Gumclaw is a Hermes agent that runs on a MacBook Pro in the office and has a ton of skills and reference files and things like that. I think right now it's like 1,400 or something different files. And then it takes actions and verifies its actions and has access. I think the part of the secret sauce is that it has actions to a lot of things, which we could add here in a, in a separate slide, but, but, but it has access to effectively anything a human software engineer at Gumroad would have access to. But even more so because it also has access to what other roles, every role in the company, you know, basically is, is intermediated via Gumclub, right? So software engineering, customer support, finance, etc. So Gunpla has access to a lot of information, access to the number database, access to Stripe, access to PayPal, has a file system, has GitHub access, etc. And so it can kind of, you know, do quite a lot. I would say that the biggest thing it cannot do is probably pay taxes to the government. And hopefully, you know, we can make it easier. And so that also should be something that can be completely automated. For anything that it can't auto-resolve, it can auto-resolve almost all tickets, as mentioned before. For things that it can't auto-resolve or that it feels like it needs human escalation, it'll create a GitHub issue. So engineers who are living in GitHub, you know, can just stay in GitHub and you just get a ping and it's smart to assign. So all the things that I would basically have to do manually, as a human now can happen, you know, by these issues like being created by gumclaw, right? So gumclaw will just go in, create an issue, do provide basically all the information such that like all I'm doing is effectively opening Claude code or gumclaw and just saying, cool, like go through the tickets and I'll help you whatever you need from me, right? Like, and then you just hope that of these 4 tickets, maybe 2, 1 or 2 to 3 or 4 of them, you can update the skill such that it's basically autonomous the next time around, right? So you can see, for example, there's like a couple, couple tasks around purchase harvesting where people are, because we've automated so much, people are trying to like use it kind of as an attack vector. And so we just have to just make sure that, I mean, this is good in the case because there are basically a lot of these cases are people pretending that this has happened to them as a sort of a way of getting the purchases moved. So it's actually, I think, good that Gumcloud's not doing it and it's sort of basically asking a human to double-check. And then sometimes they'll autocomplete and you're just using it as a log of what's happening. But yeah, it's pretty cool. And I think tickets get solved a lot faster when you have, you know, if you're living in the tools that the people are already living in, right? Like if you're like, hey, you have to use this new thing, you have to check your email and all this sort of stuff, you're gonna have a hard time like getting to like super fast responses. But if people are already living in GitHub and that's now where this opens up, you know, and it's already in the flow of someone using Cloud Code or Thumbclaw or whatnot in their workflow, you can get pretty epic response times. And we published a lot of this data in our last public annual meeting. But this, so you can kind of do a comparison, but it's gotten a lot better. I mean, like, pretty crazy, like the complexity of tickets and how fast they can get solved. And I almost think of it like a little bit like server response time where like, hopefully in the future, like, we'll be fighting for, like, how do we shave, like, 100 milliseconds off this response time, right? Like, oh, it takes time to, like, connect to Stripe and get this information, or, like, 6 API calls to piece together this profile of a user from these different services, or we have to, like, do two-factor auth here or whatever, and, you know, push more and more of the complexity into the front. I mean, I would love to believe that like the most complicated ticket could come into Gumroad. It may even require fixing a bug that requires deployment and like all of that should happen within like 15 minutes or something like that, right? Right now, a lot of these long tickets that take, I mean, they're still pretty fast. A lot of it is just waiting for Stripe or waiting for the customer to respond. You know, I think a lot of that we can try to, we at least our part, we should be able to do super, super, super fast. So yeah, so here's an example of a GitHub escalation. It's assigned to the right person. It's really easy just to be on your phone and almost like Tinder, just like swipe left. I'm curious if anyone has questions or thoughts or wants to share their own experience automating customer support. Can I ask, are you using— you said Skilz. Are you using Claude as the main agent to behind everything? And does that get expensive with the API calls? I mean, if you can share that, I'm just curious. I'm always like looking to save tokens and stuff, so I'm always kind of curious. Yeah, I guess to be clear, uh, Claude is just a model. Uh, so we— our agent layer is Hermes agent. Hermes will— has access to Claude Opus 4.8. And CodexGPT-5.5-XI reasoning, those are the two models we tend to use. But super cheap, honestly. I mean, like, the token spend is— I mean, it's negligible. And now that we've basically got to the point where, like, these models are able to answer, like, effectively 100% of tickets, like, the bottleneck— like, we have enough intelligence in these models to do customer support. For governance anyway, right? Like, and Flexon. So the exciting thing there is that like now we begin the cost savings, right? Where like basically we don't really have to— we could probably keep using the same model and just like hopefully the token cost goes down over time. But yeah, I would say on average, you're, you know, with like a lot of like maximal token usage, you're probably spending $1 to $2 a ticket on a, on a, on a solve. So for example, with Gumroad, we have about 6,000 tickets that come up, 10,000 tickets a month that come in. Not that much, you know, like a level 5 self-driving company. It turns out nothing more than what we have. So whether we have AGI or not, I don't know where that is, but certainly like there's nothing, nothing that AI can't do. That I, that I do. So yeah, what else? You have to give it access to a lot of tools. You know, for example, we built our own wrapper around Gmail called Helper, helper.ai, where if you go, you can kind of see, let me share my screen again. Show you just from the homepage. You know, You have to give it access to, like, the ability to do these things, right? So, like, if it can't do these things, all it's gonna be able to do is create a GitHub issue for a human to do these things, which is not that useful in my opinion. Like, the human in the loop is, like, not that great in my opinion. Like, you wanna move to beyond the human in the loop as, you know, as fast as possible. And yeah, so, so, so For example, you need to be able to query the production database, right? Like, you are not going to be able to answer 99.99% of tickets about your product offering. If, like, for example, when you hire a support engineer and they're doing customer support, they'd use the console, right, to, like, debug the issue, figure out what's wrong. Like, so if you don't feel comfortable giving access, uh, the AI to your production database, like, just buy a cabin in the woods, you know, like that's like you're not participating in the future. Hopefully you can afford a cabin in the woods or build a cabin using wood. Anyway, yeah, so, so yeah, that's super critical. You know, obviously, as I mentioned, Stripe, PayPal, you know, Resend, SendGrid, like literally like just look at what you give access to a human who would join the company and You should basically remove that human's access. And then we use GitHub for kind of like product management, right? So we used to have a lot of this, for example, functionality built into Helper itself, like assignments, save replies, all sorts of stuff. And basically I was like, we should just delete all. We should just collectively like get rid of most of this. So for example, we had Helper Agent, and then we, which was our own thing, which was supposedly, as Bruce mentioned, like we want to be efficient. So like, let's have like a really stupid agent really fast, like answer like help center ticket type stuff. And then like anything that he can't, you have Gumclaw solve, right? So every 15 minutes, Gumclaw. Now we just deleted that entire layer. And we just say, actually, like, Gumclaw can just answer everything. And it's gone way better, right? You have way less like, you know, bad responses that then have to get corrected, etc., etc. So like, it's sort of, we don't even have the con— we used to have the concept of like a VIP, right? Someone who makes above a certain amount of money. We don't have that concept anymore because everyone is a VIP. Like everyone should be treated as if they made $1 million on Gumroad. It doesn't really matter. Like before we would have a queue of tickets, we'd have like 5 full-time people and we'd still have a queue of tickets because we had a lot, you know, and it would take a long time to answer all these tickets. And so you had to select, right? You had to say, okay, we're going to start with the person who made $10 grand this month, then the person who made $9 grand this month. And then you hope to get that number low. Right, as low as possible. But the goal is like every single person who has a question gets an answer, as if, you know, we, you know, like the perfect answer you can't improve upon. And that's what you can do now, right? Like you can, you can, I mean, it's fun just to see the, the Telegram of like just kind of watching it do its work because you can kind of see like, oh wow, this, there are all these kinds of tickets that I would never answer. You know, because I'm lazy and annoying. But it's awesome that Gumclaw is taking the time. Like someone's like, for example, like I, you know, I sold a PDF on Gumroad years ago and I have a new version and I'm scared that if I upload the version that people who have the old version will lose it, you know? And Gumclaw can give a really, really nice like multi-paragraph answer within seconds, right? That, and that person, it's not like a live chat, it's like someone emailed, and then 5 minutes later or what, however long it took, they get a reply back via email. And it— you had both OpenClaw and Hermes. So Tim, could you say a little bit more, maybe you covered it before, like why you switched to Hermes and why you like it more? Yeah, Shako, do you want to take that one? Because you're responsible.

Um, yeah, sure. So we first started with OpenClaw and it was working. Really well actually, but we started having some issues as we started adding more skills. It started becoming a bit unstable, especially as multiple team members were talking to it at the same time. It would constantly get confused and not really know who was talking to it. And it was also constantly breaking when we were updating OpenClaw. Uh, yeah, so, so yeah, so we just decided to, to test it out in parallel at first, and the experiment turned out really well.

Uh, I said no a couple, like 3 times. I was like, it's like Hermes. I'm like, uh, I don't really want to try. Like, I'll wait, let's wait like a month. Uh, and then we eventually did try it. I'd be basically, basically like, what was your things? Like, it was like a Mac or something.

Yeah, I have this analogy like comparing to Windows versus macOS where, you know, at least as a developer, macOS is a lot more stable, like it just works. And that's kind of like our experience with, with Hermes Agent where it's a lot more stable and it also has the added benefit of the self-learning loop. So it automatically updates its memory and scales. So that's super useful.

Yeah, I'm trying to see if I can show you guys an example of it.

Yeah, I can.

Why not? Share window.

While you pull that up, I'm curious, speaking of memory, which of the memory providers do you use with Hermes? I played a little bit with it and there's a whole bunch of different ones.

Someone— we don't use any. We don't use any.

Okay.

Yeah, we just use like markdown files in a GitHub repo and that's it. We don't use anything special to do indexing or whatnot. Sorry, for some reason my FaceTime camera just imploded. But yeah, anyway, you can see my— and now it's connected to my phone. But yeah, you can see, like, for example, like, there was some things here where it was, like, sending these really weird robotic kind of messages. Sometimes it does, like, too many emojis and stuff. So I was basically like, rewrite all your skills. So, like, you should talk like a human. Like, I wanna— when I read your stuff, it should read like a human. So now you can see here, like, verified with Live Body Landed versus before it might be, like, verified in all caps, emoji, rocket emoji, whatever, right? Like, generally, humans don't talk like that. And if we liked people talking like that, we'd want humans to talk like that. So anyway, I had that. And then you, you know, you can see that basically at the, at the bottom, you know, it just does this auto patch, right? On these different things, because it missed, it may have missed something or, or, or whatnot. So it's a little bit like dreaming, right? Where it's sort of like acting on its own. That kind of, this is something I think only really works when you have a smart AI. AI, you know, because like you're not going to want— it would just get it wrong a lot, you know. You kind of don't want it to think for itself, but as the model gets smarter, you kind of want it to think for itself. It kind of suggests its own improvements and its own pruning and its own cleanup. Um, so that's really the