Top 10 Learnings Architecting a High Throughput Critical API with RevenueCat CTO Miguel Carranza

Y Combinator Summer '18, and we are essentially an API and SDK for managing, managing in-app subscriptions, which means that we provide a backend as a service to app developers to monetize their apps using subscriptions really easily. We are currently on more than 5,000 apps, and the peculiarity of our API and why it has such a high throughput is mainly because it's not only used for the initial purchase It's also used continuously to check if a consumer is entitled to a particular type of content or not. So that happens essentially every time somebody opens an app that is using our SDK. So that means that we are considered critical infrastructure for these app developers and we cannot go down because if we go down, it's pretty obvious and it's really frustrating, frustrating for consumers. I also wanted to show how our throughput has evolved over the past few years, and we are currently doing about 400— over 400 million requests per day. But the interesting thing here is how it has grown. It hasn't been linearly. As you can see, between March 2020 and June 2020, there's this big spike. And these things can happen because an app can go viral without any notice, or a big app might just sign up with our service. So this is an extra challenge that we need to be super prepared to handle if we wanted to scale properly and guarantee the reliability. So what makes a good API? In my opinion, there are two main things that make a great API. Uh, the first one, like any other product, is the user experience. In this case, our user are developers, so I would say developer experience is the main thing. But even if it's great, you need— your API needs to be reliable because if you have downtime and you're down all the time, the experience is not going to be that great. So I tried to classify our 10 learnings into these 2 different categories. So yeah, let's start with the developer experience portion. Uh, the first one is documentation. This is super cliché, This is— but it's really underestimated. I would say that this is— if you get something out of this session, this should be it. Documentation is extremely important because it's the first point of contact with your developers, and if they see that your documentation looks great, they might consider and implement your solution. Otherwise, they're just gonna go and look for a competitor or try to build something in-house. So it is super high impact. And the reality— I'm not gonna lie, I'm a developer too, and I don't read the whole documentation all the time because I like figuring things out by myself. But developers like to know that you have their back and they can fall back to the documentation if they have a problem. Obviously, there are so many other benefits. It reduces customer support tickets, This is a good rule of thumb. If you see that there's an issue that is recurring in customer support, it probably means that your documentation can be improved. And depending on your product, it can also help for SEO. In our case in particular, since we are talking about very specific concepts related to in-app purchases, having really good documentation helps with discoverability. Another thing that is important is adding as many code samples and sample apps as possible in different languages. Not only because it makes developer life easier, it's also because you can be opinionated about it and you can show them what are the best practices for integrating your API. And the last point I wanted to mention here is that you should always be aiming for a world-class documentation, but this is not something that you do once and then it's done. It should be an ongoing process. Essentially a feedback loop with the customer support team. You should be always refactoring the documentation the same way you are improving the product and adding more features. The next one is really funny. We learned this one the hard way. It's unexpected use cases. Imagine that you have your API and a lot of people are using it and they're super happy with it. It's working as expected, but then all of a sudden at scale, you start seeing these random patterns or these really weird use cases in, in crazy ways you never imagined. Uh, the natural reaction is to feel like, oh, if everybody else is doing it, is using it correctly, and the documentation is great, as we talked before, then it's the developer fault. It's not my fault, I built it really well. But that's, that's not how you should be thinking about it. You should— it's really your fault, uh, for allowing these situations, and you should really protect against these scenarios, especially if they are affecting your customers' data. So the most common scenarios that we've seen were duplicated requests or out-of-order requests. A simple example would be, I don't know, like trying to make a purchase before the user exists or before the SDK is initialized. So you really need to be protected against these cases. Uh, some things that we did were trying to protect against all the race conditions and trying to make our API idempotent, which is a fancy term for saying that, hey, if you get the same request multiple times, uh, the result should be the same, uh, without any weird duplication or anything like that. Next point is consistency. Uh, if If there is a type of user that is hating consistency, they are developers because they are super logical people. And you should pay extreme attention to detail when designing the API. And this is not only the obvious parts of your API, like naming and things like that. And obviously you want it to be consistent across your endpoints, but I'm also talking about across your whole company and across all the products you support. We made some mistakes here. The most common ones were we used to have different date formats for the API and the webhooks, our third-party integrations. Other things is time zones too. We were showing the time zones in UTC for the API, but we were using it in your local time zone in the web. Web dashboard. So those things can be a little bit annoying and frustrating, and certainly you need to— you, you will know that developers are the type of users that will notice these things. So yeah, try to be as consistent as possible. Uh, next one, this is arguably the, the most controversial one, but I wanted to touch on deprecation. I'm not saying that you're gonna make mistakes, as, as I said before, You should try to improve them, but you should really have a plan if you wanna deprecate something in your API. Just think about it the opposite way. Imagine that you're writing some code using an API. You write it today and 5 years from now, the code still works perfectly. That is an amazing developer experience. So that's what you should be aiming for. And again, there's so many tricks that you can do. You can force your new users to always use the new version and try to encourage all customers to upgrade. But I would say that the best thing to do is try to support the old version as long as you can and give extended, really extended notice and heads up so that they can have the opportunity to upgrade to the new version of your API. For us, it's particularly tricky because we're an SDK. So it's not just updating your code. It might mean pushing a new version of your app, so we need to be very mindful about it. Internal versus external endpoints. Uh, here what I wanted to highlight is that it's really amazing and really good to have a centralized and unique API that you're using both internally and externally. There are so many advantages, like, uh, you only have one API to maintain, uh, you are dogfooding, um, and yeah, you're sharing the logic everywhere. But the problem is that sometimes it might make sense to split some components, especially if, for instance, we saw that, uh, there were some endpoints for the SDK that required to do some kind of operations with more parameters than usual or with a different response because of caching or whatever. And maybe that's not the best use case for other people consuming your API. So at that point, I think you shouldn't be afraid of sometimes splitting a little bit. And I think the rule of thumb here should be always prioritize the consumer use of views. All right, let's jump to the reliability section. On core rotation, this is probably the biggest mistake I made. I suggest and really encourage you to set on-call rotation as soon as possible. My thinking originally was that, okay, I'm the founder, so if somebody has to be annoyed by crazy alarms in the middle of the night, it should be me. And I try to be very protective with the, with the engineering team. That turned out to be a mistake because what ended up happening is that both my and my co-founder were carrying our laptops 24/7 and we couldn't sleep well and all of that. Instead, I would suggest you to implement on-call rotation as soon as possible. This can be really intimidating and overwhelming for your employees. So yeah, just try to make sure that there's good training and documentation for people to be prepared. Maybe pair with them during their first rotations. Uh, yeah, follow best practices. Uh, try to make sure that your alarm signal-to-noise ratio is good. You do blameless postmortems. This is very important. Automate all the things. If something breaks, uh, try to always build something so that it never happens again. Because you, you might be tempted to rely on people's memory, but we're humans and we make the same mistakes multiple times. So just try to make sure that there's some automatization to, to avoid that. And yeah, another thing that was really, really interesting and a side effect of being a fully remote company was the multiple time zones, that can help drastically. And we can all now sleep much better because we know that while we are sleeping, there's somebody else working on— working during working hours for them. So yeah, try to leverage the multiple time zones because it can really improve the quality of life of people on call. Next one is simplicity. This one might be counterintuitive when you are designing something for scale, People tend to think, oh, I need to have this super complicated architecture to make sure that we scale. But the reality is that when you're trying to find product-market fit, that probably shouldn't be your concern. And there are also so many benefits of starting with a simple architecture. It's easier to hire, it's easier to onboard, it's easier to get a better knowledge of the tools you're using. For instance, to give you some context, our original version of of Remini Cats API was essentially Python code and a PostgreSQL database. That was it. Nothing fancier. No, a lot of microservices, no, nothing. That was it. And this is great because it was much easier to onboard people. And also it helps you really understand when, how far these tools can go, because when you're scaling that fast with these weird spikes that we used to have, There are things that are gonna break in the more— in the most unexpected ways that they are not shown in textbooks. And that's why I also always suggest to use proven boring technologies instead of fancy unused new technologies. Because yeah, when things break in these crazy ways, it's much better to know that there's a support network of people that have used it at scale and you can rely on them. Versus being the first one. Uh, this is another one that might be counterintuitive, which is do not early optimize. Uh, you're thinking that your API needs to scale a lot, so you might be tempted to, when you're developing it, uh, to write the most efficient code. And don't take me wrong, you should do that, especially if it takes you the same amount of time to write the right code. But the reality is that you never know how it's gonna be used So it's much better to always profile and optimize for the most common case. This is an example from one improvement that we did, and after the deployment, the response time for API was reduced by half. And this was mainly because we analyzed the traffic and then we realized that there were some really big apps that have a lot of freemium users, so most of their users were— didn't have any purchase. So we optimized for that particular case, and then we, we got this, uh, good— this great improvement in the API response time. Uh, so, so yeah, sometimes it's— if you, if you try to optimize for something that is not the most common case, you're not gonna get— you're gonna— not gonna get any improvement. So it's probably not worth your engineering time. However, that doesn't— that shouldn't stop you from always be thinking about which one is going to be the next bottleneck or the next single point of failure, uh, and try to predict. It is a great exercise for your whole engineering team. Um, it can be anything. Most commonly it's a database because it's a centralized, uh, point, uh, single point of failure. But it can be your servers, it can be your caching layer, it can be anything. And I'm saying that you should be doing that this constantly and reevaluating because that gives you better leverage and you're going to be able to plan and rehearse any possible migration. And trust me, it's much better to do this when you have time to do it versus when the house is already on fire. You're going to be— it's going to be less stressful for everybody and you're going to be able to test different approaches. And the last point I wanted to touch on This is probably the most— the trickiest one, but with the cloud, we used to think that, that everything is infinite. Resources are infinite, we can scale forever, but that's not really true. There are always limits, especially when you're dealing with computers. There are always limits. So I would suggest to always think about them earlier and not assume that everything is going to be infinite, especially things related to your product like rate limits. Size of certain columns in your database, the size of the metadata that developers can attach to your API endpoints, all of those things. Uh, why? Because it's very difficult to correct once it's in production. The tricky part here is that you never know if the limits are going to be too high or too low. Uh, so yeah, there's some techniques to decide, but I would suggest that Try to come up with some limits that are reasonable because it's much easier to increase than to decrease them once you have users. Because once you have users and they are used to use your API in a certain way, it's like the preparation, right? It's going to be very difficult to, to reduce them.

Cool.

So my top takeaways here is that when you're designing your API, you should treat this as, uh, pay extreme attention to detail, treat Treat your APIs as you would treat any other product of your company. Manage it the same way and don't think it's a second-class citizen because your API is the way your developers are going to interface with your company. Always be obsessed about the developer experience. Make sure they want to use your API and they love your API. And once you have them using your API, the key is to, to be to have it work consistently and in a very reliable way. Thank you very much, guys.

Great, thank you so much, Miguel. This has been a great session. We already have quite a few questions in the queue, so I'm going to get going. Awesome. First question is, when is the right time to pull the plug on your old version? What do you look for?

Well, I mean, ideally nobody's using it. That's the ideal scenario, right? I mean, it really depends. If it's something that is really broken, I think communication with the customer is the first thing and letting them know, hey, these are the advantages of improving it, especially if it's something that is broken. Sometimes it's just only an advantage for the company and not for the developer, so there's no incentive for them to take the work and upgrade. In that case, I would always try to batch it with something that is really meaningful for them. So that they feel compelled to do it. But yeah, I don't think there's a good answer here. But as I said, I would always try to leave it on as long as possible.

Great. And just a reminder, everyone, please drop in your questions for Miguel into the question portal within the event portal. Next question, Miguel, for you is, how do you have a 5+ person on-call team in early days?

One second, I'm gonna stop sharing my screen. That's, that's, uh, I mean, that's tricky, obviously. I would say that maybe that is the case in which maybe you can stretch the founder a little bit more, especially if the team is early, so that people don't get super overwhelmed. But yeah, I think that's something that we noticed, that at some point there were not many people on the on-call schedule and it was we were using one-week rotations, so being on call multiple weekends a month is not ideal. So that's why I say that maybe 5 people is when it started to get good. But yeah, I mean, at the beginning, obviously the founders are gonna have— and the technical founders are gonna have to be on call for a period of time. But once you start hiring— what I'm trying to say is that you shouldn't delay it, and we did it for too long. We did it for probably longer than a year, and we should have done it a little bit earlier. Got it.

Okay. We've actually got this question a few times already. So, um, what stack did you use? What monitoring tools do you use? And at what stages did you upgrade this?

Yeah, that's a good question. Uh, well, let's start with the stack. Uh, stack we were using, as I said, Python and PostgreSQL. That was it, essentially. Uh, we actually had, uh, Our servers were in Heroku because we had some credits from Y Combinator, and our database was in AWS. And now everything is in AWS. We use New Relic and Honeycomb for monitoring mostly everything. The cool thing is that right now, the way we have it set up is we had redundancies on many of the instrument— many parts of the instrumentation, which is great for critical infrastructure. It was kind of a side effect, but it's good because if something goes down, like Cloudflare goes down or something major goes down, chances of having a backup for seeing what's going on in your API is really amazing. So that was something that— that's why we are still using New Relic and Honeycomb, even though maybe we could just use one of the— or the other.

Got it. Okay, another one.

Yeah, sorry, so the last part of the, of the question that you didn't answer, uh, yeah, the sooner the better, I would say, uh, especially when you're, when you're starting getting traffic. You really want to— you don't want to be flying blind.

Makes sense. Um, another question regarding limits and capacities: what's the right buffer? 10 times is often hard to achieve in practice.

I guess, is it in terms of scaling? I guess, yeah, I guess that's the right thing. It's an order of magnitude, right? For us, the way I normally think about it is, well, there's a thing. You can do at a big scale, you can do a lot of load testing, but sometimes that's not gonna be enough because the load is gonna be very high anyways. And things break in the most unexpected ways. Like, things don't break linearly. It's not like, oh, this— the CPU is going up and then all of a sudden— no, it can— it can— something can happen and the memory might go to the limit and things like that. So, but yeah, I would, I would say that at least try to align it with the business goals, more or less. Like, if you're expecting to— it is not a good rule of thumb because it really depends on the customer and all of that, but If you're planning to triple the revenue in a year, make sure that your API is able to at least triple in capacity.

Fair enough. Next question: What were top unanticipated issues once your API got massive, i.e., almost Stripe, like in scale so quickly?

Yeah, well, many things. So many, many things. I guess for us the biggest thing is that it's pretty spiky because since it's mobile apps, most of the interaction with our APIs is through our SDK. So if an app goes viral or an app has a big campaign to millions of users at the same time, they can— they end up doing API requests at the same time to our servers. So that might cause some issues for us. So that's when we try— in order to solve that, we try to be a little bit smart about how we— because at the SDK level, we can control how those HTTP requests are issued. So we have some leverage there to optimize for that. But yeah, I guess the biggest thing was when there was one week that we were serving 2 of the top 5 apps in the App Store, and that was pretty massive. And we were seeing it real time how the throughput was going up. And yeah, that was, was very interesting time to, to be at the company for sure.

Next question, and again reminder, you can just drop in your question for Miguel into the question area on the right rail of your event portal. Our next question for you, Miguel, is If your API is mission-critical, what have you learned on SLAs, throwing errors, etc.? How do you handle unavailability for a mission-critical API?

Yeah, uh, that's a good one. This is one of the things that I think it's mostly cultural at the beginning. You need to let people understand. We used to do that from the beginning when even when was my co-founder Jacob and I and we have 2 beta users, we were like, yeah, there's 2 ways of going around this. The easy way, which is having a 2-minute downtime, or over-engineer a solution that would take us a week to do, but there would be no downtime. So that's, that's how you should think about it, I think. There are some blog posts and some others that we still need to write, some content about how we've done migrations that the easy way would have meant '10 minutes of downtime,' but we pull it off doing— without any downtime. And yeah, if that's part of the culture, chances of screwing it up are much lower. Still, disasters can happen, and I think there are things that— there are some things that you can do to try to improve the experience of developers in these situations. Main one is like Well, if we have an SDK, we can do some cool things on the SDK side to make sure that it's going to work even if our servers are having a hard time. Also, making sure that we have a way to replay the requests. Like if our servers are down or having a hard time, if we can replay them later so that even though there's a delay, the actual result is going to be the same, super helpful. What else? What, what, what else kind of things can you do? Or how we know— other things that we've done is like we rely heavily on Google and, Google and, and Apple's APIs. So there's been some times that they have been, they, they have been down for a long time. So us being in the middle has been really good for our developers because we know, okay, we've detected that Apple or Google are down. So we can bypass it and maybe just give you the subscription right away, and then we'll verify with Apple later if it's valid or not. And those are things that are great because the developer is not going to notice that Apple is down, and we're going to still provide a good service.

Got it. Great. Next question: How do you market the robustness of your API? 'Twilio used to go down all the time. Now it's enterprise-grade. How do you prove you can scale for larger customers?' Yeah, that's mostly a marketing question, I would say.

We're working on, on, uh, customer success stories. We should, uh, be launching one very soon. Uh, yeah, we're essentially is showing that we had an app that ended up going viral and became the number one of the App Store for a few weeks. So that means that if we were able to handle that, you're probably better off relying on us than on your team, especially if you're— and I think another thing is that with all the apps that we have right now, the volume is probably much higher than any single app can ever do. So that's, that's a good thing. We can do so many cool things like sharding or whatever and have an enterprise plan for you, which we haven't done yet, but we can. But yeah, in terms of more on enterprise sales, yeah, I guess it's a matter of like agreeing on SLAs and things like that. But yeah, I don't think— yeah, I haven't thought about that that deeply. For now, most people trust in us. Also, the word of mouth has been really good.

Great. Question, next question. How did RevenueCat get early traction?

I would say content. I think that was the main thing. And this is probably super relevant for the SaaS community. But yeah, when right before getting into YC, my co-founder Jacob wrote a lot of blog posts, specifically talking about how the Google and Apple APIs were broken. And how we solve— not necessarily how we solve the problem, but hey, this is what you have to do if you want to do subscription on mobile really well at an enterprise level. And essentially pretty much giving all the secret sauce and all the rocket science from our product away. But then people get to see it and it's like, oh, these people know what they're talking about. This is really good content. It's not trying to sell me anything. And I probably don't want to handle this myself because it looks like it's a lot of work. And that with word of mouth helped a lot at the beginning. I would say that was probably the, the biggest thing. And obviously another thing is just, yes, being in the community and talking with developers and understanding their pain points. And yeah, as soon as you start getting— well, another thing is that this is a problem that we had already solved before at a previous company So we knew more or less what the customer wanted, but you never know 100%, right? You know your solution, but maybe there are so many different types of apps, so you need to constantly be talking to them and reprioritizing your roadmap. It's sometimes really cool to see, okay, they are asking for something that is in the roadmap. We are not— our direction is not super, super off. But still, you need to recalibrate constantly.

Okay, another question for you, Miguel. How do you differentiate an API from a smaller competitor with a seemingly similar feature set?

Well, a few things here. I think the documentation and the developer support are They're probably the main signs. Also, if it's open source, you can see how often it's updated and if they pay attention to the issues and how active they are in the community. And obviously, who uses it. That can be— that can be— and that's probably— that's not the only thing because sometimes your use case is different to the bigger company or to the indie developer. And sometimes you never know. Sometimes you see this this indie developer that you think is very small, but it's actually a really big subscription app. But yeah, I would say I, as a developer, what I would look at is mainly the documentation, the level of support, and if it's open source, I would definitely look at the source code.

Okay. Okay, another question. How do you balance team time on new features versus bugs, debt versus infrastructure? That's the first part of the question.

I mean, it's really difficult. It's really difficult. I think, as an example, last year we focused a lot on infrastructure and very small features. This year we have a way more aggressive roadmap, mainly because we made all these infrastructure improvements that are allowing now allowing us to now focus on the big features. It's hard. I would say at the beginning, at the beginning, you should do whatever the customer wants and go as fast as you can and iterate it. And, but yeah, I think things that helps are having a clear roadmap for the year. That helps. And another thing that we have been learning now is that most of our developers We actually don't have any expert in infrastructure reliability. We've been learning along the way. Most of our engineers are more product-focused, so it's been like, oh, some sprints are more focused on infrastructure, other sprints are more focused on product. So I think the key here is to try to find— to start once you're scaling, find teams or find people that are really focused on one particular problem. And again, it should be a shared responsibility between everybody. It's not like one person is going to be the person on call or the person in charge of it. But it's good to have somebody that is following up on all these postmortems and things like that that keep happening so that you can always keep iterating on the product as well as making sure that infrastructure is holding up.

Thank you. Well, we have some great questions, Miguel. Thank you so much for hosting this session today. I guess I'll let any last moments come through. We got another question coming through. Okay, here we go, Miguel. What table stakes for security at API level today? What are bigger customers asking for? SOC 2, data residence in Europe, etc.?

Yeah, this is— we are currently working on certification. It makes sense now, especially because we are trying to go for more— we have a sales team now, so we're talking with more enterprise customers. SOC 2 is a big one. I've been doing a lot of security questionnaires lately. I mean, with these things, it's like, I don't know, there are so many, many questions that some of them are really outdated too, especially with the cloud. Some of them, some of these questions are more focused on having your own data center or your servers at your office or something like that. But yeah, I mean, I guess security is super important here, especially when you're dealing with third-party data. You, you— I would say that these are the two main risks for a company like ours. One is reliability, and the second one is security. Because I don't know, maybe, maybe you can go down once in a big way or have a big security breach, but you shouldn't be relying on that people are going to be super forgiving about it because it's it has a reputational hit. So yeah, I don't think— I mean, I think the certification is really important just to prove it and so that it gives assurance to the customer that you're doing the right things. But I think it's mostly focusing on having that part of your culture and really doing industry best practices and And I don't know, having a bug bounty program, penetration test, things like that to make sure you're always up to date.

Great. Another question. What are the top infrastructure and other improvements you are thinking about to scale another 10x?

Well, that's a good question. I think it depends on the stage, right? One of the good things, as I mentioned, is that right now we are in a much better position than we were a year ago or 2 years ago because we know the tools so much better. And when we— you know, never know them 100%, but still you, you kind of know, okay, this is potentially what's the next thing that can break. So you're always thinking about these things in terms of, okay, what would it take to improve this? Uh, I think Uh, the magic is when you have an architecture that it will— even if it's temporarily, you can scale it up by just throwing more money at it, right? That shouldn't be— that shouldn't be your initial goal or anything like that, but if you have something that you can scale on demand and take advantage of the cloud, then you are in a pretty good position. But that's not always— not always the case. You might be— that's why we started with a very, very, very small database that was compared comparable to a Raspberry Pi, essentially. That was great because we were aware of how much we could achieve with that compute power. And then you have your— you have a better sense of how much you can scale it versus like if you're overprovisioned from the beginning, you don't really know what's going on.

Great. Well, thank you again, Miguel, for today's session and all the answers to these great questions that we received. Thanks again for joining us at SaaStr Build, everybody, and see you soon.

Awesome.

Have a great afternoon.

Thank you for having me.

Thanks, Miguel.