Top 10 Learnings Architecting a High Throughput Critical API with RevenueCat CTO Miguel Carranza

Combinator Summer 18 and we are essentially an API and SDK for managing in app subscriptions, which means that we provide a backend as a service to app developers to monetize their apps using subscriptions really easily. We are currently on more than 5,000 apps. And the peculiarity of our API and why it has such a high throughput is mainly because it's not only used for for the initial purchase, it's also used continuously to check if a consumer is entitled to a particular type of content or not. So that happens essentially every time somebody opens an app that is using our SDK. So that means that we are considered critical infrastructure for these app developers and we cannot go down because if we go down, it's pretty obvious and it's really frustrating for consumers. I also wanted to show how our throughput has evolved over the past few years. We're currently doing over 400 million requests per day. But the interesting thing here is how it has grown. It hasn't been linearly. As you can see, between March 2020 and June 2020 there's this big spike. These things can happen because an app can go viral without any notice or a big app might just sign up with our service. This is an extra challenge that we need to be super prepared to handle if we wanted to scale properly and guarantee the reliability. What makes a good API? In my opinion, there are two main things that make a great API. The first one, like any other product, is the user experience. In this case, our user are developers. So I would say developer experience is the main thing. But even if it's great, your API needs to be reliable because if you have downtime and you're down all the time, the experience is not going to be that great. I tried to classify our 10 learnings into these two different categories. Let's start with the developer experience portion. The first one is documentation. This is super cliche. This is. But it's really underestimated. I would say that if you get something out of this session, this should be it. Documentation is extremely important because it's the first point of contact with your developers and if they see that your documentation looks great, they might consider implementing your solution. Otherwise they're just going to go and look for a competitor or try to build something in house. It is super high impact. The reality, I'm not going to lie. I'm a developer too, and I don't read the whole documentation all the time because I like figuring things out by myself. But developers like to know that you have your back and they can fall back to the documentation if they have a problem. Obviously there are so many other benefits. It reduces customer support. Tickets is a good rule of thumb. If you see that there's an issue that is recurring in customer support, it probably means that your documentation can be improved depending on your product. It can also help for SEO in our case in particular, since we are talking about very specific concepts related to in app purchases. Having a really good documentation helps with discoverability. Another thing that is important is adding as many code samples and sample apps as possible in different languages. Not only because it makes developer life easier, it's also because you can be opinionated about it and you can show them what are the best practices for integrating your API. The last point I wanted to mention here is that you should always be aiming for a world class documentation, but this is not something that you do once and then it's done. It should be an ongoing process, essentially a feedback loop with the customer support team. You should be always refactoring the documentation the same way you are improving the product and adding more features. The next one is really funny. We learned this one the hard way is unexpected use cases. Imagine that you have your API and a lot of people are using it and they're super happy with it. It's working as expected. But then all of a sudden at scale you start seeing these random patterns or these really weird use cases in crazy ways you never imagined. The natural reaction is to feel like oh, if everybody else is doing it, using it correctly and the documentation is great, as we talked before, then it's the developer fault. It's not my fault, I built it really well. But that's not how you should be thinking about it. It's really your fault for allowing these situations and you should really protect against these scenarios, especially. If they're affecting your customers data. The most common scenarios that we've seen were duplicated requests or out of order requests. Simple example would be, I don't know, trying to make a purchase before the user exists or before the SDK is initialized. You really need to be protected against these cases. Some things that we did were trying to protect against all the race conditions and trying to make our API even potent. Which is a fancy term for saying that hey, if you get the same request multiple times, the result should be the same without any weird duplication or anything like that. Next point is consistency. If if there is a type of user that hate inconsistency, they are developers because they are super logical people and you should pay extreme attention to detail when designing the API. This is not only the obvious parts of your API like naming and things like that. Obviously you want it to be consistent across your endpoints, but I'm also talking about across your whole company and across all the products you support. We made some mistakes here. The most common ones were we used to have different date formats for the API and the webhooks, our third party integrations. Other things is time zones too. We were showing the time zones in UTC for the API, but we were using it in your local time zone in the web web dashboard. So those things can be a little bit annoying and frustrating and certainly you will know that developers are the type of users that will notice these things. So yeah, try to be as consistent as possible. Next one. This is arguably the most controversial one, but I wanted to touch on deprecation. I'm not saying that you're going to make mistakes. As I said before, you should try to improve them, but you should really have a plan. If you want to deprecate something in your API, just think about it the opposite way. Imagine that you're writing some code using an API. You write it today, and five years from now the code still works perfectly. That is an amazing developer experience. That's what you should be aiming for. Again, there's so many tricks that you can do. You can force your new users to always use the new version and try to encourage all customers to upgrade. But I would say that the best thing to do is try to support the old version as long as you can and give really extended notice and heads up so that they can have the opportunity to upgrade to the new version of your API. For us, it's particularly tricky because we are an SDK, so it's not just updating your coding, it might mean pushing a new version of your app. So we need to be very mindful about it. Internal versus external endpoints here. What I wanted to highlight is that it's really amazing and really good to have a centralized and unique API that you're using both internally and externally. There are so many advantages, like you only have one API to maintain. You are dogfooding and yeah, you're sharing the logic everywhere. But the problem is that sometimes it might make sense to split some components, especially if, for instance, we saw that there were some endpoints for the SDK that required to do some kind of operations with more parameters than usual or with a different response because of caching or whatever. And maybe that's not the best use case for for other people consuming Your API. So at that point, I think you shouldn't be afraid of sometimes splitting a little bit. And I think the rule of thumb here should be always prioritize the consumer use of use. Right. Let's jump to the reliability section on core rotation. This is probably the biggest mistake I made. I suggest I really encourage you to set on call rotation as soon as possible. My thinking originally was that, okay, I'm the founder, so if somebody has to be annoyed by crazy alarms in the middle of the night, it should be me. And I tried to be very protective with the engineering team. That turned out to be a mistake because what ended up happening is that both my and my co founder were carrying our laptops 247 and we couldn't sleep well and all of that. Instead, I would suggest you to implement on call rotation as soon as possible. This can be really intimidating and overwhelming for your employees. So yeah, just try to make sure that there's good training and documentation for people to be prepared. Maybe pair with them during their first rotations. Yeah, follow best practices. Try to make sure that your alarm signal noise ratio is good. You do blameless post mortems. This is very important. Automate all the things. If something breaks, try to always build something so that it never happens again. Because you might be tempted to rely on people's memory. But we're humans and we make the same mistakes multiple times. So just try to make sure that there's some automatization to avoid that. Another thing that was really, really interesting and a side effect of being a fully remote company was the multiple time zones. That can help drastically. We can all now sleep much better because we know that while we are sleeping, there's somebody else working during working hours for them. So, yeah, try to leverage the multiple time zones because it can really improve the quality of life of people on call. Next one is simplicity. This one might be counterintuitive. When you are designing something for scale, people tend to think, oh, I need to have this super complicated architecture to make sure that we scale. But the reality is that when you're trying to find product market fit, that probably shouldn't be your concern. And there are also so many benefits of starting with a simple architecture. It's easier to hire, it's easier to onboard, it's easier to get a better knowledge of the tools you're using. For instance, to give you some context, our original version of Revenuecat's API was essentially Python code and a PostgreSQL database. That was it. Nothing fancier. A lot of Microservices, nothing. That was it. This is great because it was much easier to onboard people. And also it helps you really understand how far these tools can go. Because when you're scaling that fast with these weird spikes that we used to.

Have.

There are things that are going to break in the most unexpected ways that they are not shown in textbooks. And that's why I also always suggest to use proven boring technologies instead of fancy unused new technologies. Because when things break in these crazy ways, it's much better to know that there's a support network of people that have used it at scale and you can rely on them versus being the first one. This is another one that might be counterintuitive, which is do not early optimize. You're thinking that your API needs to scale a lot, so you might be tempted to, when you're developing it, to write the most efficient code. And don't take me wrong, you should do that, especially if it takes you the same amount of time to write the right code. But the reality is that you never know how it's going to be used. So it's much better to always profile and optimize for the most common case. This is an example from one improvement that we did, and after the deployment, the response time of our API was reduced by half. This was mainly because we analyzed the traffic and then we realized that there were some really big apps that have a lot of freemium users, so most of their users didn't have any purchase. So we optimize for that particular case and then we got this great improvement in the API response time. So yeah, sometimes if you try to optimize for something that is not the most common case, you're not going to get any improvement. So it's probably not worth your engineering time. However, that shouldn't stop you from always be thinking about which one is going to be the next bottleneck or the next single point of failure and try to predict. This is a great exercise for your whole engineering team. It can be anything. Most commonly it's a database because it's a centralized point, single point of failure. But it can be your servers, it can be your caching layer, it can be anything. And saying that you should be doing that this constantly and reevaluating because that gives you better leverage and you're going to be able to plan and rehearse any possible migration. And trust me, it's much better to do this when you have time to do it versus when the house is already on fire. It's going to be less stressful for everybody and you're going to be able to test different approaches. The last point I wanted to touch on this is probably the trickiest one. But with the cloud we used to think that everything is infinite, resources are infinite, we can scale forever. But that's not really true. There are always limits, especially when you're dealing with computer. There are always limits. So I would suggest to always think about them earlier and not assume that everything is going to be infinite. Especially things related to your product like rate limits, size of certain columns in your database, the size of the metadata that developers can attach to your API endpoints, all of those things. Why? Because it's very difficult to correct once it's in production. The tricky part here is that you never know if the limits are going to be too high or too low. There's some techniques to decide, but I would suggest that try to come up with some limits that are reasonable because it's much easier to increase than to decrease them once you have users. Because once you have users and they are used to use your API in a certain way, it's like the preparation, it's going to be very difficult to reduce them. Cool. My top takeaways here is that when you're designing your API, you should treat it as Pay extreme attention to detail treatment. Treat your APIs as you would treat any other product of your company. Manage it the same way and don't think it's a second class citizen because your API is the way your developers are going to interface with your company. Always be obsessed about the developer experience. Make sure they want to use your API and they love your API. And once you have them using your API, the key is to be to have it work consistently and in a very reliable way. Thank you very much.

Thank you so much, Miguel. This has been a great session. We already have quite a few questions in the queue, so I'm going to get going. First question is when is the right time to pull the plug on your old version? What do you look for?

Well, I mean, ideally nobody's using it. That's the ideal scenario, right? I mean, it really depends if it's something that is really broken. I think communication with the customer is the first thing and letting them know, hey, these are the advantages of improving it, especially if it's something that it's broken. Sometimes it's just only an advantage for the company and not for the developers, so there's no incentive for them to take the work and upgrade. In that case, I would always try to Batch it with something that is really meaningful for them so that they feel compelled to do it. But yeah, I don't think there's a good answer here. But as I said, I would always try to leave it on as long as possible.

Great. And just a reminder, everyone please drop in your questions for Miguel into the question portal within the event portal. Next question, Miguel, for you is how do you have a five plus person on call team in early days?

One second. I'm going to stop sharing my screen. I mean, that's tricky. Obviously I would say that maybe that is the case in which maybe you can stretch the founder a little bit more, especially if the team is early so that people don't get super overwhelmed. But yeah, I think that's something that we noticed that at some point there were not many people on the on call scale and it was. We were using one week rotations. So being on call multiple weekends a month is not ideal. So that's why I say that maybe five people is when it started to get good. But yeah, I mean, at the beginning, obviously the founders are gonna have. The technical founders are gonna have to be on call for a period of time, but once you start hiding. What I'm trying to say is that you shouldn't delay it. And we did it for too long. We did it for probably longer than a year and we should have done it a little bit earlier.

Okay. We've actually got this question a few times already. What stack did you use, what monitoring tools do you use and at what stages did you upgrade this?

Yeah, that's a good question. Well, let's start with the Stack Stack we were using, as I said, Python and PostgreSQL. That was it. Essentially. We actually had. Our servers were in Heroku because we had some credits from Y Combinator and our database was in aws. Now everything is in aws. We use New Relic and Honeycomb for monitoring mostly everything. The cool thing is that right now the way we have it set up is we had redundancies on many parts of the instrumentation, which is great for critical infrastructure. It was a side effect, but it's good because if something goes down, like Cloudflare goes down or something major goes down, chances of having a backup for seeing what's going on in your API is really amazing. That was something that. That's why we are still using New Relic and Honeycomb, even though maybe we could just use one or the other.

Got it.

Okay.

Another one.

Yeah.

Sorry.

So the last part of the question that you didn't Answer. Yeah, the sooner the better, I would say. Especially when you're starting getting traffic, you really want to. You don't want to be flying blind.

Makes sense. Another question regarding limits and capacities. What's the right buffer? 10 times is often hard to achieve in practice.

I guess. Is it in terms of scaling? I guess. Yeah, I guess that's the right thing. It's an order of magnitude right for us. The way I normally think about it is, well, there's a thing at a big scale you can do a lot of load testing, but sometimes that's not going to be enough because the load is going to be very high anyways and things break in the most unexpected ways. Things don't break linearly. It's not like, oh, the CPU is going up and then all of a sudden, no something can happen and the memory might go to the limit and things like that. But yeah, I would say that at least try to align it with the business goals, more or less, if you're expecting to. It is not a good rule of thumb because it really depends on the customer and all of that. But if you're planning to triple revenue in a year, make sure that your API is able to at least triple in capacity.

Fair enough. Next question. What were top unanticipated issues once your API got massive? That is almost stripe, like in scale so quickly.

Yeah, well, many things. So many, many things. I guess for us the biggest thing is that it's pretty spiky because since it's mobile apps, most of the interaction with our APIs is through our SDK. If an app goes viral or an app has a big campaign to millions of users at the same time, they end up doing API requests at the same time to our servers. So that might caused some issues for us. So that's when we try, in order to solve that, we try to be a little bit smart about how we. Because at the SDK level we can control how those HTTP requests are issued, so we have some leverage there to optimize for that. But yeah, I guess the biggest thing was when there was one week that we were serving two of the top five apps in the App Store and that was pretty massive. We were seeing it real time, how the throughput was going up. That was very interesting. Time to be at the company for sure.

Next question, and again, reminder, you can just drop in your question for Miguel into the question area on the right rail of your event portal. Next question for you, Miguel, is if your API is mission critical, what have you learned on SLAs throwing errors, et cetera. How do you handle unavailability for a mission critical API?

Yeah, that's a good one. This is one of the things that I think it's mostly cultural at the beginning. You need to let people understand. We used to do that from the beginning even when was my co founder Jacob and I and we have two beta users. We were like, yeah, there's two ways of going around this. The easy way, which is having a two minute downtime or over engineer a solution that would take us a week to do, but there will be no downtime. That's how you should think about it. I think there are some blog posts and some others that we still need to write some content about how we've done migrations that the easy way would have meant 10 minutes of downtime but we pull it off without any downtime. And yeah, if that's part of the culture, chances of screwing it up are much lower still. Disasters can happen and I think there are things that there are some things that you can do to try to improve the experience of the developers in these situations. Main one is like, well, if we have an SDK, we can do some cool things on the SDK side to make sure that it's going to work even if our servers are having a hard time. Also making sure that we have a way to replay the requests if our servers are down or having a hard time if we can replay them later so that even though there's a delay, the actual result is going to be the same. It's super helpful. What else, what else things can you do or have we done? Other things that we've done is we rely heavily on Google and Apple's APIs. So there's been some times that they have been down for a long time. So us being in the middle has been really good for our developers because we know, okay, we've detected that Apple work OS are down so we can bypass it and maybe just give you the subscription right away and then we'll verify with Apple later if it's a valid or if it's valid or not. Those are things that are great because the developer is not going to notice that Apple is down and we're going to still provide a good service.

Got it. Great. Next question, how do you market the robustness of your API? Twilio used to go down all the time, now it's enterprise grade. How do you prove you can scale for larger customers?

Yeah, that's mostly a marketing question. I would say we're working on customer success stories. We should be launching one very soon essentially is showing that we had an app that end up going viral and became the number one of the app store for a few weeks. So that means that if we were able to handle that, you're probably better off relying on us than on your team, especially if you're. And I think another thing is that with all the apps that we have right now, the volume is probably much higher than any single app can ever do. So that's a good thing. We can do so many cool things like sharding or whatever and have an enterprise plan for you, which we haven't done yet, but we can. But yeah, in terms of more on enterprise sales. Yeah, I guess it's a matter of like agreeing on slis and things like that. But yeah, I don't think. Yeah, I haven't thought about that that deeply for now. Most people trusting and also the worth of bounce has been really good.

Great. Next question. How did RevenueCap get early traction?

I would say content. I think that was the main thing and this is probably super relevant for the Saster community. But yeah, right before getting into yc, my co founder Jacob wrote a lot of blog posts specifically talking about how the Google and Apple APIs were broken and how we solve. Not necessarily how we solve the problem, but hey, this is what you have to do if you want to do subscription on mobile really well at an enterprise level and essentially pretty much giving all the secret sauce and all the rocket science from our product away. But then people get to see it and it's like, oh, these people know what they're talking about. This is really good content. It's not trying to sell me anything and I probably don't want to handle this myself because it looks like it's a lot of work that with word of mouth helped a lot at the beginning. I would say that was probably the biggest thing. And obviously another thing is just being in the community and talking with developers, understanding their pain points and yeah, as soon as you start getting. Well, another thing is that this is a problem that we had already solved before at a previous company. So we knew more or less what the customer wanted. But you never know 100% right. You know your solution. But maybe there are so many different types of types of apps so you need to constantly be talking to them and reprioritizing your roadmap. It's sometimes really cool to see okay, they are asking for something that is in the roadmap. We are not. Our direction is not super off.

But.

Still you need to recalibrate constantly.

Another question for you Miguel. How do you differentiate an API from a smaller competitor with a seemingly similar feature set?

Well, few things here. I think the documentation and the developer support are probably the main signs. Also if it's open source, you can see how often it's updated and if they pay attention to the issues and how active they are in the community and obviously who uses it, that can be probably that's not the only thing because sometimes your use case is different to the bigger company or to the indie developer and sometimes you never know. Sometimes you see this, this in the developer that you think is very small, but it's actually a really big subscription app. But yeah, I would say I, as a developer, what I would look at is mainly the documentation, the level of support, and if it's open source, I would definitely look at the source code.

Okay. Okay. Another question. How do you balance team time on new features versus bugs, debt versus infrastructure? That's the first part of the question.

I mean it's really difficult. It's really difficult. I think as an example, last year we focus a lot on infrastructure and very small features. This year we have a way more aggressive roadmap mainly because we made all these infrastructure improvements that are allowing now, allowing us to now focus on the big features. It's hard, I would say at the beginning, at the beginning you should do whatever the customer wants and go as fast as you can and iterate it. But yeah, I think things that helps are having a clear roadmap for the year that helps. And another thing that I've been learning now is that most of our developers, we actually don't have any expert in infrastructure or reliability. We've been learning along the way. Most of our engineers are more product focused. So it's been like some sprints are more focused on infrastructure, other sprints are more focused on product. So I think the key here is to try to find, to start once you're scaling, find teams or find people that are really focused on one particular problem. Again, it should be a shared responsibility between everybody. It's not like one person is going to be the person on call or the person in charge of it. But it's good to have somebody that is following up on all these post mortems and things like that that keep happening so that you can always keep iterating on the product as well as making sure that infrastructure is holding up.

Thank you. Well, we had some great questions. Miguel, thank you so much for hosting this session today. I Guess I'll let any last moments come through. We got another question coming through. Okay, here we go. Miguel, what table stakes for security at API level today? What are bigger customers asking for SOC2 data? Residents in Europe, etc.

Yeah, we are currently working on certification. It makes sense now especially because we are trying to go for. We have a sales team now, so we're talking with more Enterprise customers. SoC2 is a big one. I've been doing a lot of security questionnaires lately. I mean, with these things it's like, I don't know, there are so many questions that some of them are really outdated too, especially with the cloud. Some of these questions are more focused on having your own data center or your servers at your office or something like that. But yeah, I guess security is super important here, especially when you're dealing with third party data. I will say that these are the two main risks for a company like ours. One is reliability and the second one is security because I don't know, maybe you can go down once in a big way or have a big security breach, but you shouldn't be relying on that. People are going to be super forgiving about it because it's, it has a reputational hit. So yeah, I don't think, I mean, I think the certification is really important just to prove, to prove it and so that it gives assurance to the customer that you're doing the right things. But I think it's mostly focusing on having that part of your culture and really doing industry best practices and, and I don't know, having a bug bounty, program penetration test, things like that, to make sure you're always up to date.

Great. Another question, what are the top infrastructure and other improvements you are thinking about to scale another 10x?

Well, that's a good question. I think it depends on the stage. Right. I think one of the good things, as I mentioned, is that right now we are much better positioned than we were a year ago or two years ago because we know the tools so much better. You never know them 100%, but still, you know, okay, this is potentially what's the next thing that can break. So you're always thinking about these things in terms of, okay, what would it take to improve this? I think the magic is when you have an architecture that even if it's temporarily, you can scale it up by just throwing more money at it. That shouldn't be your initial goal or anything like that, but if you have something that you can scale on the mat and take advantage of the cloud, then you are in a pretty good position, but that's not always the case. That's why we started with a very small database that was compared to comparable to a Raspberry PI, essentially. That was great because we were aware of how much we could achieve with that compute power. And then you have a better sense of how much you can scale it versus like if you're over provisioned from the beginning, you don't really know what's going on.

Right. Well, thank you again Miguel for today's session and all the answers these great questions that we received. So thanks again for joining us at Saster Build everybody and we'll see you soon.

Awesome.

Have a great. Thanks, Miguel.