Founder's Guide to Scaling Applications: When to Build, When to Buy and What Breaks

Bonsoir. I'm Julien, co-founder and CTO of Algolia. We develop a search API to help any developer to have a very good search in their application. But today I won't talk about search at all. I will discuss about build versus buy decision. And I will try to cover a few mistakes we did about those decisions. I actually think I never met any company that have not made a few mistakes about build versus buy, and most of the time it's linked to the long-term cost, the long-term estimation of what it means to build or to buy a solution. So I will first give you a few numbers about where we are, and when I will give you a few examples, I will try to explain where we were at this stage when we did the decision, when we took the decision, because of course the stage is important in such decisions. So today we are a distributed team. We founded the company in 2012 in Paris. We then flipped the company to the US. We are today around 350 people in 6 different offices. We have mainly a bare metal infrastructure with close to 3,000 servers distributed in more than 70 data centers, and we handle between 200 and 300 billion API calls a month. All of that was not built in a day. It took us a few years, and I will give the context for all the examples. Of course, behind this product, we have a super strong team. Like, none of that would be possible without a very strong engineering and product team. One of the very misleading concepts For me is that the better your team is, the more difficult it is to take the good build versus buy solution. I'm sure it will be very like misunderstood, like probably you won't see it at first. We all see the positive effect of having a great engineering team. Of course, good engineers can build a product way faster than others. Of course, it contributes and has a big impact, a key impact for your company. But we don't discuss enough about the negative aspects of a very strong engineering team. I think it's not something we discuss enough. First of all, I think any good engineer can easily give you 10 good reasons to build instead of buy. It's super easy. Like, you will never find enough arguments to buy. You can always have a very specific problem. You can always find arguments that you need a custom solution. You need to build it pretty much. And what I have seen is that most of the time, You can discuss, you can argue, they will come with a prototype in 2 hours or the day after, and they will tend to show you that it's easy to build. And they will always underestimate the long-term cost, the long-term impact of building versus buying. So I will try to discuss with few concrete examples and a few concrete mistakes. So some were positive, with good effect, and some has been negative for us, and we had to change. So I will cover them to give you a bit of insight about when we have built and when we have decided to buy. First big decision we took, it was in the very early days of the company. We were only two. We have not even fundraised. We had no salaries, so very, very early days of the company. We decided to use bare metal infrastructure and not use cloud infrastructure, VMs, which was kind of completely crazy at this moment. And a lot of people misunderstood us when we took this decision. It was, of course, risky, and it was a big bet. The reason we did this, we took this decision, is because of our bet on the market. Like, we were working on this field for more than 10 years before creating the company. We were convinced people wanted to have good performance, people were investing a lot of time to have good performance, but they were not reaching good enough results. So our initial bet was performance could make a big difference. We wanted to have a factor 10 on performance, which means we had to be crazy about performance on all the different aspects. Of course, the software, we need to build something which is really different in terms of performance. But the hardware, even the network, play a big role in the performance. Like, search engines are very intensive in terms of CPU, SSD, memory. So we had to pretty much select a very specific machine, very specific hardware to make sure we have the best performance of the market. None of that was available on cloud infrastructure at this time. So that's why we took this crazy decision to use Bare Metal. And pretty much we have to build everything from scratch. Like we distribute the data by a factor of 3. To make sure we have some high availability. You can do that on cloud provider with availability zone. You have a lot of features out of the box to do that. We have to build everything from scratch in software to be able to have high availability. And of course, at the beginning, we did not have the time to do that. So the first version of the software had zero high availability at all. It was one server running on like one provider, and it was on one side kind of risky because one hardware, one big hardware failure could cause us a lot of trouble. But it was the easiest way to get the feedback we needed to have. Like we have these big bets, we needed to get some feedback from the markets about these bets. Is it realistic? Does performance make such a big impact as we have planned? Or is it difficult to sell the software based only on a big difference in performance? So to imagine the product at this moment, the feature set of the product was super limited. We did not get even 1% of the features we have today. But we had a big difference in terms of performance compared to everything else on the market. And the reaction we got was super positive. Pretty much it confirmed our initial bets. So this decision was super risky. It was at a moment where it could have hurt the company, like a big failure, a big hardware failure could have hurt us in terms of trust. But we were in the initial days, and it has been positive. We validated our assumption. One thing, so even if we decided to use bare metal infrastructure, even if we decided to do everything ourselves, we did try to lower our investment as much as possible. So of course we use leasing and not colocation. We did not buy ourselves the hardware, assemble the machine and so on. Like we took some provider to do that for us. Like we use LeaseWeb, OVH, those providers that help us to do that. Long story short, today we have infrastructure on the cloud. And it took us some time to have the good infrastructure, the good hardware in cloud to run our API. Today there is way more variety of hardware on cloud provider than it was the case in 2012. So this one was a good decision. That's the only example, positive example, I will share with you. But sometimes a build versus buy can be positive. It's not only about always selecting one. It's something that can be your big differentiation. It can be your factor 10, like on performance for us. But it could also be a negative decision. And I will try to cover more negative examples. So I have 3 more concrete examples of mistake we did. So second decision, very important decision. We were end of 2014, we were about to launch what we call our distributed search network. So this one is the ability to distribute the search in different geo location without any pain for the customer. So we have one API, they push the data, we distribute the search worldwide, and then we redirect the user to the closest data center. Every customer can decide which location they want to have to control their cost. Like, we have 15, 16 regions worldwide, but you may not want to use all of them, so you can control your cost. The geo-distribution is done by the DNS, and we were using a solution from Amazon AWS, Route 53. 53, which is their managed DNS, to do that. We had one big issue first, which was the number of regions they were supporting were not the same than us. Like we had at the moment of the launch 12 regions, Amazon was covering only 7 of them. So we were not able to do the geo-routing for all our customers, all our regions. Second problem, which was a performance problem, when we were creating a region, like a specific entry for one customer, it was taking a few minutes. Meaning, in the onboarding flow of our users, it was taking a few minutes to set up the account, which was causing a huge drop in the process. So we decided to look for all the solutions of the market. The engineering team was super, super small at this moment. We were in total 4 engineers working on the product, including me, including our VP of Engineering, so 4 in total. So very, very small team. First reaction was to look for existing products. Hopefully there were a Wikipedia page listing like all the managed DNS available on the market. So we took them one by one, we contacted all of them, we look at their offer, none of them was able to address our use case. So it was a big disappointment. Like, I think there were a list of 20 different providers, none of them has a solution that could suit our use case. What we did was to select the most advanced and had a product discussion with them. Maybe they don't have the solution on the market today, but maybe they have it in their roadmap. None of them was able— was able to give us like their own map or an ETA of when they will be able to address our use case. So we were stuck. As I mentioned, a good engineering team likes challenges. So in no time, engineers built a small prototype based on open source, and we negotiated with a network provider to build like an anycast network with our 12 regions. To redirect the traffic to our custom-built DNS based on an open-source software. The prototype was built, I think, in 2 days. Working prototype, of course, not at scale. But then, like, transforming a small prototype in a product in production, not with 10 API calls but billions of API calls every month, is another story. When we did these studies and where we were about to contract, to sign the contract with the Anycast provider, we took some time. The reason is that this provider wanted the minimum 3 years commitment on the solution, which is huge for a small company like us. It was pretty big budget. And the budget was so big that we decided to take some time to analyze the situation. By luck, we have met someone in San Francisco that was launching a startup on DNS. So it was a completely random meeting in San Francisco, meeting a small startup building a managed DNS solution. The chance to have something that would fit our use case was super low. But still, this small startup was building something in a domain where we had done a lot of analysis, so we had super interesting discussion. And they proposed us to transform their roadmap and to address our use case in a month. The company was super young. They were just founded a few months before this discussion, so super early stage. And of course it was risky for us because like we were a bit more advanced. It was key for our product, so relying on a very small startup is a bit risky. But we decided to take the risk. The big reason we took this risk is because the team behind this startup was working on DNS for 20 years. So they were knowing a lot on the domain and actually they make us realized that our custom-made solution were lacking of a few things that could hurt us. Like, we have not anticipated like DDoS on the DNS, which is a big problem and that happened a lot of time. So it could hurt us. We have a lot of things that we have not anticipated. So we decided to use them at the last minute, like maybe a few days before signing like the 3-year contract with the provider and launching our own solution. And this story does not end at the moment we decided to use a startup. Because as I mentioned, when your engineers got excited about one problem, one specific problem, and there is no solution on the market, they try to think about differentiation. Like, oh, we could do this specific feature. It would be 10 times better than other products. Or we could develop this one, which is also a big differentiation. So we had kind of a long roadmap of big differentiation on DNS compared to existing products while we were not expert on DNS, which seems a bit wrong. But I think a good engineering team will always find this and they will come to you and give you a big list of things you could do way better than anyone else, even if they have no strong expertise on this market. My understanding is that we always— there is this famous quote, we always We overestimate what we can do in a year, and we underestimate what we can do in 10 years. I think the big mistake here, the team was completely overestimating what they could do in a year timeframe. But this SaaS solution give us a lot of learning. All the crazy idea we had, actually they develop them. So we have contract— contracted with them. And they give us a lot of advantage of learnings. Like most of their big ideas actually were not working in our use case. We would have spent a huge effort in R&D to try them and they were not working. With this SaaS product, they did all the investment, we tested it, and on our use case it was not working. So the investment was not crazy compared to the results. But we were about to do the mistake. And I think one of the key learning is that finding the provider is super complex. It's not just listing the— going to Wikipedia and list for the providers. It's looking for the startup, and maybe the startup is not existing and the startup will exist in a year. So it's also about thinking about complexity to switch to another solution, like this 3 years commitment. Would be crazy. And if today I was in the position to take the decision, I would not go with the 3-year contract. I will try to find a workaround, maybe reducing the number of regions, maybe 12 was too much, finding something where I don't have a 3-year locking. Like 3 years is way too much to lock in a situation that can potentially cost us a lot of money just to maintain. So we were about to do big mistakes, and by a lot of chance, at the last minute, we discovered this startup and decided to take the risk. Maybe the story would have been very different if we would have developed this DNS, and I think it would have been the case, because like it would have took all the R&D to develop something where we were not expert. Another very similar decision. So this one was way later. We were like 35 engineers. We are focusing on performance. So of course, the monitoring of our API, monitoring of performance was key since the beginning. And we were using a SaaS product to monitor our infrastructure. But we eat a few limits in the solution we were using. And those limits were mainly because the solution we were using were designed to monitor a specific machine, whereas our service is distributed across a set of machines and we need to see like some graph distributed on a few machines. So again, we discussed with several providers and we had like Even discussion about roadmap, discussion about what could be like a good implementation for our use case. We discussed during a year with the provider we were using, and there were still no delivery on what could make an impact for us. While we were discussing with providers, the engineering team one day decided to build a prototype. So they built a prototype internally. Based on open source using Collecty, Grafana, Graphite. And in a day they were able to have something better than our SaaS solution. And I think here maybe the mistake is to think that because in one day you have something which is better than what you have, it's super easy to have your own solution which does not cost so much money because it was built in a day and that you can tailor to your needs. I think there is a few drawbacks in this conclusion. The first one is that it's a prototype in one day. It was deployed on one machine where we had at this moment like 200 or 300 machines. So just taking this solution, making it scale at 300 machines is not a matter of hours. So first big drawback. Is about limitation. Then even if it was better, when we were looking at what we're missing in this solution, the list was huge. We had dozens, if it was not hundreds, of missing features for our needs. So it means our team would have to develop all those features for our performance. And it means my roadmap would be like putting some emphasis on those metrics which are not game-changing for the business. It's not something which is exposed to customers. It's something for us to track the performance of our service. And I think that's again a drawback, like thinking short-term, building something in a day, which is great. It's a good engineering effort. But then on the long term, like putting our investment of something which is not game-changing for our business I prefer to pay for an external service rather than taking my best engineer and making them work on this aspect. And I did a mistake at this moment because of the previous story about DNS. I tried to convince them to use an external solution. So I tried to discuss with them, argue that the long-term cost would be like super, super expensive and we have to select something more efficient. I never managed to convince them. It's pretty much one argument against another, and if you discuss about one specific thing, they will come with another prototype, and it's an endless discussion. And you will never manage to convince them. Again, good engineers, they have 10 different ways to explain you that the best solution is to build and not buy. I think— At least one thing which is good with smart engineers, they learn quickly. And they still learn from this previous experience with DNS. I never managed to convince them, but one day the lead engineer discovered a startup, and we had exactly the same pattern. Like, they discover a startup building a super good solution on monitoring. They build a small prototype. They convince the team, and we use the SaaS solution. But we have this long intermediate step with open source developed, and we have put some effort on open source. With some— like, with a step back, even this 6-month investment on open source was a loss of time. Like, we, during this moment, we never put the open source in production. We never had like a business impact because it was just some investigation. Like using our previous provider, trying to push them a bit in our direction, even if they were not able to execute like our vision of monitoring, would have been way better than spending our time building prototypes. So again, small team, even if it was 30 people, it's way bigger than 4. But if you have 3, 4 people in this team focusing their time on building a solution on open source, and then you argue and they spend more time and it's endless. Basically, you lose your R&D on something which is not game-changing. So what I did learn in this story is never try to convince them, but challenge them. Challenge them on the cost of developing their prototype. Challenge them on the fact we cannot use our existing monitoring solution for 6 months to a year. While we find a good solution on the market. Challenge them on every single decision, but do not try to convince them. It will never work, I think. My personal feedback is that you will never convince them, but you can challenge them, and they will look at the problem differently. Empower them, but don't give them like pretty much an order. So for this one, again, We took the decision to build internally, which was a bad decision. During 6 months, we built those prototypes, we did some iterations, I tried to convince them. The decision was not a good one. And the good solution comes 6 months later from the lead developer. Because they found a good startup building a good product. And I have another thing which is funny in this story. Like, the engineering team recognized that they would never have been able to develop the solutions they discovered. And I think that's the big learning with SaaS. Your internal team will never be able to compete with a SaaS product. Never. It's hopeless. If the solution is successful, the team will scale. And maybe today they have 2 engineers, but in 2 years they will have 50, while you will stay with a limited effort and only 2 engineers working on it. So this solution was so good that they were convinced in a week that they would never be able to build the same thing. But again, they took the decision to build internally. The last example I wanted to cover is a bit more complex. It's a decision we redo several times. So this one is about our analytics solution. So we on every Search API call of our users, we monitor all the API calls and we give them some statistics. So we did this analysis several times. When we created the product, we did the first version of the analytics, 200 lines of Ruby code, 1 hour of development, super good investment. And this analytics is also business critical because we use it for the billing. We count the number of API calls, and that's what we use for the billing. But like, beginning of the company, we knew the solution was not scaling. 200 lines of code, but it was perfect. It was not even worth to look at the existing solution of the market. In 1 hour, it was solved. But we knew it was not something that could scale. A few months later, with the growth, not even a year after, we have to redo this part, which was expected. We knew it was not able to scale. And we give the engineer, like one engineer, the autonomy to look at the problem, but not at the market. What we give him was a mission to replace this component that was not scaling. So what he did, he took a better programming language. Instead of Ruby, he took Java, which is better in terms of performance. Instead of 200 lines of code in one thread, he multiplies the number of threads, so he gives us some autonomy. This development was good in the sense that it was able to run 100 times the load of the previous application, and it gave us like 2 years. During 2 years, we were able to use this software. But then again, the software was not scaling. We look at the solution on the market, and again, kind of the similar solution. Nothing was perfect. Perfectly good for our problem, and especially our scale was so big. We were already processing billions of API calls, but the price was too high for us. We would lose money by using a provider on analytics. We did a mistake at this moment, which is to, okay, it's not possible for the pricing, let's look for another solution. We did use at the end a hybrid solution, which is a mix of APIs plus custom code, and we use the GCP platform for that with Pub/Sub, Dataflow, a lot of different products. But it took us close to a year to develop it. We reached a lot of problems, a lot of bugs, like those APIs have never been used at this scale, so we had a lot of problems. Problems that were not anticipated. But during this time, we already reached the limit of the previous program. So one of our best engineers, during close to a year, has to do his best to maintain the solution that was crashing several times a day, spending a lot of time just on maintaining a solution, which is not good. At the end, with some step back, we should have paid for the very expensive solution for a year. We would have lost money for sure. But I would have gained the time of one of my best engineers that could have worked on something more critical than maintaining a solution that was supposed to die. So that's another mistake, like looking at the cost maybe too early and spending some money for a limited period of time would basically buy us some time and the time of one of our best engineers, which is a big thing. Like we never have too many engineers. I have never found anyone that have too many engineers. So at the end, those decisions are critical for the business. Like those decisions are pretty much reinventing the wheel. Like that's pretty common sentence we hear in engineering and we always do mistakes. My feedback is that it's super difficult to don't do any mistakes. We all do mistakes. And I think we have to deal with it. And one of the best ways to deal with it is to accept we can replace components easily over time and don't think about building something for the long term. Like, the cost of maintaining a solution can be super, super high. So how to do not try to reinvent the wheel? I think one of the things where I have the most proud of is probably that our engineers have learned a lot. Like, those engineers have learned way faster than I personally learned in my career because they saw all those mistakes very quickly in a fast-growing environment. And I'm super proud of what they— like, they become better leaders, engineering leaders. So having some very strong experience in your team, people that have a lot of experience, did the mistakes several times, I think is one of the best way to avoid that. Then of course, thinking about keeping a solution for the short term. I have some very funny story about that, but I work in a big group that in the '80s developed a specific software to version their source code. This software was great. Today they still use it. This software is 10 times, 10 times worse than any solution of the market. And they have a whole team to maintain it. So 30 years later, they still maintain their internal choice and they are not able to migrate. So the earlier you can move from something you build to an external solution, the earlier you can keep your focus. And the speed of iterations, the focus, I think is a critical element in any engineering team. Like the speed of iteration is what we will make the difference. And I think one of the only good reasons for the long term to build is you have a factor 10. Factor 10 of your product, factor 10 on anything, then it's a good way and good reason to build. So that's it. I think we have 1 minute for question. So first question, which companies should not build the infrastructure on their own? I think most companies, I think it's really an exception to build infrastructure on your own. I think 99% of the company or 99.99% should not build the infrastructure. I think you need to have a big bet and it needs to be a huge difference in your product, in your company, in your business to have a good reason to build it. Otherwise, I don't think— I think if we have discovered that the factor of the performance was not a big differentiation, we would have moved to cloud. So I think most companies should not build their infrastructure. Another question, how do you navigate the make versus buy decision with multiple big data software vendors? Who have more offering than ever. So the complexity of the offer is definitely something which is an issue. I think when you evaluate all the solutions, I never want my team to think in terms of features. I think you have the product with the feature set as it is today, but if you use and you buy a solution, you need to project your decision on the long term. So looking at the roadmap, discussing the roadmap with the provider when it's strategic, looking at what the product will be in a year, in 2 years, discussing their vision of the market and if it can help you, I think is a very good way to make the big difference and move out of the feature list comparison or of the complexity of the solution. And another thing is looking at the integration cost. If it takes you 2 years to migrate to a solution, then it's an issue. Another question: What do you think is the main benefits of Innosbuild over Vondor solutions? I think there is only one for me, which is a factor of 10. Like, if you have this factor of 10 on one specific area which is important for your business, It can be the UX, it can be the performance, it can be the relevance, it can be anything. If you have this factor 10 and it's something you can measure, if it's your strong difference on the market, then it makes sense to own it. If it's not your factor 10 and if it's more a feature of your product, then I think it does not make sense. If it's your whole product, your whole differentiation, I would recommend to build. If it's not, then just buy and find a good provider. Thank you.