Ep. 035 - Tech DD’s, Performance Projections, Supply Chain, Investment Thesis (Consulting)

Hello everyone. Welcome back to SemiAnalysis. My name's Jordan. I'm here with Abhilash. This week we are gonna do something a little bit different. We're not talking about an article. Instead, we're giving everybody a little bit of an explanation on the SemiAnalysis consulting business. Abhilash leads our consulting business. He's got a great team. And a lot of people have some questions about like what we do in consulting, what types of clients we serve, what are the projects that we do. And things that we've learned. This covers everything from like DDs for investors or companies that are looking at M&A activity to investors who want to go beyond the core products and customize something about one of our research products like the accelerator model, data center model, things like that, as well as strategy guidance, feedback on product roadmaps, go-to-market activities for lots of clients and industries. So Avlash, welcome to the show. Excited to have you today.

Thank you, Jordan. Excited to finally make my debut to the renowned Semi Analysis Weekly.

Avlash and his team have lots of opinions that get expressed on this podcast, but for the first time we're going to hear it from him directly, so I'm excited for that.

Oh, absolutely.

Okay, give us a quick overview. What's Semi Analysis consulting business? What do we do?

I think so. So by the way, Semi Analysis has been doing consulting ever since I think Dylan started SemiAnalysis, but it was only last year, I would say, when we set up a dedicated team. And the reason is SemiAnalysis does a lot of things. We publish our newsletter, we have institutional models. I think we have close to 13 to 14 different institutional models right now. But a lot of this, and we have open projects, open research projects like InfinX and ClusterMax. So there's a lot happening, but a lot of this stuff is actually very off the shelf. So if any industry participant or any investor wants to engage more deeply with Semi Analysis, we didn't really have an operating model internally to be able to service those requests. So that's the genesis of Semi Analysis Consulting. I think what we do is we work with various different clients on a very dedicated basis. All the work that we do is very custom. We typically staff a dedicated team for anywhere between 3 to 4 weeks to all the way from a year. And I'm happy to talk more about it, but that's essentially what we do. All the work that we do is custom.

Yeah. Awesome. How's the work? Maybe start by talking about the types of clients that we serve. I gave a little high-level overview of the different types of projects that we've done, but yeah, take us through that.

Yeah. In my opinion, I think what we are witnessing is something called, we always talk about hardware-software co-design, right? In the world of AI infra, there's something else that's also happening, which is, which I call tech finance co-design, I would say. So we need to teach tech bros more finance. We need to teach like more finance bros more tech. And that's essentially what we do. I think there are 2 industry— there are 2 sorts of client groups that we serve. The first one is the financial institutional investors, and these are large private equity funds. These are large hedge funds. who are very savvy investors and they are very excited about investing in AI infra. And now when they're deciding to drop $1 billion, $2 billion in a chip company, in a NeoCloud, in a data center, they want Semi Analysis to come in and essentially help them figure out various aspects of the deal. So we do various sorts of diligences for them. We also do a lot of thesis development for hedge funds. A lot of hedge funds are looking for The next wave. People got excited about memory, people got excited about CPU. So they're trying to think about, okay, what's next? And we also work with a lot of hedge funds in combining various different data sources and different market narrative to figure out what's the next wave. On the other hand, we also serve a lot of industry clients and these are, I would say, largely the hyperscalers, the AI labs, the chip companies, semis companies. And here the range of topics can be really, really large. It could range from helping a semi company figuring out their product and engineering roadmap. So a common question is, if LLMs are evolving in a certain way, what should be the products that we should be thinking about 3 years out, 4 years out, and 5 years out? It could also be helping certain hyperscaler figure out data center sites in remote locations in India, for example. It could also be doing go-to-market strategy for various different companies who have a strong product, but don't know how to connect and figure out their customer journeys. So it's a range of different options, but that's essentially the type of clients that we serve as part of Symbiosis Consulting.

Makes sense. And quick advertisement for Abhilash on his behalf, if you're interested in If you're listening and you're interested in tech finance co-design, you should consider applying to work on consulting at SemiAnalysis. It's a lot of fun.

We are hiring a lot. So, shameless plug, if you're AGI-pilled, excited about semis, curious to learn, please hit us up. We are desperately in need of people.

Yeah. So, take me through some of these customization examples. The place where I get involved in some of this work work is that we're taking research that me and other guys on the team are doing and trying to package it up in a more digestible way, as well as customize it for somebody's specific interests. So take me through what that process looks like, where we lean on our existing data sources to actually go and execute on these projects.

Right. I think because at Semianalysis, we have so many different things that are happening. Usually if you're able to just combine some of these data products, in a more comprehensible manner, we are able to answer some of the questions that each of these data products are not able to answer by itself. So I think one of the examples was very recently there was a hedge fund client that was very interested in understanding what's the total token supply in the world assuming the chip install base. And if you think about it, right, like it's a fairly simple equation, what is the total install base in the world today and how is it expected to grow in the next 5 years? How much of that capacity is allocated to training versus inference? If you have the install base, the second part of the equation is what is the throughput per GPU and how many tokens can each GPU generate in a certain day, in a given year? And finally, what's the pricing in terms of dollar per million tokens for these tokens? Each of these 3 different parts of the equation got answered in a different manner, but the most complex was estimating the throughput per GPU because we had to combine a lot of our InferenceX data that's already public, but on chips that are not out yet and on models that are not out yet. So we had to build an inference simulator where we had to estimate, okay, if NVIDIA comes up with trainments with certain set of characteristics, what does it mean for the throughput? If memory gets despecced like it has happened, what's the impact on throughput? If OpenAI trains a $10 trillion, $15 trillion, $20 trillion model, like parameter model, what does it mean for throughput? Now, a lot of this is not readily available and something that we had to build out custom and we had to work with various inference engineers at SemiAnalysis to be able to do that. But that's an example of a project that would be custom work.

Yeah. Okay. Take me through some of those assumptions there. Like in the inference simulator, I know, well, maybe explain how it gets delivered to the clients because I think a lot of people hear consulting and they think slide deck. And we make plenty of slide decks, but for complicated questions like this, the trend for us has been to actually move towards live dashboards where people can edit their assumptions over time and we have to maintain this. So it's kind of like a software product that we're delivering to them, right?

Exactly. So this was like, there were so many permutations and combinations in this product that it was not feasible for us to put this in a static Excel. So we had to build a dynamic dashboard and that was essentially what was delivered to the clients. This is a dashboard that they could use, play around with different sets of assumptions in a very visual manner and see literally every number changing. So the key assumptions here were, I think to start with, how much How much of your compute are you allocating towards inference versus training? What are the— we know the existing chip specifications, but what are the chip specifications that you're imagining for the next set of silicon that are being developed by AMD, TPUs, NVIDIA?

Yeah, we know from one of the episodes, 2 or 3 episodes ago, that those specs can change pretty quickly. Rubin Ultra just got despecced from a terabyte of HBM down to 256 or 192. Yeah.

What's your— another set of assumptions could be around what's your input-output ratio? Are you using agent X harnesses? Are you using fixed sequence lengths, context lengths, sort of data types? What's your cache hit ratio? How does throughput change if you're hitting 80% cache versus 85% versus 90%? So I think those are some of the assumptions that make it interesting.

Yeah, makes perfect sense. Now, tell me more about an example that goes for a different category, which is strategy. This could be on product roadmap, it could be on a business decision about build versus buy versus partner. What are some of the domains that you're helping people with right now?

Yeah, I think strategy is a very broad term, I guess, and it could mean like product strategy, it could mean like go-to-market strategy, but I can give you an example of both. So as I said, like one of the things that we have built internally is an inference simulator and like one of the memory storage companies is essentially using that simulator to replay like agentic traces for a combination of different models. As well as chips to figure out the next set of memory products that they should be developing. And they're answering the questions like, how does memory bandwidth versus sort of capacity trade-offs play out in serving production inference workloads? So that's one example of a product strategy thing that we are doing and we are working with their teams on a very dedicated basis. The other example could be— We are helping one of the semis companies figure out their ideal set of customers. So they have a product, but they don't know who to sell to. Like there are probably like 100 customers that could buy their product at any given point in time, but it's a long lead time product. It takes at least 3 to 4 months to qualify a customer. So it's not realistically possible to go and qualify your product with like 100 different customers, right? So we are helping them figure out who are the top 10 or 15 customers that they should be prioritizing, what's the right way for them to approach sales conversations, what are the key technical topics that they as customers would be interested in, and how should you position yourself to be able to deliver on those technical specifications that these customers want. It seems easy to say, it's very difficult I think to execute because we need to go out and literally speak with 20, 30, 40 different customers, get their perspectives, aggregate a market view. And this is something that a lot of different companies are not able to do because SemiAnalysis is so very well connected. We don't really use an export agency to carry out these export calls or whatever you call them. Because we have such a wide network, we are able to just use our relationships to figure out important questions that would help the wider ecosystem.

Yeah. And this also informs our research. I mean, maybe you can explain a little bit about how people who have been on this podcast in the past, like say Eric a few weeks ago working on modular data centers, are, you know, when they go and do this research effort, it contributes to SemiAnalysis products. And in many ways, some of the institutional products that we sell, the data products originally started as a consulting project, or at least the inspiration to start working on that actually started because somebody's asking us about this. They clearly think it's important, therefore let's go do it. Right?

Right. Because I think because we work on a very custom basis with a lot of these clients and we go very deep on a topic, I think oftentimes we're able to identify that there is a, like, if this question or problem is relevant for one client and we have been able to figure out a standardized solution for that, I think we're able to scale that up to a broader set of clients and that becomes a product. I think our data center model started as a consulting project for one of the clients. Maybe ClusterMax started as a consulting project. I don't know. You have to tell me that.

No ClusterMax, but the AI Cloud TCO model definitely started with consulting projects, which was, we continue to use to build analysis on how, what are the input costs for GPU clusters? How profitable are given companies that are serving tokens based on the choice of chips, based on the amount of OpEx, based on the as you're saying, utilization ratio on an inference endpoint or something like that. The way we actually dig in and do that is like, you get inspired to do the work because people that you trust, that your clients are taking some problem seriously and asking questions and let's go get answers, that becomes a useful product. So anyway.

Exactly. And I think the other example, the latest example of that is the inference simulator that we are building, right? I think we started building that for one client and not even for one client. I think it started as a cool like science project within SemiAnalysis and then we offered it to one client, they found value in it, and then a few other clients found value in it and now it's becoming a consult, like sort of a product by itself. So absolutely.

Yeah, makes perfect sense. So how about DDs? That's obviously a big thing for the investors as well as some companies that are interested in M&A activity.

I think Dan talks about $7 trillion of debt being issued to fund AI infra, right? So there's a lot of, I think, investors that are coming into the game. There are a lot of newer players that are also coming into the game. And so what a DD project essentially looks like is an investor coming to us and asking us, hey, can you either help us? Let's take a NeoCloud for an example, right? Because that's what we do a lot of the times. If a private equity fund wants to put in $1 billion in a NeoCloud, essentially they're looking for 2 things. They're looking for a technical analysis and they're looking for a commercial analysis. On the technical side, I think what they're interested in is what are the SLAs that a customer— that the NeoCloud is offering to the customer, and will this NeoCloud be able to deliver on those SLA guarantees. Because if they don't, there are like wider set of implications that we can talk about later, but it's not good for any of the parties, right? They're also interested in understanding what is the OEM support and warranty structure and contracting looks like, because there's a lot of devil in the details there. And if you are not transacting properly with the OEMs, you could expose yourselves to a variety of risks. They're also interested in understanding the cluster performance. So we are able to test their clusters with a very ClusterMax-inspired methodology and be able to tell what exactly are the things they're good at and what exactly are the things that really, really bad at. And what's the 100-180-day plan for them to be able to improve on those things? I think on the commercial side, because it's such a financially savvy decision, they need to understand what's the CapEx and the OpEx for these builds and how does it compare really to the market standard because as part of our TCO model, we are able to benchmark in real-time costs as well as through various other means like performance.

Yeah.

We are able to benchmark these things in real-time for these clients. So that's what our typical DD projects looks like. And it varies depending if you're diligencing a NeoCloud versus a chip company versus a model company. or a data center or a powered land, and we have literally done everything.

We've done a lot in just a year or so since you've been here. And I mean, we're just like stacking up experience across it, building the team. It's pretty cool to see it kind of grow as people really, as more and more people in the industry take this stuff seriously, because I think the need for technical due diligence on a lot of these projects is of the utmost importance. It's, it's all of it's new, right? It's not like certain investors can understand or have the capability to understand, you know, a type of model that's 6 months old or 3 months old. I mean, it's new for everybody, right?

Anyway. Exactly. It's new for everybody. And also I think what's happening is that In my opinion, I think the off-taker base is diversifying beyond the top labs to also like now the neo labs who are actively transacting in the market. The neo cloud operator base is also diversifying. If you think about it, right, like hyperscalers are the OGs and then you have the crypto miners and then specialist players like CoreWeave and others. But then now you see a lot of real estate developers who have, who do not have any experience in operating cloud entering the scene. You see family offices of rich people trying to build a new cloud now. And so SLAs become even more important because if you don't have the experience of operating such things before, and if SLAs are not in your favor, like the implications would be really massive.

Yeah. Let's talk about the 2 types of them. Obviously, this is something I'm quite familiar with. There's one, Type of, you know, SLA that's related to the delivery timelines. Really, an SLA is just like, if something happens, a contract can be terminated. That's roughly the legalese behind a service level agreement. And it doesn't, you know, contract termination rights is maybe a more broad definition. And that's really what we're talking about here as opposed to SLAs. The first reason why you might be able to terminate a contract is because of late delivery. Cluster shows up months late. So therefore cancel the contract. But the most common one is downtime. If somebody's got GPUs installed, they expect to have 3 nines, 2 nines, 1 nine of availability. And if you're under 90%, you know, 1 out of 10 days in the month, 3 out of 30, you don't have access to the GPUs you're paying for. Generally, you're getting some credits in response, or you have the right to terminate the contract. So Maybe talk to me about the motivation for the lenders or for the NeoCloud itself, or even for the buyer to have some sort of insurance against either of these possibilities of a contract termination. They want downside risk protection, right?

Exactly. I think so. So Jordan, you hit the nail right on the head right there. So one, why SLAs are important, and then I'll talk about why insurance is important. I think you said it right. The first thing is as a new cloud without strong SLAs, you won't be funded, like period. Like you won't be able to start your journey. Like all the investors, all the lenders, the most important thing that they care about is what is their risk exposure? If they're getting interest payments every month, every quarter, what is the chance of that payment not hitting their bank account because your cluster is down? So that's the number one thing. I think the second thing is I would say SLAs are really for the bad times. We have seen this in various settings. If things are going well, nobody talks about the SLAs. If things don't go well, people talk about the SLAs and it gives sort of the parties to be able to get out of the contract scot-free. And the third thing is, we have talked about it, at least for newer clouds, they typically sign a 15, 20-year data center lease. They typically sign the GPU contracts for 5 years. And so if you're not committing, or if you're not sort of delivering on your SLAs, you carry a reputation risk, which means that you will not be able to get your second deal and you'll not be able to get your third lease, right? And if you're not able to do that, then how do you— like pay for 20 years of data center leases. So SLAs are important. I think the reason why it's important for lenders, as I said, is they're looking for downside protection. And one way to protect your downside is by having an SLA insurance. We know a few companies now who are ensuring SLA uptime for the NeoCloud. So if you, as a NeoCloud, if you pay a certain basis points of premium, You know, you get a certain protection that if you have to pay an X amount of penalty, the insurance company will take care of it. It's not easy, in my opinion, to get this insurance because these insurance companies are also pretty savvy in terms of how they underwrite these neo clouds. So it's not an excuse to have bad performance that, you know, you can continue to have bad performance, but then you will not, and then you'll kind of compensate for it by having the insurance. If you have bad performance, you won't even get insurance. But if you have good performance, I think a way to protect your downside and make maybe your contract more investible is to probably think about electing for some of these insurances.

Yeah. Now, we've thrown around the term SLA and talked about a few different things. Obviously, there's many different components of a given cluster or data center or site. That can have an SLA attached. So this could apply to the GPUs or the nodes themselves. It could apply to the racks, it could apply to the clusters, it could apply to the site itself. It could also apply to the power systems and cooling systems in there. And when there's so many parties at play doing this and there's so many contracts trading hands, there's actually a lot of reasons why you might want to have a risk assessment done for just even one individual part that you're less comfortable with when compared to you know, some other part of the contract that you have more confidence in because maybe you've worked with that vendor before or because you understand the technology better. Talk to me just about the concept of a risk assessment and the scope, because, well, I know the— I'm leading the witness here, but the scope can be pretty tight, pretty quick, or it can be pretty big, pretty broad, right?

Right, right. I think As you said, like a risk assessment could be like fairly critical, especially in cases where lenders are, or investors are demanding for it. And that's a necessary condition for them to fund a particular sort of a cluster. I think it could be as lightweight as doing a contract review of the most important like sort of contracts for a NeoCloud. And these are the off-taker contract, the OEM contract, the DC contract. Because you're promising something to the customer and then you're relying on the DC partner and you're relying on the OEM to be able to service that. There are like other side contracts as well that we can get into, but like we don't have to right now, but that's one. I think the second thing is a more, like a slightly higher touched review of, or risk assessment you can call it, which is data center readiness, which is understanding the milestones around When will the data center clear like L3, L4, L5 testing? When will the data center be made available to you to do like a bunch of different other testings? When will the racks arrive on the data center? When can you install like sort of power and cooling? And like when all of these things can be tested as a system all at once. And we can talk about like other aspects here as well. The third thing is, doing a bare metal review, which Jordan, I think you are the master of with all the cluster max testing you have been doing. So I think I'd let you do it and then maybe we can come back for the final one. Yeah.

I mean, the summary is basically that we get access to the systems, we log in, we test them, we do things like simulate failures, and we test to make sure that people have the monitoring system set up so that they can identify that the failures occurred and how quickly they can recover from it. There's many, many different ways in which things can fail in a cluster, and your monitoring systems have to have full coverage because we've seen things fail in spectacular ways, even in just our few weeks of testing, where it's like a fully contrived environment. People have all hands-on to set it up just through ClusterMinds testing. And then we hear the stories from all the customers of these providers about how poor the quality of services that they're experiencing. Yeah.

Yeah. And the root cause can be like really like sort of varied, right? And I think this assessment helps you understand what exactly is the root cause of bad performance. Like, is it because problem in sort of like the way your electricity is set up? Is it because of the way your cooling is set up? Is it because of the challenges in the racks itself? Is it because of the challenges in software? And ultimately it's like only one, like Even if there is only one reason that like your cluster is not performing well, it still does not mean that you don't have to pay the SLA penalty. So like only one thing has to go bad for you to like be liable for paying all the SLA credits back to the customer.

Yeah. I mean, that's why we go back to the contract review stage, which is the first one, right? People need lots of advice on how to set up these contracts because as you said, contracts are for when there's bad times, not when there's good times.

Yeah.

Okay, let me ask a kind of related question, but it can keep us going here. When we talk about giving strategy advice and stuff, a lot of people are getting into this space where they're considering build versus buy specifically with the NeoClouds, because, you know, there's such long lead times, it's so hard to find a data center site. And we've done a little bit of that consulting work where we've helped people make that build versus buy versus partner decision. And we've done it in more than just the NeoCloud decision for other stuff, let's say supply chain management, if you're a chip company or something. So maybe you can talk a little bit about those examples as well.

Yeah, I think, uh, uh, because we are definitely seeing a lot of constraints around getting colo capacity to do, uh, sort of large site buildouts, uh, for GPU clusters. We are also seeing some tightness around chips and there are definitely shortages around labor. So in principle, even when a lot of the companies want to eventually build their own clusters, that is oftentimes not the most feasible path to get near-term capacity. And so ultimately it boils down to a few factors. I think the first is time to market. Second is Your willingness to pay a premium to get compute. Third thing is how technically sort of savvy your own team is and do you have any experience in running cluster operations? And finally, in terms of financings and everything, like do you have enough credibility in the capital markets to be able to raise funding to set up a cluster from scratch or to be able to be an off-taker of a company. And so recently we have been working with a few clients where we have let— we have arrived at a few different answers. I think in one of the cases we have recently advised a company to be able to just like go out and rent because they wanted 100 megawatts in literally in the next 6 months and there was no way they were able to they would have been able to build a cluster that fast. But there have been cases where somebody was willing to wait until 2028 and 2029, and the recommendation is completely opposite. I'd say these are typically 3 to 4 week projects, but definitely very interesting from the range of content we are able to cover and options we are able to explore with them. Okay.

Personal question. What's your Most, what's the projects you find most interesting right now of this whole suite that we've been talking about? And what's your favorite? Is that the same thing or is that a different project?

So you asked favorite and what else?

Most interesting. You just said it's an interesting project. I assume that means you're learning and, you know, doing interesting stuff.

Yeah. Yeah. Yeah. I would say like one of the most interesting projects right now that we are doing is where we are Like there's a bigger company that's looking to acquire a smaller company and that company doesn't have a very big team, but they have very sharp people. And so I get paid to speak to a lot of these very sharp people and I get to learn a lot in the process about deeply technical projects about LLM architectures, kernel engineering, GPU optimizations, fast inference, slow inference, high throughput inference, like whatnot. I think any of these topics definitely sound very interesting to me. I think my favorite projects are the ones where we are able to just use a repeatable process to get to an answer because we have done so many of these Neo Cloud Diligences projects over the last one year that now it has just become muscle memory. Like we know exactly What information request list we need to send out on day 2, what interview we need to conduct on day 4, what 5 follow-up questions that we always anticipate land up on day 6, what's the midpoint checkup, like readout that we need to do on day 8. And finally, like what's the conclusion? So yeah, it's because of efficiency.

The conclusions are not predetermined. You can't be saying that on the podcast, man.

You'd be surprised how often they are, how oftentimes they are. Yeah.

The, for the NeoCloud diligences, you think it's pretty straightforward to analyze some of these contracts and come to pretty clear results right away?

Yeah. Yeah. Yeah.

Kind of cookie cutter at this point, but it makes sense. It makes sense that people want the peace of mind that, you know, we've reviewed a lot of the stuff. Okay. How about I'll ask you a more broad question. What are you most excited about going forward with consulting?

I would say we are definitely a very, very fast-growing organization. I think the breadth of topics that we are covering today versus 6 months ago is a lot higher. And I think what I'm more excited about is partnering with Like literally, like we already partner with the people who are building in the frontier of AI, but getting more opportunities to partner with such people, I think it's definitely the most exciting thing because you get to learn so much in such little time. It's crazy like how fast the knowledge graph and the learning graph here is at SemiAnalysis. The other thing I say is definitely Dylan has told me to hire 20 more people in the next 6 months. I don't know how I'm going to do that, but that'll be an exciting journey as well. Like interviewing a bunch of people, spending 50% of your day on hiring.

Yeah.

I know.

It sounds like your current day is pretty exciting because if you're excited for that, you're, it's just more of the same. Yeah.

More of the same, man. More of the same. Yeah.

Cool.

Okay.

What do you think jumps to mind when I'm like, what did we not cover in this call so far?

I think we have pretty much covered everything that I could publicly say, to be honest.

Okay.

Because a lot of, a lot of this stuff—

Let's talk about what you can't say publicly, Abhilash.

No, no, no. There are a lot of things that I can't say publicly, but I can't say them.

Yeah. Cool, man. Well, I appreciate you joining us today. It's a lot of fun. Love working with you on a lot of this stuff. Hopefully next time we can have some more guys on the team and we can give an update on sort of trends that we're seeing as we execute on a lot more of these projects and we keep learning.

Yeah, I think today was more about I think what consulting is, we should definitely do an episode where we talk more about like what our learnings are, because there are definitely a lot of learnings we have as a team. And I think the team will be excited to come and just share everything that we have learned so far.

Yeah. Well, I'm sure the audience would be excited for that too, but we'll leave that for a little teaser, a teaser behind the podcast paywall coming soon. 15 minutes of podcast behind a paywall. How about that? Everybody loves the newsletter because it's free. But there's a paywall there. So how can we paywall this video?

Let's do it, man. Let's do it. Let those dollars come in.

We might need to watermark right over our faces too. I think that might be a better thing. Yeah, yeah, yeah. Like, uh, translation watermark. Yeah, cool. Good job today. Thanks everybody for listening. See you next time.

Thank you, guys.