Game of Thrones, but for AI Chips

Today on the AI Breakdown, we're looking at a set of news in the AI chip wars. Before that on the brief, OpenAI fires some researchers for alleged leaks the AI breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our Discord, our YouTube and our newsletter. Welcome back to the AI Breakdown Brief. All the AI headline news you need in around five minutes. We kick off today with a couple pieces of OpenAI news, the first being quite the tea. According to the information, OpenAI has fired two researchers for leaking sensitive information. The two researchers include Leopold Aschenbrenner and Pavel Ismailoff. Both of these researchers have been on the team dedicated to AI alignment and safety research. And according to the information, Aschenbrenner was also an ally. Their word of OpenAI Chief Scientist Ilya Sutskever, who of course we have no idea where he actually is or what he's actually still doing with OpenAI. We also have no information about what the two allegedly leaked. Right now, it's all a big bucket of speculation. The information talks about Aschenbrenner's ties to the effective altruism movement, the most notorious proponent of of course being Sam Bankman Fried. And the intimation is that the leaks probably had something to do with a disagreement around how OpenAI was proceeding with regard to safety. But again, that's all speculation right now. Whether these two actually talk or if they just show up at a competing lab remains to be seen, but that is the story. Next up earlier this week OpenAI announced a more advanced GPT4 Turbo with Vision model, but it was only available initially through the API. Well, now they've brought that new advanced model to ChatGPT, the company writes. Our new GPT4 Turbo is now available to paid ChatGPT users. We've improved capabilities in writing, math, logical reasoning and coding. For example, when writing with ChatGPT, responses will be more direct, less verbose, and use more conversational language. We continue to invest in making our models better and look forward to seeing what you can do. So the example they give of it being less verbose the prompt was SMS reminding friends to RSVP to my birthday dinner invite. The previous response was hey friends name. Just checking in to see if you've had a chance to RSVP for my birthday dinner. I'm finalizing the table reservation and really hope you can make it. It wouldn't be the same without you. Please let me know by RSVP deadline. Looking forward to celebrating together. The new response is hey friend's name. Just a friendly reminder to RSVP for my birthday dinner. Hope you can make it. Let me know soon. Now of course the proof is in the pudding and so far I've seen a lot of people say that this does feel like GPT4 is now once again caught up with what they were seeing with Claude 3 opus. There's some disagreement around whether that's the case with coding, but I've seen some people talk about how the coding has gotten significantly better as well. Of course there is a larger question, which is why we're only getting these very incremental improvements rather than some big model launch, and a lot of speculation that OpenAI is still very much holding things back. Next on the brief, Amazon has added well known computer scientist Andrew Ng to its board. Andrew is an extremely well known figure in the AI space, having led projects at Google and Baidu and seemingly showing that Amazon is here to compete in this area. Finally, the Humane AI pin is getting lots of reviews, and they're not all that great. For example, the Verge's review is called Humane AI Pin Review. Not even close. For $6.99 and $24 a month, this wearable computer promises to free you from your smartphone. There's only one problem, it just doesn't work, the reviewer writes. I came into this review with two big questions about the AI pin. The first is the big picture. One Is this thing anything? In just shy of two weeks of testing, I've come to realize that there are in fact a lot of things for which my phone actually sucks. Often all I want to do is check the time or write something down or text. My wife and I end up sucked in by TikTok or my email or whatever unwanted notification is sitting there on my screen. Plus, have you ever thought about how often your hands are occupied with groceries, clothes, leashes, children, steering wheels, and how annoying unsafe it is to try to balance your phone. At the same time, I've learned I do lots of things on my phone that I might like to do somewhere else. So yeah, that is something. Maybe something big. AI models aren't good enough to handle everything yet, but I've seen glimmers of what's coming and I'm optimistic about the future. That raises the second question. Should you buy this thing? That one's easy. Nope. Nuh. No way. The AI pin is an interesting idea that is so thoroughly unfinished and so totally broken in so many unacceptable ways. I can't think of anyone to whom I'd Recommend spending the $6.99 for the device and the 24 monthly subscription. Now there really was not just one nasty reviewer, there were a bunch of people who said something similar. However, for a more generous and let's call it historical point of view, I'll read this tweet from investor Khalid Jalanbo, who writes with AI Pin Humane will go down in history as the company that created the world's first ever AI computer. Yet as history proves, version 1.0 is just the beginning of a founder's vision. Those who understand what it takes applaud entrepreneurs that have the guts and imagination to challenge giants rather than scrutinize them like they're already trillion dollar mega caps. The Humane team are pioneers that catalyze the conversation around how we apply AI in consumer hardware, and that in itself merits recognition and respect. I think that's a great way to put it, and very clearly we are at the beginning of a flowering of new, interesting experiments when it comes to AI hardware. Who knows where that will all land? That's going to do it for today's AI Breakdown brief. Next up, the main AI breakdown. Welcome back to the AI Breakdown. Today we are catching up on the AI chip wars, which, if they get any more intense, are really going to start being reminiscent of Game of Thrones pretty soon. So let's start with the most recent story, although just one of many from this week. Yesterday, Bloomberg's Apple whisperer Mark Gurman reported that Apple is planning to overhaul their entire Mac line with a new custom silicon chip they're calling the M4. It's obviously the fourth in their line of M chips, and this one is specifically focused on AI. Now, what's not surprising is that Apple is thinking about AI and leveraging their custom silicon to try to win an advantage in that space. In fact, a lot of our conversations about Apple's strategy, or likely strategy when it comes to AI have been about how it is trying to, on the one hand, increase the power of chips in its devices and on the other hand shrink powerful models such that they can be running on device without having to touch the cloud. What is surprising about the announcement is that the M3 was first released just five months ago. That's an extremely short turnaround even for Apple, which has a reputation for making its old technology obsolete very quickly. According to sources, the M4 chip will come in at least three varieties, and Apple is planning on updating every single Mac model with it. Now, obviously on this show we are focused on the AI dimension of this, but Just to understand the broader business dimension for Apple, their Mac business is tough right now. Mac sales fell 27% during the last fiscal year and even during the holiday period revenue was flat. Between that and their seeming sort of strategic flailing given their recent ending of Project Titan, their electric car project, their their unclear AI strategy overall, it feels like Apple needs to get things back on track. Investors seem to like this report. Apple had actually been down 13% on the year, very different than the other AI focused tech stocks, but on Thursday saw their share price go up 4.3% which was their biggest single day gain in 11 months. The names of the M4 chips include Donen, which is the entry level version, a middle powerful version called Braava, and a top end version called Hydra. These are their code names, so they might not be called that when everything comes to light in terms of getting more information sources. Expect us to hear more about these chips and the devices that they will run in at the Worldwide Developer Conference, which is coming up in June. Bloomberg also writes, as part of the upgrades, Apple is considering allowing its highest end Mac desktops to support as much as a half terabyte of memory. The current Mac Studio and Mac Pro top out at 192 gigabytes, far less capacity than on Apple's previous Mac Pro, which used an intel processor. The earlier machine worked with off the shelf memory that could be added later and handle as much as 1.5 terabytes. With Apple's in house chips, the memory is more deeply integrated into the main processor, making it harder to add more. So one of our competitors in the game of thrones for chips is Apple, but an old player seems to be coming with some new tricks as well, Ars technica writes. Intel's Gotti 3AI accelerator chip may give Nvidia's H100 a run for its money. Intel claims 50% more speed when running AI language models versus the market leader. So basically this week intel held its Vision 2024 event and their big reveal was a new AI accelerator chip that they're calling the Gaudi 3. Now intel is pursuing a number of different strategies simultaneously. They're increasingly competing with TSMC as a chip fabricator that is a builder of other people's chips. But the Gaudi 3 suggests that they're not giving up on their own chip business either. Now it is important to Note that although Gaudi 3 is projected to have faster performance in terms of training time and inference than Nvidia's H100, H100 is no longer Nvidia's most powerful chip, the H200, is not out yet, but is expected basically any day now. And then there is of course the Blackwell B200. One of the big questions is what the price of the Gaudi 3 will be. Analysts think that it could be an attractive alternative if it can come in meaningfully under the 30 to 40,000 in H100 costs. I think the big takeaway from this Game of Thrones perspective is that intel seems willing to fight on multiple fronts at once and is putting out some pretty compelling products. Moving on in our battle, Google has announced its own ARM based CPU to support AI work in data centers and is also introducing a more powerful version of its TPU or Tensor Processing Unit chips. The new CPU is called Axion and will be initially used to support Google's internal AI workloads before rolling out to business customers of Google Cloud later this year. Google said that the Axion chips are already powering YouTube ads, Google Earth and other Google services, said Mark Lohmeier, Google Cloud's Vice president. Axion is built on open foundations, but customers using ARM anywhere can easily adopt Axion without re architecting or rewriting their apps. Reuters reports that Axion ARM based CPU will offer 30% better performance than general purpose ARM chips and 50% more than Intel's existing processors. Google also discussed their overall Cloud strategy in terms of what chips they offer, specifically announcing that they are not offering AMD chips again. Mark Lohmeier said, at this time we're deploying Nvidia GPUs and our own TPUs. And he added that they are also supporting Intel's new central processing unit, which he claimed was quote, good for inferencing certain workloads. Lohmire said, we offer those three and we feel good about that combination for our customer base. Meta also made a chip announcement which was perhaps understandably a little bit overshadowed by the fact that they revealed that llama 3 was coming soon. But the company says that their new Meta training and inference accelerator MTIA is a big piece of its long term plan to build infrastructure around how it uses AI and its services, Meta said in a post. Meeting our ambitions for our custom silicon means investing not only in compute silicon, but also in memory, bandwidth, networking and capacity as well as other next generation hardware systems. Meta had first announced their V1 of the MTIA in May of 2023 and said at the time that they were focused on providing those chips to data centers. That V1 was not expected to be released until 2025, but meta now says that both that first version and a new Mtia chip are in production, writes the Verge. Right now, Mtia mainly trains ranking and recommendation algorithms, but Meta said the goal is to eventually expand the chip's capabilities to begin training generative AI like its Llama language models. Over in the world of Amazon, CEO Andy Jassy discussed the chip wars in his annual letter. He wrote, to date, virtually all the leading fms have been trained on Nvidia chips, and we continue to offer the broadest collection of Nvidia instances of any provider. That said, supply has been scarce and cost remains an issue as customers scale their models and applications. Customers have asked us to push the envelope on price performance for AI chips, just as we have with Graviton for generalized CPU chips. As a result, we've built custom AI training chips named Trainium and inference chips named Infertia. In 2023, we announced second versions of our Trainium and Infertia chips, which are both meaningfully more price performant than their first versions and other alternatives. This past fall, leading FM maker Anthropic announced it would use Trainium and Inertia to build, train and deploy its future FMs. We already have several customers using our AI chips, including Anthropic, Airbnb, Hugging Face, Qualtrics, Ricoh and Snap. So the point here is not necessarily that Amazon announced anything new, but that the battle is significant enough that it warrants mention in this annual letter. But what about the geopolitics of chips? Obviously this is a huge global issue and we've gotten some news recently in that respect as well. The Biden administration is continuing to distribute money through the Chips act, with the latest recipient being TSMC, who are receiving a $6.6 billion grant to increase their US manufacturing. TSMC is of course building out their first major US plant in Phoenix. It's actually two plants with this money being used to build a third. And as part of the announcement, TSMC said that they're increasing their total investments in the US from 40 billion up to 65 billion. This is so far the second biggest grant under this program, with the first being the 8.5 billion in grants plus 11 billion in loans that were announced for Intel a couple of weeks ago. Now, as an aside, TSMC is doing well right now. In general, they saw a 16% rise in quarterly sales, which outstripped projections and also was the fastest growth in more than a year. Another chip exporting nation, South Korea, is investing around $7 billion to stay competitive in chips as well, said South Korean President Yoon current competition in semiconductors is an industrial war and an all out war between nations, writes Reuters. Yoon has set a target for South Korea to become one of the top three countries in AI technology, including chips, and take a 10% or more share of the global system semiconductor market by 2030, Yoon said. Just as we have dominated the world with memory chips for the past 30 years, we will write a new semiconductor myth with AI chips in the next 30 years. So lots and lots going on in the AI chip world. Like I said, this is a battle that is not slowing down even a little bit. That is going to do it. However, for today's AI breakdown, until next time, peace. Sam.