Nvidia’s Blackwell is Radically More Powerful AI Compute

Today on the AI Breakdown, Nvidia announces their new Blackwell system. Before that, on the Brief, the Department of Homeland Security announces its own AI pilots. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our Discord, our newsletter and our YouTube channel. Welcome back to the AI Breakdown Brief. All the AI headline news you need in around five minutes. One of the things that I've often noted on this show is the sort of disparity between, on the one hand, the talkie side of Washington D.C. the politicians, Congress, the Senate, et cetera, running a seemingly never ending set of meetings and discussions and hearings on AI legislation without ever really doing anything about it as compared to the US Military establishment as well as private agencies who are just moving ahead full steam. For example, we talked about how the State Department is hosting a military conference on AI this week and yesterday the Department of Homeland Security also announced three pilot programs around generative AI. So what are those three efforts? The first is basically to use AI to better extract information from the huge volume of investigative reports and other materials that the department creates. As Axios sums up, investigators will be able to use LLMs to more quickly summarize investigative reports and to improve the process of searching through reports. The agency hopes the pilot program will improve detection of fentanyl related networks and help identify perpetrators and victims of child exploitation. The second initiative is a training focused initiative that is going to attempt to use generative AI to personalize materials to help train immigration officers more effectively and in a way that is more up to date with changing legal issues and policies. Finally, they are trying to simplify and reduce the time cost around resilience and disaster planning with fema, making it easier, for example, for communities to submit grants and gain funding in these areas. The New York Times characterized Homeland Security as embracing AI and says the agency will be the first in the federal government to roll out a comprehensive plan to integrate the technology into a variety of uses, said Alejandro Mayorkas, the Secretary of dhs. One cannot ignore it. And if one isn't forward leaning and recognizing and being prepared to address its potential for good and its potential for harm, it will be too late. And that's why we're moving quickly. So how big is the scope of these initiatives? Well, for some reference point, DHS employs 260,000 people for these new AI pilots. The Planning to hire 50 AI experts and spend $5 million overall, meaning in other words, that these are relatively small efforts. However, I think the fact that they are doing a full court press strategy around this indicates where it sits in terms of a priority. As part of the initiative. They'll be working with OpenAI, anthropic and meta, and we'll also use Microsoft, Google and Amazon Cloud in these pilots. Basically, it sounds to me like they're testing everyone credible to see whose tools are the most helpful. Importantly, this is not something that's meant to be drawn out. The agency has to report on the results of the pilot programs by the end of this calendar year. I think it will probably not surprise you that my very strong assumption is that we're going to see a lot more efforts like this coming from basically every area of government. One additional area of overlap between the AI industry and the US government may be in combating spying and cybersecurity issues. Another piece from Axios today is called Insider Threats Are AI Developers Next Hurdle? They write. AI developers hiring quickly to keep pace with market demand are struggling with a new threat spies and employees looking to steal company secrets. USAI companies are likely already prime targets for nation state adversaries espionage campaigns. Experts predict that AI developers could become even bigger threats than chip manufacturers and biotechnology companies. Of course, we recently covered the Justice Department's indictment of a former Google software engineer for stealing AI secrets and sharing them with two Chinese companies. It seems to me like as the US Government gets more interested in this area, this could be a point of overlap for these companies. One more vaguely US Government Related Story the United States securities and Exchange Commission has recently at least partially moved on from its incredulity around the cryptocurrency industry to focus on warning of AI related scams. Reuters reports that the SEC has now fined two investment advisors over their AI claims. The SEC said that Toronto based Delphia Inc. And San Francisco based Global Predictions Inc. Agree to pay a combined $400,000 in fines to settle civil charges related to AI, Washington, writes Reuters. The SEC found that from 2019 to 2023, Delphia made false and misleading statements in SEC filings, a press release and on its website over its purported use of AI and machine learning. Global Predictions did the same in 2023 on its website and in social media. As part of the settlement, neither company was forced to admit or deny the SEC's charges. Given the small amount of the fines and the nature of the settlement, to me it just looks like a ratcheting up of the warning shots from the SEC around other investment advisors throwing around AI language too loosely. Over In Big Techland, YouTube has announced a new tool that will allow creators to self label when their videos contain AI generated material. Basically this is a new optional checkbox that comes up as creators are uploading and posting their content which asks them to disclose whether there is altered or synthetic content that seems realistic. Examples given include making a real person say or do something they didn't, altering footage of real events and places, or showing a quote realistic looking scene that didn't actually happen. The flip side is that they are distinguishing for things like beauty, filters, special effects and and clearly unrealistic content like animation. Right now these disclosures are voluntary. Finally today, an interesting chart on which jobs are most likely to be impacted by AI, with the source being a World Economic Forum report called Jobs of Tomorrow at the top of the heap. That report found that 73% of IT tasks could be automated or significantly altered, 70% of finance tasks, 67% of customer sales, 65% of operations, 57% of HR, 56% of marketing, 46% of legal and 43% of supply chain. Of course, one of the big questions when it comes to how AI does ultimately impact jobs is whether the disruption will be on the task level or on the actual job level. In other words, will this change what people do or will it straight up replace people or some combination thereof? This will obviously continue to be one of the most important questions in terms of society's relationship with AI, but for now that is going to do it for the AI Breakdown brief. Up next, the main AI Breakdown welcome back to the AI Breakdown. It's hard to overstate how unique Nvidia's position in the AI industry is. It's rare, first of all, to have a company so singularly associated with a new technological revolution, particularly an infrastructure company like Nvidia. But there's no denying that when people think of AI. Right now the only company that might have some more mind share or mental association is OpenAI, because they were the ones who introduced ChatGPT, which got this whole party started. Still, when it comes to the world of mainstream finance and just the mainstream in general, Nvidia is probably the company most associated with the generative AI transformation. It has over the course of the last year become the third most valuable public company and frankly it shows no signs of slowing down. This week is Nvidia is holding its GTC conference. This is an event that used to just be for hardcore computer engineers and for Nvidia customers, but now has taken on the trappings of a festival. Extending that analogy, the headline act yesterday was of course CEO Jensen Huang, whose announcement of Nvidia's new GPU platform Blackwell did not disappoint. Zairehul sums up Nvidia just announced Blackwell the most powerful GPU in the market. Main features AI Superchip 208 billion transistors 2nd generation Transformer engine 5th generation NVLink RIS engine 100% in system Self Test Secure AI Full Performance Encryption Decompression engine 800 gigabytes a second. That all sounds like Greek to you. Don't worry because the upshot comes in the next line when Cy writes Analysts estimate this could potentially be 10 to 100x faster than Nvidia's current Hopper A100 GPUs for very large transformer model workloads requiring multi GPU scaling. This represents a monumental leap in scale for accelerating the trillion parameter AI future Nvidia is targeting. In Jensen's presentation he showed one chart in particular that just did a great job of showing the exponential growth of AI compute. The headline reads 1000 XAI compute in eight years and goes back to 2016 when Nvidia introduced Pascal at 19 teraflops. The next year, 2017 saw Volta with 130 teraflops, 2020Ampere comes with 620 Hopper the next great update, what we've been working on 2022 with 4000 teraflops and now Blackwell with 20,000 teraflops. CEO Jensen really summed it up very succinctly when he said in his keynote, hopper is fantastic, but we need bigger GPUs and if the compute capacity of Blackwell is a lot of the initial focus, Nvidia is also clearly trying to tell a different story about its future, writes cnbc. Nvidia executives say the company is becoming less of a mercenary chip provider and more of a platform provider on which other companies can build software, said Jensen Huang. Blackwell's not a chip, it's the name of a platform. Nvidia's enterprise VP Manavir Das extended this conversation. The sellable commercial product was the GPU and the software was all to help people use the GPU in different ways. Of course we still do that, but what's really changed is we have a new commercial software business now. The software that they introduced was called Nim, and the basic idea is to make it easier for people to deploy artificial intelligence applications again from CNBC VP Doss said Nvidia's new software will make it easier to run programs on any of Nvidia's GPUs, even older ones that might be better suited for deploying but not building AI, said Doss. If you're a developer, you've got an interesting model you want people to adopt. If you put it in a NIM, we'll make sure that it's runnable on all our GPUs so you reach a lot of people. Now, if you are interested in some of the more technical parts of the announcement, there's tons that you can go find on that. For example, in his presentation, J. Jensen explained how the Blackwell GPU is the first chip to combine two separately manufactured dies into one chip. There were of course, a lot of other announcements from this presentation as well. We're going to talk specifically about Omniverse, which is software that creates digital twins of real world items and is slated to come to Apple's Vision Pro as well as Project Groot, which is infrastructure for humanoid robots. First, though, I want to talk about the market reaction to this, because again, part of what has put Nvidia at the center of the conversation is the fact that it has become the standard bearer for AI stock performance. On the one hand, as Bloomberg points out, the announcement of these new chips was widely anticipated, which quote made it hard for the presentation's details to impress investors who sent the shares down about 1.6% in pre market trading on Tuesday. Bloomberg gets at part of why that might be, they write. For all its success, Nvidia revenue has become highly dependent on a handful of cloud computing giants Amazon, Microsoft, Google and Meta. Those companies are pouring cash into data centers, aiming to outdo their rivals with new AI related services. The challenge for Nvidia is broadening its technology to more customers. And and that of course, is where we get some of this additional focus on Nvidia becoming a platform and its software business. And yet, if the immediate response of investors was not to send Nvidia stock to the moon, there is still a sense of long term strength. For example, Goldman Sachs raised their price target to 1,000, saying, we came away from the keynote with renewed appreciation of Nvidia's unique ability to innovate at data center scale, their large ecosystem and breadth of its customer and partner engagements, and ultimately compelling position as one of the key enablers and beneficiaries of the ongoing buildout of the generative AI infrastructure. Even more than just Nvidia alone this week, we've been seeing the same back and forth that we had throughout last year, where macro wobbles compete with AI enthusiasm to reign supreme in terms of how Wall street is thinking about the markets. Indeed, Reuters wrote a piece yesterday called Wall Street Ends Investors Juggle Fed Nerves with AI Enthusiasm. Investors, they wrote, were torn between enthusiasm about the prospects for AI on the technology sector and worries about the Federal Reserve's policy update on Wednesday, said Lindsey Bell, Chief Strategist at 248 Ventures, this is a market that really wants to hold onto the momentum trade, but what's weighing on investors minds is what happens with the Fed this week. Let's talk briefly though, about some of these other parts of the announcement. Jeremy Dalton, the head of immersive technologies at PwC, writes, Nvidia just announced it is bringing its Omniverse platform to the Apple Vision Pro, allowing complex 3D assets to be streamed directly to the headset for further design, assisted by generative AI. By doing this, you don't need to worry about storage or be constrained by the Device's processing power. VR filmmaker and YouTuber Hugh Howe writes, Omniverse shows the capabilities of spatial computing by blending 3D photoreal environments with the real world. Bilawal Sidhu is at the event and writes, I got to try this demo in person today. Mind blown Immaculate streaming quality full res CAD models with real time ray tracing. Biliwal shared the same demo video, which for those of you listening to the podcast, involves modifying a car. And when someone asked you were able to go in the car and still see the room around you? Biliwal responded, yes, there's a mixed reality mode and it was surprisingly good. Perhaps even more exciting to many was something called Project Groot. Dr. Jim Fan, who I often quote on this show, writes, today is the beginning of our moonshot to solve embodied AGI in the physical world. I'm so excited to announce Project Groot, our new initiative to create a general purpose foundational model for humanoid robot learning. The Groot model will enable a robot to understand multimodal instruction such as language, video and demonstration, and perform a variety of useful tasks. We're collaborating with many leading humanoid companies around the world so that GRP may transfer across embodiments and help the ecosystem thrive. Announced in Jensen's keynote, Project Groot is a cornerstone for the Foundation Agent Roadmap of the newly founded GEAR Lab. At gear, we are building generally capable agents that learn to act skillfully in many worlds, virtual and real. Nate Barsi tried to sum up Groot will enable a robot to understand multimodal instructions like language, video and motion. Very soon we will see them cooking, preparing coffee in supermarkets, changing tires, etc. So what makes something like Project Groot different? Than for example, the figure 01 robot that we've talked about recently is that this is meant to be a system for many different robots, not just one humanoid robot that Nvidia is building. So it's coming at the problem in a very different way, assuming a future in which there are many different types of humanoid robots. One additional note on Groot, Nvidia also announced a dedicated chip called the Jetson Thor chip that's specifically designed for humanoid robots, writes VentureBeat. To make sure humanoid robots can run complex multimodal models like Groot, Nvidia has launched the Jetson Thor computing platform for humanoids based on the company's Thor SoC. The computer includes a high performance CPU cluster, a next generation GPU based on the Nvidia Blackwell architecture with the transformer engine delivering 800 teraflops of 8 bit floating point AI performance. I think if you zoom out, one of the remarkable things about this really is just how unusual it is to have a hardware company like this, not even a consumer hardware company, but an infrastructure company, be so frankly hot and at the center of a transformational technology. It's a fascinating thing to watch and something that doesn't show any signs of slowing down. If and as there are more interesting announcements from gtc, I am sure we will be covering them. But for now that is going to do it for the AI breakdown. Until next time, peace.