Data + AI Summit Keynote, Thursday Part 7 - Lin Qiao

Hi everyone, I'm Lin Qiao. Today I will talk about developer-centric AI, the past, present, and bright future. I have always been passionate about building awesome tools, giving to developers to do the best in their jobs. When I started my career, I built one of the earliest in-memory columnar databases, fast document store with inverted indexes, and many data tools from logging, metrics, experiment platform, and so on and so forth. I moved to AI about 6 years ago, seeing a huge surge of data volume driven by AI workload. I led PyTorch team at Meta, accelerate AI researchers, um, in open source and bring huge production workload in production at meta scale. After having done all this, I started Fireworks.ai with Ning Qiao, Dimitri, and awesome group of engineers. Continue our journey to bring awesome tools to broader set of developer here, including all of you in this audience, and, uh, help you to be the most equipped, prolific, creative in this new AI generation. Fireworks aims to accelerate product innovation building on top of GenAI. We offer experiment and production platform where you can get access to the most state-of-the-art research of open-source models. Bring your data and use our pre-built tuning evaluation recipes Customize your model, deploy easily into our inference services, optimize for cost to serve, and then we free you up to focus on your product innovation, ideation, and iteration. When your product idea is ready to deploy full speed, full scale to production, you continue to stay with our inference service and go from there. Before I talk about more details of product and why us, Let me start with where AI journey began. AI and deep learning is not new concept. More than 30 years ago, Yann LeCun, as one of the pioneers of deep learning, published a paper in 1989, backpropagation applied towards handwritten zip code recognition. The model still reads very modern today. From that point, for many years, AI remained as a research topic. Researchers need a maximum amount of flexibility to innovate and maximize model quality most efficiently towards the Trinity that they have. When we started, PyTorch was started for researchers to explore and innovate faster. And we observe around every 3 years major model architecture disruption happens, and every few months significant incremental improvement on model quality happens. The pace of value created by research hasn't slowed down but accelerated in recent years. What is PyTorch? You can think about it as NumPy for AI accelerated by GPUs, TPUs, your accelerators. Of course it runs on CPU as well. PyTorch uses model-as-code philosophy for dynamic neural networks, including dynamic shapes, control flows. Those are essential for AI researchers to disrupt, building disrupting building blocks in AI. Why model-as-code is so good? Because programming first— program-first structure is the most open and composable, as opposed to rigid, closed closed systems with hidden functionalities. Many research institutions, including OpenAI, switched over to use PyTorch a few years ago for its flexibility to accelerate their innovation speed. So many researchers become entrepreneurs or move on to deploy models in production. Production deployment shifts the focus on system performance stability, versioning, scalability, cost to serve, and so on. As you can see, there are quite natural tensions across these two cohorts of AI developers. Researchers naturally gravitate towards breakthroughs and break current paradigms to new ones, and production developers optimize for current paradigms. PyTorch continues to speed up research-to-production transition but ease the tension in between. And that's my focus in the past 6 years. One of the most important use cases we innovate on is ranking recommendation because it's everywhere in our digital life. PyTorch team released recommendation libraries last year, driving more than 10 times performance improvement and memory utilization reduction. It makes the large complex recommendation model practical to be deployed in production. We continue to, to bring more awesome models on mobile devices, which is more resource and power constraint. It requires order of magnitude improvement of reduction of PyTorch runtime size and the latency improvement. As you can see, there are many cool features we enabled through this on mobile products. PyTorch is today deployed very broadly in production, driving more than 50 trillion inferences per day on servers and more than 70 billion inferences per day on mobile phones. PyTorch application industry has penetrated across many domains, from content understanding of language, vision, speech, to complex task of ranking recommendation robotics, to high precision task of medical cancer diagnosis, treatment, drug discovery, many variations of autonomous driving vehicles. And PyTorch built a vibrant community, as we can see from the implementation of archive paper covering both research, and production. It's constantly growing over the past years towards a dominant position. As GenAI is the most interesting topic in this summit, most of GenAI models are written in PyTorch. So we are actually going through a very interesting model development paradigm shift now. From before where a lot of individual developers or institutions are spending lots of energies collecting, curating, cleaning up data, and training a particular model architecture from scratch, and then deploying to production, to now you can easily customize your model using much less data. As you can see from this paradigm from bottom up, there are many foundation models already pre-trained and the weights released based on public data available on internet. And then you can specialize those foundation models with your domain-specific knowledge. For example, for finance, legal, healthcare, and so on with your, with your company proprietary data. Then you can further customize towards your company-specific product by building your moat. And further along, you can even provide a personalized model For example, building avatars or AI assistants to make your product even more interesting. So we have a foundational shift right now because of GenAI in AI development. You still get the state-of-the-art models from research, but with much less data, you can start to customize and build interesting products. And do product exploration much, much faster. One of many product ideas you tried get into full-scale production, and your production drives more data, pushes better model quality improvement, and you experiment and explore more product ideas. And this quick iteration cycle is new. As you can see, in AI Developer Cohort, we have a growing, a new growing product developer coming up like many of you sitting in this, in this hall right now. So Fireworks is created to accelerate product innovation building on top of GenAI by solving many of the challenges we heard from many of you. First of all, the challenge is, is there's a huge amount of dynamism of GenAI models. And there's a stark competition of many model providers constantly improving model qualities one after another. It is very concerning to many people. How do, how do I manage this ever-changing AI landscape? And which model should I use and stick to? As you can see from the right-hand side of diagram, with increasing timeline, the intensity velocity of model innovation has accelerates. Second, business task is never simple. It usually is a complex business workflow that needs to be decomposed into one or multiple AI models. And which layer in this hierarchy do you interject? And there are many interesting business problem people coming to us and ask how to solve that. For example, I want to use natural language as the interface to retrieve precise information from my company internal documents. Or precise information from my product catalog. Or I want to automate my construction workflow to minimize timeline and minimize cost. Or I want to create diagram charts at scale based on my data. This is a first-mile problem, and many people get slowed down not knowing where to start. More specifically, within each model family, usually there are different model sizes being provided from small, medium, large, sometimes to extra large. And you can choose to zero-shot access the base model or prompt tuning to get better quality or fine-tuning user data to get even better quality. For specific tasks, you can actually fine-tune an equivalent quality or better model with a smaller size. So which one would you choose? Of course, AI, general AI is expensive because it's computational and memory intensive. In the past 2 to 3 years, AI research has been pushing for bigger and bigger model size until Chinchilla Law gets established that there's an optimal token-to-parameter ratio about 20 to 1. But still, most people are using around 7 billion to 13 billion model parameters, and that's much bigger than traditional machine learning algorithms, for example, of XGBoost. Many companies get slowed down or stuck because of the, um, the intensity of the computational cost and quickly run out of preplanned hardware budget. So we think the competition in the JML model space is actually great because the product developers and machine learning engineers can ride a free wave of ever-rising model quality. There will not be one model provider dominating the whole space. There will be many of them keep advancing the quality altogether. Fireworks want to enable you to continuously testing and evaluating the best model to incorporate into a product. Yes, continuously, because the cadence of change here is extremely fast. Second, cost to serve. We love this because we have been driving cost to serve price if performance and general performance optimization for PyTorch for many years at hyperscalers. We hyper-optimized the cost for fine-tuned models. Fireworks today offer at equivalent model size 5 times cheaper cost compared with OpenAI-based model and 30 times cheaper cost compared with their fine-tuned models. That's a whopping cost reduction. We also quickly optimize latency for open-source models. For example, a nascent one that we deliver more than 10 times latency reduction based on the original implementation. This requires deep PyTorch runtime optimization and integration with the Bask kernel, as well as carefully crafted scheduling, orchestration, design, and optimization all coming together. We also understand to accelerate your product exploration, you need to do a lot of concurrent parallel experiments at the same time. And oftentimes with increasing number of experiments, it increases your memory footprint as well. That means you have to pay more. We want to take away that concerns from you. As you can see, we can effectively control the memory footprint with increasing degree of experiments. That means with fixed budget, you can run a lot more experiments at same time, and your time to market will significantly be reduced. Tying all this together, Fireworks provide a rapid experiment and production platform. We give you free access to the state-of-the-art research with your own data. You can use our pre-built fine-tuning and evaluation recipe. You can change it, adjust it, adapt it as you wish. Run the fine-tuning on-prem or on cloud as you wish. And then your custom model will be deployed to our General AI inference service with minimal cost to serve. You can then focus on your product iteration and exploration. When you like your idea and want to grow it to full production scale, do that within the inference service without changing to anything else. Today we build our inference service on top of GCP, AWS, and CoreWeave. We plan to expand our cloud provider partnership quickly down the road. Our API is fully compatible with OpenAI API and LangChain. So if you, you don't need to change your code much to use our services. If you feel the challenges I just mentioned resonate with you and also want to know more about Fireworks, find me after talk and also find the awesome Fireworks AI team. We can give you a demo and give you more information about what we're building. Also, stay tuned for many more exciting announcements coming up soon. Thank you. Looking forward to seeing all of you. [FOREIGN]