Vision of PyTorch | PyTorch Developer Day 2020

[FOREIGN LANGUAGE] Hi everyone, I'm Lin Qiaoqiao. Today I will walk the journey we had, our core principles as an anchor, and our future focus for PyTorch. In 2016, we identified a gap in what research needs and what's available. We saw the need for a more flexible and easy-to-use frontend for more complicated neural network architectures. Build a small, flexible frontend in Python with dynamic graphs with Torch 7's backend. This is built upon ideas from Trainer and TorchGardlegrad. A key principle for us was to interoperate with as much of the Python ecosystem as we can and be a good citizen. As a result, PyTorch played well with NumPy and other Python staples. We welcome everyone to contribute, co-develop, and collaborate however they best could. Researchers then want to scale to multiple machines, so we built distributed GPU backend. As we grew, researchers wanted to build larger and larger models, so we built torch.distributed. And then they want to build small models in NLP, in the reinforcement learning, we built in DWA-10 a fast and lower overhead engine. Later on, they become product owners, and many of them went on to either start a company or be part of larger ML deployment of their research. They asked for better production support and performance. We took this as a challenge to keep research flexibility and yet enable production. In order to serve that need, we made a huge pivot of PyTorch 1.0. We merged PyTorch with Caffe2 and built TorchScript to support production. Everyone wanted options on hardware backend, so we worked with many hardware vendors to increase hardware portfolio for PyTorch. From early on, we built small domain libraries but rely on the community for empowering a diverse set of ideas in domains such as object detection, privacy-preserving ML, graph neural networks, and healthcare. During PyTorch production development, we discovered huge demand in having the building blocks of serving and deployment, model lifecycle management, workflow management on top of PyTorch. Such ecosystem needs to be modular, flexible, and has extension points to launch either on-premise settings or various cloud environments. Privacy brought a critical transition of AI technologies. We now have an ecosystem of privacy-preserving ML libraries to offer tools in privacy-sensitive areas, thanks to our community. Now I would like to share the core principles we're holding near and dear to our hearts. Ultimately, these are the 4 guiding principles that we worked with. Our community in the center to empower us to build out PyTorch for enabling cutting-edge research, PyTorch to be highly performant with production-grade quality, and the PyTorch ecosystem for smooth user experience and high productivity. We dance to the tune of machine learning. A research community will keep investing whatever the community goes and always. As PyTorch production adoption grew, we support more users and scale. That also means more responsibility. We need to enable both research and production. There's a natural tension between these two, as in researchers tries to break free from the existing set of ideas, while performance and production try to overfit to current paradigm. It's critical to keep this tension alive at the right sweet spot. To address that, community-wide adoption of PyTorch help us analyze a comprehensive set of workload and drive informed decision on how to strike a sweet spot in the balance. We're also serving a variety of users. Not everyone needs everything. They should only cognitively pay for what they use. If we don't enforce this in our overall API and implementation, it'll be impossible for us to move forward. We hope to not become a slow-moving project, and this principle is critical in making sure of that. Our ecosystem aims to provide a comprehensive coverage, modularity, and interoperability to allow users to only use what they care about. The only way machine learning community scales is to help each other. A single framework team cannot do all the software development. We will keep building the generic building blocks and make informed bets together with our community. In addition, our community also serves as a natural regularizer to avoid overfeeding to the ideas of a few. There are many examples of enabling new ideas via ecosystem strength. Everyone benefits from each other. This is only a small set of PyTorch ecosystem, but good example of the organic growth. We recognize how small we are, We have always strived to play well with everyone in the Python ecosystem and beyond. We're not going to have all the innovations and ideas ourselves. These principles have worked well so far for PyTorch. We have more than 600 contributors, 45,000 GitHub downstream projects, 34,000 users in discussion forums. Now I would like to talk about the focus in the next few years. We want to continue to enable cutting-edge research. That means we need to build out better frontend APIs, including NumPy compatibility, complex numbers, other language bindings, and support for research reproducibility. We want to enable new modeling paradigms, and there will be support for sparse data and even tensors for graph neural networks. And then we want to continue to build out domain libraries to speed up research innovation velocity. We built Torchvision, Torchtext, Torchaudio, and there's so many more new domains to build. With our community together. And next, we want to continue to push PyTorch production. That means we need to accelerate a wide range of production models, achieve 100x speedup in speech, CV, recommendation domains, and so on. And we're making great progress in that regard. We want to build and leverage AI compilers. We already have PyTorch JIT. To convert from type PyTorch eager mode to graph mode. Then we'll do graph optimization, scheduling optimization, and do code generation to a specific hardware backend. We need to increase the diversity of hardware portfolio. Want to cover both server-side accelerators, embedded hardware, support both training and inference workloads. And next, we want to push on model scaling. And here we're thinking about supporting the trend of significant increase in model complexity. This is where research and production boundaries are blurry. And here we're thinking about trillions of parameters, petaflops of computational intensity, tens and hundreds terabytes of embedding table size. And this will push the boundary of large-scale training to the next level. And here we need to build out distributed data model pipeline parallelism to the next level. Elastic fault-tolerant training, high-bandwidth data loading, and even explore SSD, DRAM, high-bandwidth memory hierarchy. After we have the large model trained, we need to deploy into production. So here we need to invest in different model optimization techniques to shrink the model down or quantize the model and so on. We also need to potentially explore distributed inference. And here we'll be battle testing all the large-scale training and inference at Facebook production scale, as well as in the industry. And next, we want to continue investing on device AI, and this is a very critical area. First, we build a unified runtime for wide coverage across a wide variety of hardware, including more than 5 compute CPU microarchitectures, 8 versions of Android, 4 different numerics, and more than 12 microarchitecture for specialized hardware like DSP, GPU, NPU from more than 5 SoCs. There are specific model optimization techniques to help us strike a balance across power efficiency, latency, accuracy dimensions. And also we want to enable end-to-end workflow for on-device AI pipelines. Eventually we would like to see us enable on-device AI for more than multiple billions of devices. To get there, let's zoom in to look at end-to-end workflow. First, we need to enable, obtain, and prepare data, potentially fetching extra signals from feature storage, turning data from data frames into tensors ready to be consumed by the model, constructing model from scratch or pull the best architecture components or pre-trained components, And then we analyze model understanding and attributing where the accuracy differences are from, do AutoML-like techniques from simple hyperparameter optimization, neural architecture search. Then to shrink the model down for serving to data center or on embedded device. And there are additional things like pipeline authoring, AI artifact management, production deployment monitoring. All those are very important aspects. But the best user experience can only be provided by co-engineering with many other awesome projects. There's a ton of opportunity you can help building it up. Concretely, we would like to build a cloud-agnostic, open-source, and end-to-end machine learning production workflow. We believe this will lower the cost for more users to productionize their models, but also maintaining a significant amount of customization scalability. Here the challenges we face, very diverse infrastructure options. That's why Databricks, Amazon, Microsoft, Facebook, and many other companies joined force to tackle this challenge together. This is a super exciting journey and those critical partnership is important to build out PyTorch open source production ecosystem. In addition, there are libraries and tools to significantly increase machine learning engineers' productivity, in-domain libraries to speed up research, privacy-preserving ML to rethink data and AI in the necessity of privacy. The list keep goes on and on. To achieve all this, we have worked together as a very tight-knit developer community. However, GitHub tooling is no longer scaling for us to connect all the PyTorch developers closely. We need to do more at facilitating high signal developer communication. There are many ideas on the table, such as dev mailing list, open roadmaps, meetups, better relationship building, and so on. Let's all work together to define how to build our developer communities in a more efficient way. Thank you all, and looking forward to work closely with all of you.