François Chollet: History of Keras and TensorFlow | AI Podcast Clips

Let's go from the philosophical to the practical. Can you give me a history of Keras and all the major deep learning frameworks that you kind of remember in relation to Keras and in general tensorflow, Theano, the old days. Can you give a brief overview, Wikipedia style history and your role in it before we return to AGI discussions?

Yeah, that's a broad topic. So I started working on Keras. It was the name Keras at the time. I actually picked the name just the day I was going to release it. So I started working on it in February 2015. And so at the time there weren't too many people working on deep learning, maybe like fewer than 10,000. The software tooling was not really developed. So the main deep learning library was Caffe, which was mostly C. Why do.

You say Caffeine was the main one?

Caffe was vastly more popular than theano. In late 2014, early 2015, Caffe was the one library that everyone was using for computer vision.

And computer vision was the most popular problem.

Absolutely. Like convnets was like the subfield of deep learning that everyone was working on. So myself. So in late 2014 I was actually interested in RNN synchronic neural networks, which was a very niche topic at the time. H3, H3 took off around 2016 and so I was looking for good tools. I had used Torch seven, I had used a lot in Kaggle competitions, I had used Caffe and there was no good solution for RNNs at the time. There was no reusable open source implementation of an LSTM for instance. So I decided to build my own. And at first the pitch for that was it was going to be mostly around LSTM molecular neural networks, it was going to be in Python. An important decision at the time that was kind of non obvious is that the models would be defined in via Python code, which was kind of like going against the mainstream at the time because Caffe, Pylon 2 and so on, like all the big libraries were actually going with the approach of static configuration files in YAML to define models. So some libraries were using code to define models like Torch 7 obviously, but that was not Python. Lezine was like a Theano based very early library that was I think developed, I don't remember exactly, probably late 2014.

It's Python as well.

It's Python as well. It was like on top of Theano. And so I started working on something and the value proposition at the time was that not only what I think was the first reusable Open source implementation of lstm, you could combine ordinance and convnets with the same library, which was not really possible before. Like caffeine was only doing covenants and it was kind of easy to use because. So before I was using 10, I was actually using ScikitLearn and I loved ScikitLearn for its usability. So I drew a lot of inspiration from SciKitlearn when I met Keras. It's almost like SciKitlean for neural networks.

The fit function.

Exactly. The fit function. Like reducing a complex training loop to a single function call. Right. And of course some people will say this is hiding a lot of details, but that's exactly the point. The magic is the point. So it's magical, but in a good way. It's magical in the sense that it's delightful.

I'm actually quite surprised. I didn't know that it was born out of desire to implement RNNs and LCMs.

It was.

That's fascinating. So you were actually one of the first people to really try to attempt to get the major architectures together. And it's also interesting you made me realize that that was a design decision at all. Is defining the model in code just. I'm putting myself in your shoes. Whether the YAML, especially if Cafe, was the most popular.

It was the most popular by far.

If I was. If I were. Yeah, I don't. I didn't like the YAML thing, but it makes more sense that you will put in a configuration file the definition of a model. That's an interesting, gutsy move to stick with defining it in code.

Just if you look back, other libraries were doing it as well, but it was definitely the more niche option.

Yeah, okay. Keras and then Keras.

So I released Keras in March 2015 and it got users pretty much from the start. So the deep learning community was very, very small at the time. Lots of people were starting to be interested in lstm. So it was going to release at the right time because it was offering an easy to use LSTM implementation exactly at the time where lots of people started to be intrigued by the capabilities of RNNs for NLP. So it grew from there. Then I joined Google about six months later and that was actually completely unrelated to Keras who actually joined a research team working on image classification, mostly like computer vision. So I was doing computer vision research at Google initially and immediately when I joined Google, I was exposed to the early internal version of TensorFlow and the way it appeared to me at the time and that was definitely the way it was at the time is that the. This was an improved version of Theano. So I immediately knew I had to port Keras to this new TensorFlow thing. And I was actually very busy as a Noogler, as a new Googler, so I had not time to work on that. But then in November, I think it was November 2015, TensorFlow got released. And that was kind of like my wake up call that, hey, I had to actually go and make it happen. So in December I ported Keras to run on top of TensorFlow. But it was not exactly a port, it was more like a refactoring where I was abstracting away all the backend functionality into one module so that the same code base could run on top of multiple backends. So on top of TensorFlow or Theano. And for the next year the stayed as the default option. It was easier to use, somewhat less buggy, it was much faster, especially when it came to audience. But eventually TensorFlow overtook it.

And TensorFlow, the early TensorFlow has similar architectural decisions as Theano. So there was a natural, as a natural transition.

Yeah, absolutely.

That's still. Keras is a side, almost fun project, right?

Yeah. So it was not my job assignment. I was doing it on the side. And even though it grew to have a lot of users for a deep learning library at the time, like Stroud 2016, but I wasn't doing it as my main job. So things started changing in I think it must have been maybe October 2016. So one year later. So Rajat, who was the lead on TensorFlow basically showed up one day in our building where I was doing like. So I was doing research and things like. So I did a lot of computer vision research. Also collaborations with Christian Zeighedi and Deep Learning for Theorem Proving. That was some a really interesting research topic. And so Rajat was saying, hey, we saw Keras, we like it, we saw that you're at Google, why don't you come over for like a quarter and work with us? I was like, yeah, that sounds like a great opportunity, let's do it. And so I started working on integrating the keras API into TensorFlow more tightly. So what followed up is a sort of like temporary TensorFlow only version of Keras that was in tensorflow.com trib for a while and finally moved to TensorFlow core. And I've never actually gotten back to my old team doing research.

Well, it's kind of funny that somebody like you, who dreams of, or at least sees the power of AI systems that reason and theorem proving we'll talk about has also created a system that makes the most basic kind of Lego building that is deep learning, super accessible, super easy, so beautifully so it's a funny irony that you're responsible for both things, but. So TensorFlow 2.0 is kind of. There's a sprint, I don't know how long it'll take, but there's a sprint towards the finish. What do you look. What are you working on these days? What are you excited about? What are you excited about in 2.0? Eager execution. There's so many things that just make it a lot easier to work. What are you excited about? And what's also really hard? What are the problems you have to kind of solve?

So I've spent the past year and a half working on TensorFlow 2, and it's been a long journey. I'm actually extremely excited about it. I think it's a great product, it's a delightful product. Compared to TensorFlow1, we've made huge progress. So on the Keras side, what I'm really excited about is that previously Keras has been this very easy to use high level interface to do deep learning. But. If you wanted a lot of flexibility, the Keras framework was probably not the optimal way to do things compared to just writing everything from scratch. So in some way the framework was getting in the way. And in TensorFlow 2 you don't have this at all, actually. You have the usability of the high level interface, but you have the flexibility of this lower level interface and you have this spectrum of workflows where you can get more or less usability and flexibility trade offs depending on your needs. You can write everything from scratch and you get a lot of help doing so by subclassing models and writing some training loops using eager execution. It's very flexible, it's very easy to debug, it's very powerful. But all of this integrates seamlessly with higher level features, up to the classic chaos workflows, which are very psychically unlike and are ideal for a data scientist, machine learning engineer type of profile. So now you can have the same framework offering the same set of APIs that enable a spectrum of workflows that are more or less low level, more or less high level, that are suitable for profiles ranging from researchers to data scientists and everything in between.