deeplearning.ai's Heroes of Deep Learning: Dawn Song

Welcome, Don. And thanks a lot for doing this interview series.

Thanks a lot for having me here.

Of the people in deep learning, I think you have one of the most inspiring stories because you started off as a computer vision. Excuse me, started off as a computer security researcher, also using a lot of machine learning throughout that process, and then, but more recently became one of the great deep learning researchers, in addition to great computer security researchers, including I remember last year or this year being one of the most prolific authors at the ICARE conference.

Thank you.

And so your physics undergrad and then your path to get here has not been a linear path. Right. So tell me more about that.

So after undergraduate in physics, I first went to the PhD program in physics at Cornell for one year, and then I decided to switch to computer science. So I switched to computer science. PhD program first at Carnegie Mellon, I was there for two years, and then I transferred to UC Berkeley. So in the end I got my PhD in computer SC at UC Berkeley.

And when did you decide to switch from physics to security?

Back then, computer security was a very, very young and small field. But I could see that security is a really, really important problem. And I can see that as the Internet, as computers grow, security will become increasingly important.

And so you started doing security at Comedy Melon right away? Yes, with Doug Tiger.

So actually with both Ed Clark and Doug. So I also started early on looking at essentially how to. Because security, as I mentioned, so security is a field that you need to think about how to defend against this really smart attacker. But one of the best ways to do that now doing it as a cat and mouse game, is to have certain formal proofs of security properties of a computer system. And so I actually started very early on looking at how we can use the formal verification techniques to help us to provide formal proofs for security, properties of protocols and systems and so on. So it's very much intersection of aspects of security. They're looking at how we can bring more formal foundations and guarantees to the security systems.

So you have already been having a great career in security. So going from there, how do you also have simultaneously a great career in deep learning?

I do have some unique experience that I think can be helpful for people who are looking to how to get into deep learning and AI. So for me, yes, I've been doing computer security research for a very long time, but also for me personally, I've always had really broad interests as a researcher as well as a person, and I've always found AI and machine learning to be a really exciting field. And also, I think just always in the back of my mind, I've always thought that figuring out how the brain works and also how to build intelligent machines is some of the most intriguing and exciting questions for us to think about. So one thing that I did that's quite unusual, I would say, is that even back very early on actually the summer when I got my tenure from Berkeley, I decided to go to a summer school in Woods Hole, Computational neuroscience.

This one out on the east coast, right near Boston.

Yes, yes, in 2009. And that as a presence to myself for getting tenure.

But I think that's great. But you know, as a tenured professor, you decided to go play the role of student. Yes, something new. I think that's great. So, so, so you finished this month long course, learned a lot about neuroscience, and decided as a computer scientist you want to do more building and algorithms and so. And then what did you do next?

I spent on average like about four days a week just staying at home reading papers and reading books in deep learning. And I actually remember that I was really happy during that time. And I would say that was one of my happiest times for a long time when I was doing that.

Tell me more about that. That's great. Four days a week, you know, reading and studying. What did you. So what you get up in the morning and then read what, five papers.

And then, I mean, sometimes even more. Right. So for this week, so basically it's two days during the weekdays and two days on weekends. My life is in general fairly simple. And on weekends I actually really enjoy doing reading. I actually designed a reading program myself, figuring out what books I wanted to read and what papers I wanted to read. Of course, deep learning amount of papers has grown exponentially and there's just so much to read that also took time to figure out how initially, I think it's like anybody who enters a new field. Initially as you read papers it takes longer, but as you actually read more into it, then you become faster and faster as well. And also from the papers, the initial papers you read. And you also figure out who are the great authors to read and what are the follow up papers to read, and so on.

Boy, lots of great ideas here. I want to get a clear picture of how you got started and how others may be able to follow in your footsteps. So when someone wants to break in a new field, such as deep learning or an area in security, how did you go about finding the papers to read? When I'm trying to form reading lists for myself, I Often talk to my friends, you know, ask everyone I know and then do a lot of web searches, read the blog posts and look at a few papers and the references and use that to try to triangulate to like a top 10 or a top 20 or top whatever reading list. Right. Was that a similar process?

I think just to have a natural way to figure out what problems I'm interested in very early on, like even I think AI and deep learning, for example, very early on I figured out what are some of the interesting problems that I cared about and then.

And so you became well known, you know, for like your program synthesis work, your work in, you know, deep learning and security. But did you figure that out very early on?

From very early on, one of the key questions that I wanted to study in AI and deep learning is to learn how we can really build, how we can learn to build great representations. Because even to today I still feel that one of the core questions in AI and deep learning is what is the right way to really learn and really represent this representations? And I think even today still this is a wide open question. Even though I think as a co community we have made a lot of progress, but we are still just at the very, very beginning.

You know, the concept of representation is such a deep concept in AI. I feel like those of us that have worked in AI for a long time, there's a very deep meaning to that word. But can you explain a bit more what, what the term representation, I think.

It'S such a broad and deep, I mean in some it's a very abstract.

Concept, so it's hard to understand.

Right. And also in some sense you can also see there's a holy grail, like once you have the right representation in some sense, a lot of the other questions in AI and machine lear much simpler because of my goal is to figure out how to learn how to and figure out how to build intelligent machines. I realized very early on that the representations and reputation learning is a key question. So I actually from early on I structured my personal study program around that question. So I actually read very broadly to look at how different people have thought about representations, how even outside deep learning.

So you read very broadly about representations, Right, right.

I mean, of course again as I mentioned, still today it's a wide open area. We don't have any satisfying answers, I would say. But I think that was very helpful as a way to structure the learning process and also as a learning process going through then it helps me to learn more about different areas. I remember even I think that that.

Makes a lot of sense. Right after you've learned a broad overview, you picked an area which is representational learning to focus on because there's just too much stuff to read. And by having a core center in your case, you know the concept of representational learning, that allows you to do the literature search and focus your learnings more efficiently around. Around 1C.

Right, exactly. And in the end, I didn't really necessarily do particular research projects on representational learning itself, but it's such a fundamental concept and it really helped me to learn about the fields better. And then as I do other projects, for example, AI and security and also program synthesis, I think it's a very important foundation to help essentially doing research in these other areas too.

For Wordless Web, I think the concept of word embedding, so the, the idea of embedding representations, that was a huge step forward that you can. Yes, but obviously it's not the final answer. And word vector is not the final answer of how to represent words, far from it. But I think the idea that you can have a vector of real numbers represent very complicated things like a word or an image or a face.

Exactly. So even though I don't work in the area of computer vision, but I actually did spend a lot of time reading computer vision papers because again, you know, how humans is able to parse the visual world, how we are able to navigate in the 3D world and so on a key part of it is how we represent the world, how we visually and the reason about it.

And, and I feel like, you know, to first approximation, right, Very rough cut. I feel like maybe the way the human brain represents the world is there's a pattern of neuronal firings. Right. Maybe that's a. Okay, simplification. And if a pattern of neuronal firings can be approximated with a vector of real numbers measuring the intensity of the firing rate, then maybe a vector of real numbers is. Is actually maybe not that bad a representation for a lot of things. Not the final answer.

Right. I think, I think at the current stage that's a lot of deep learning work, how it is done. Um, but I think again, we are still at a very, very.

There's very surface level. There's something much more complicated to fig.

Yes, yes, I think you're right. I mean, you can say that it's a mechanism as the neuronal firing. That's one mechanism. But the actual representation, how it is done will be. Is much richer than that because the neuronal firing you can actually encode different types of representations. So for us really to learn what those kind of representations are. I think exactly. I think they'll. Yeah, it's such a wonderful thing for us to think about.

So. So I think that it feels like the 2009 thing. Right. Was a, was a little bit of a turning point or acceleration point for your interest in deep learning. But I think I remember actually even way back working on computer security, you're already applying machine learning to monitoring encrypted traffic. Right. To see if even encrypted traffic leaks some information. And then I remember we were talking about deep learning like way back. Right.

Actually okay. Even for that project, there's an interesting story that I took McJoyance class in I think graphical models Statistical machine learning. And I did that as one of the class projects. It was a simple. Hmm. By using hmm. To show that you can actually figure out what the user was typing even though the traffic was encrypted.

And does this change the way people encrypt traffic?

So certain things they know. So for example, so that was used for figuring out even for example for ssh you can figure out what users were typing for the passwords. So people need to be careful that.

So when people break into a new area, as you've sort of done, but you've been a machine learning person a bit for a long time, one of the things you did that I think is very interesting was work on deep learning but also simultaneously leverage your strength in security. Um, and so tell and, and so I think in your work I've seen a lot of using computer security to help AI or help deep learning and also using deep learning to help security. So, so maybe let me ask about the first one. Uh, just how, how. Tell, tell us, tell us a bit about your work on using your background in security to actually help your work in machine learning.

AI and machine learning really actually needs a lot of help from the security field for a number of reasons and in several different aspects. So one, given my experience as a security researcher, we have seen a lot in the Internet world. We are really attackers are there, they are incentivized, they are very well resourced and they are very creative in terms of developing new attacks to achieve their goals. We have seen over and over again every time when new technology develops and becomes more widely adopted, the attacks always, they follow very, very fast. We've seen that from, from the networking level initially, then from web mobile every step. As the computing technology improves and makes advancements, we see that the Attacks are also following very, very quickly. Given that I think definitely as we deploy AI and machine learning systems more broadly and have them take on more and more critical roles in our society, then attacks are going to be there and attackers are going to be there. And also this time I would say the stake is even higher because we are now more and more reliant on these AI and machine learning systems to make really critical decisions for us. So because of that, I think it's of paramount importance for us to figure out how to build AI machining systems that are resilient against attacks. So today we are seeing increasing attacks on machine learning systems. So for example, in industry people have seen evidence showing that attackers are actually using some of the advanced computer vision technology to solve captchas, which are these.

That was annoying. Like type the warped blurry words to prove your human test right.

Right, Exactly. And also for example, with the fraud detection, for example, when the bank, when some financial services need to decide who to give out loans, they typically use machine learning systems to figure out who should give out loans. What's the risk factor? Try to detect fraud. Again, attackers are actually being very smart trying to evade those machine learning systems. And so going forward, as I remember.

Was leading Baidu's AI group, there were definitely parts of my team that was at war with their forces. And it was a war, it was.

Pretty much, and it's a really as a fast evolving war, if they launch.

An attack, you better respond in a small number of hours.

Exactly. So going forward, as AI and machine learning systems being more widely used and also for making more and more critical decisions, we need to figure out how we can build resilient AI and machining systems that can defend against these increasingly smart attacks as well.

So you've been one of the leading researchers in building those resilient AI systems. Can you say a bit about the techniques that help?

I think so. We've done a lot of work trying to study essentially today's machine learning deep learning systems, how vulnerable are they, what kind of attacks we can launch against them and also then study what kind of defenses we can build. So for example, in one of our recent work, we also showed that these kind of attacks, for example, there's this adversarial example that can make very small modifications, for example on an image that can fool a deep learning based image classification system. And in one of our recent work, we also showed that this type of attacks can even be effective in the real physical world. For example, an attacker can modify stop sign a real world Stop sign just by putting on some small stickers on stop sign to humans. Humans can still recognize a stop sign with no problems. But for the computer vision system, the attack can cause the computer vision system to mistake it as a speed limit sign. So a car could autonomous vehicle powered by this computer vision system could then just drive through the stop sign. So this is just one example showing that why we really need to figure out how we can build resilience AI and machine learning systems to defend against attacks.

In fact, I guess in your early work, in the earlier work on this literature, the attacker needed full access to the neural network weights and parameters. But your team also showed that even when the attacker doesn't have that level of access. Do you want to say a bit about that?

So there has been research showing that this is called black box attacks, where the attacker doesn't even doesn't need to know what's what the neural network architecture is and doesn't need to know the parameters, even the neural network architecture for the deep learning system. So we actually have done a number of works showing that this type of attacks can be very effective by developing new attack methods including ensemble based attacks as well as essentially when choosing another way is that with black box attacks there are two different types. One is that you really just don't know anything about the victim model. And also you don't even have query access. So we call that a zero query access black box model. But also you can have a black box model where the attacker doesn't know the details of the model, but the attacker can have a query access to the model, for example, call their API and get it access. Right. So we have done work showing that in both cases attackers can launch successful attacks in this black box setting to create these effective adversary examples to fool the machine learning system.

A zero query black box model, this is without access to the training data, right?

Exactly.

And without access to architecture or access to anything at that store.

Right. So this is through transferability. So in one of the techniques that we developed is called an ensemble based attack, where essentially attackers can have compute itself a few white box models itself, for example, for computer vision image classification. And then it can use an ensemble of these white box models to then craft adversary examples that can fool this ensemble. As a very interesting phenomenon in machine learning and deep learning, that with high likelihoods this type of adversary examples can transfer to a victim model that's completely black box. And also we show that if the attacker does have a query access to the model, it can also help in certain Cases make the attack even more effective in the sense that here you can actually make an interesting game that attackers through this query interface can then gain information about how to develop more effective attacks, essentially as adaptive query, an adaptive game. And in certain cases it can succeed with very small number of quarries.

How about the defensive side then?

So on the defensive side it's still a very, very challenging open question. So today we still do not have a general solution that's strong enough to defend against strong and adaptive attackers.

And there have been some solutions like thermometer encoding that helps a bit. But what's your perspective on, I mean.

Still again, so today we really don't have very good solutions. So we do have a paper upcoming at ECCB that looks at one defense approach that I think in general could be a very interesting direction to explore further. We call it the approach as a form of consistency. Check this. In the area of looking at segmentation for humans, when we look around we can do segmentation, we can identify, separate out different objects. One interesting thing is that if I give you one image and I show you different sub patches of the image to a person, then the person will give the same segmentation even when the person only sees essentially in sub views like just different patches from the same image. So in this case the person's understanding of the image is consistent. So the segmentation from the different patches are the same, so they are consistent. So for example, attacker can fool a segmentation, a deep learning based segmentation system. One challenge is that then if you gave the attacker for example.

By forcing consistency, it makes it harder for the attackers, right?

Exactly. So attacker can generate, let's say one adversary example, a deep linear based segmentation system to give the wrong segmentation. But now if you take patches of this adversary example and you gave the patches to the vision to the deep learning based segmentation system, it will give you different solutions of segmentation. And it's actually difficult for attacker to generate an adversary example that would make these different patches the results, the segmentation result to be consistent. Because then the attacker not only has to fool the segmentation system once for one image, but also as the deep learning system look at different patches of the image, it needs to be fooled in a consistent way. So this makes the attacks job much much harder. And in our experiments we have demonstrated that at least in terms of data of the art attacks, this solution is fairly effective. So of course there's still more study that's needed. But I think overall as direction, I think this can be a very interesting Direction for us to further explore as building a defense against attacks. And I think in some sense, again, maybe not necessarily drawing on how human brains work because we don't really know how it works. But I do think in some sense that's also how we do it. Even when we make decisions, we actually pulling information from different sources and different angles and then try to see whether they are consistent. So I think this approach of having consistency checks as a way to detect attacks I think is a very fruitful direction.

You've also done a lot of work on AI and privacy. Can you say a bit about that?

Right. I think that's another area where security and privacy can really help AI and machine learning. As we know that one of the main power in AI machine learning is data. That's how we train machine learning models is actually from lots and lots of data. Unfortunately, a lot of this data can be very, very sensitive, can be uses text messages, can be sensitive images and so on. Hence as we build machine learning models, we also need to be very careful to protect users privacy.

So you don't end up with a language model that reviews someone's Social Security number.

Exactly. So that's one of the recent work that we did in collaboration with the researchers from Google. Then we actually demonstrated that if you train a language model from email, from emails. And so we show that the original email data set contained uses Social Security and credit card numbers for an attacker. By just querying the trained language model, the attacker actually can extract the original Social Security and credit card numbers that were in the training data sets. In this case, we actually do have a good solution to solve. The problem is that instead of just training a vanilla language model of a vanilla deep learning model, if you train a differentially private language model in this case, we show that we can significantly mitigate the attack.

And the differential private just means, you know, adding noise in an appropriately chosen way so that you still learn a good model but without memorizing the Social Security number.

Right. So there are many different ways to achieve differential privacy. In this particular case it is that during the training process we actually do add noise as you do. Right. The parameter.

See, Cool. I thought that's actually one of the most inspiring pieces of work on privacy actually that you could do that without hurting the performance in a material way, but while also protecting, you know, things like Social Security numbers. So I thought that's a very interesting piece of work when I came.

Yeah, yeah, thanks.

So in addition to all the work you're doing on security and privacy, helping AI You've also done a lot of work on AI helping the field of security. Can you say a bit about that?

Yes, I think security has lots of really challenging problems and definitely for some of these problems. And the machine learning techniques can be very, very helpful. So one of the recent work that we did is in the area of security for IoT devices, Internet of Things devices. A lot of these IoT devices, they use open source third party code in their firmware. Then a lot of these open source therapy codes contain vulnerabilities. Later on, after the IoT device has already been deployed, people discover vulnerabilities in these third party open source codes that was actually included in the IoT device firmware. So once that happens, how can you quickly tell Whether this particular IoT device firmware contains some of these newly discovered vulnerabilities? So essentially we need to do a very quick detection for what we call code similarity detection is to check whether the IoT device firmware contains similar code to a vulnerable piece of code. So this is a challenging problem because these devices can actually, it could be the same source code, but be compiled down to different hardware platforms. Oftentimes you will only have access to the binary code. So how do you do this kind of similarity check? We actually leveraged a graph embedding method in deep learning where we converted this binary code into essentially code graphs. And then we use the graph embedding methods to essentially check whether sitting like for example, two graphs are similar. And using this approach, we're able to identify actually detect vulnerabilities in real world IoT device firmware.

Yeah, that's really cool.

So that's just one example of how we can use deep learning for solving important security problem.

And so in addition to all this work on AI and security and privacy, you're also CEO of OASIS Labs. Tell us about that.

Right. So besides, AI and security have also been working in blockchain. And of course, I mean, there is very deep connection between blockchain and security. A big part of blockchain research and technology is actually security. So the two huge overlaps.

So one of the things that OASIS Labs is hoping to do is to empower more AI applications as well. Can you say a bit about that?

That's one of the key motivations that how we started the OASIS technology as well. I see in AI there are several challenges besides just building up the AI machine learning algorithms itself. So one problem, as we talked a little bit earlier, is the problem of data. So today, in order to train these powerful AI deep learning Systems we need a lot of data, but a lot of this data is very sensitive as a consequence. There are two main issues when for example startups and even companies try to build their machine learning or deep learning systems is one is how to get data, where to get the data and two, then how do you ensure the security and privacy protection as you train the machine learning models and build machine learning systems. The OASIS technology actually one of the key motivation is to actually help address these problems. So at Oasis we are building what we call a privacy first cloud computing platform on blockchain. And on top of that one of the key primitive that we are enabling is what we call privacy preserving smart contracts. One example application that we are building with actually application developers and in collaboration actually with doctors from Stanford and other institutions is application called cara. So with this KERA application, users can upload their medical data on the OASIS blockchain platform and the data will be encrypted and a medical researcher can write a privacy preserving smart contracts. Inside the smart contracts it can contain code for training machine learning models and for serving machine learning models. And the smart contract will specify terms of use. For example, it will specify that if a user contributes data to the smart contract, the data will only be used to train a privacy preserving machine learning model inside the smart contract, it will not be used for anything else. Also it can specify how the user may get compensated. And then if a user agrees to the terms of use of the smart contract, then the user can contribute its material data to the smart contracts. And as the smart contract collects enough of the data then it can start training machine learning models. And then the medical researcher can then in the end evaluate the trained machining model. So in this way it can help significantly reduce the friction for medical researchers and pharmaceutical companies to get access to users data to help them train machine learning models and to help find better cures for diseases. We can also really help with truly decentralization of AI in the sense that today we are still seeing that the AI capability is highly concentrated in a small number of big players like Google, Facebook and so on, because they have the data and they have the compute and they have talent. But we hope that with a blockchain platform like this, that we can really again help more companies, more researchers to be able to utilize data from broad range of data sources while at the same time ensuring the privacy protection of users data. And in this way we can significantly increase the accessibility of machine learning capabilities.

You're someone that's constantly looking to new areas and break into New areas. So your advice for the people watching this, if someone's looking for something exciting to work on, what areas would you suggest they look at?

So we talked about AI and security. I think that's a really important and exciting area to work in. And besides that, I think so my personal favorite area is program synthesis, teaching computers to write code. I think also that's an ultimate test and also ultimate goal that as we build intelligent machines, we want the machines to be able to write code.

And you want a test paper, what, at ICLR for this as well.

We showed that how we can learn recursive neural programs to make it essentially achieve much better generalization and also even to have proof of generalization. I think also for synthesis, I think it's a really useful lens through which for us that we can look at the world. So for many, many different problems, we can actually take a synthesis view. So with program synthesis, we synthesize code, and also we are also exploring how we can do automatic theorem proving essentially to improve synthesis. And even for robots, essentially it's a program synthesis that we are synthesizing action plans for our programs and so on. So I think that's why studying and researching program synthesis is really, really useful, both in terms of that if we can do that, it really helps us solve a really, really important problem. And also just studying the approaches for synthesis, it's a very useful perspective for us to solve a much broader range of problems.

What final words of advice do you have?

So again, I think I'm an example for someone who really enjoys entering new fields. And I think oftentimes when people enter new fields, they are afraid. There's a lot of new things to learn. And also someone typically people may have already been established in the existing fields. And when you enter a new field, you really have to start from scratch, from everything, from the basics and so on. That can be a lot of hard work. But I think, and my number one advice is don't be afraid, be brave. And I think it can be a very, very rewarding process.

That's great. Yeah, be brave and it can be very rewarding.

Yes.

Thank you. That was great. Thanks a lot, dawn, for doing this with us.

Thank you. Thanks a lot for having me. This is great.