Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU)

Sound is good. Okay, great. So I wanted to talk to you about unsupervised learning. And that's the area where there's been a lot of research, but compared to supervised learning that you've heard about today, like convolutional networks, you know, unsupervised learning is not there yet. All right, so I'm going to show you lots of, lots of areas. Parts of the talk are going to be a little bit more mathematical. I apologize for that. But I'll try to give you a gist of the foundations, the math behind these models, as well as try to highlight some of the application areas. What's the motivation? Well, the motivation is that the space of data that we have today is just growing. If you look at the space of images, speech, if you look at social network data, if you look at scientific data, I would argue that most of the data that we see today is unlabeled. So how can we develop statistical models, models that can discover interesting kind of structure in unsupervised way or semi supervised way? And that's what I'm interested in as well as how can we sort of apply these models across multiple different domains? And one particular framework of doing that is the framework of deep learning where you're trying to learn hierarchical representations of data. And again, as I go through the talk, I'm going to show you some, some examples I've tried. So here's one example. You can take a simple bag of words, representation of an article or a newspaper. You can use something that's called an autoencoder, just multiple levels. You extract some latent code and then you get some representation out of it. This is done completely in unsupervised way. You don't provide any labels. And if you look at the kind of structure that the model is discovering, it could be useful for visualization, for example, to see what kind of structure you see in your data. This was done on the Reuters data set. I've tried to kind of cluster together lots of different unsupervised learning techniques. And I'll touch on some of them. It's a little bit, it's not a full set, but the way that I typically think about these models is that there's a class of what I would call non probabilistic models, models like sparse coding, autoencoders, clustering based methods. And these are all very, very powerful techniques. And I'll cover some of them in that talk as well. And then there is sort of a space of probabilistic models and within probabilistic models you have tractable models, things like fully observed belief networks. There's a beautiful class of models called neural autoregressive density estimators. More recently we've seen some successes of so called pixel recurrent neural network models. And I'll show you some examples of that. There is a class of so called intractable models where you are looking at models like Boltzmann machines and models like variational Autoencoder, something that's been quite. There's been a lot of development in our community and deep learning community in that space. I Helmholtz machines. I'll tell you a little bit about what these models are and a whole bunch of others as well. One particular structure within these models is that when you're building these generative models of data, you typically have to specify what the distributions you're looking at. So you have to specify what the probability of the data and generally doing some kind of approximate maximum likelihood estimation. And then more recently we've seen some very exciting models coming out. These are generative adversarial networks, moment matching networks. And this is sort of a slightly different class of models where you don't really have to specify what the density is. You just need to be able to sample from those models. And I'm going to show you some examples of that. So my talk is going to be sort of structured. I'd like to introduce you to the basic building blocks models like sports sparse coding models, because I think that these are very important classes of models, particularly for folks who are working in industry and looking for simpler models. Autoencoder is a beautiful class of models. And then the second part of the talk, I'll focus more on generative models. I'll give you an introduction into restricted bolt machines and deep bolt machines. These are sort of models, statistical models that can model complicated data. And I'll spend some time showing you some examples, some recent developments in our community. Specifically in the case of variational autoencoders, which is I view them as a subclass of Helmholtz machines. And I'll finish off by giving you an intuition about a slightly different class of models which would be these generative adversarial networks. Okay, so let's jump into the first part, but before I do that, let me just sort of give you a little bit of motivation. I know Andre's done a great job and Richard sort of alluded to that as well. But the idea is if I'm trying to classify a particular image and If I say, well, if I'm looking at specific pixel representation, it might be difficult for me to classify what I'm seeing. On the other hand, if I can find the right representations, the right representations for these images, and then I sort of get the right features, so get the right structure from the data, that it might be easier for me to see what's going on with my data. So how do I find these representations? And this is sort of one of traditional approaches that we've seen for a long time is that you have a data, you're creating some features and then you're running your learning algorithm. And for the longest time in object recognition or in audio classification, you typically use some kind of hand design features and then you start classifying what you have. Like Andrew was saying, in the space of vision, there's been a lot of different features, designs of what's the right structure we should see in the data. In the space of audio, same thing is happening. How can you find these right representations for your data? And the idea behind representation learning in particular in deep learning, can we actually learn these representations automatically? And more importantly, can we actually learn these representations in unsupervised way by just seeing lots and lots of unlabeled data, can we achieve that? And there's been a lot of work done in that space, but we're not there yet. So I wanted to lower your expectations as I show you some of the results. Okay. Sparse coding. This is one of the models that I think that everybody should know what it is. It was actually first has its roots in 96 and it was originally developed to explain early visual processing in the brain. I think of it as an edge detector. And the objective here is the following. Well, if I give you set of data points x1 up to xn, you'd want to learn a dictionary of bases Phi1 up to Phi K, right? So that every single data point can be written as a linear combination of the basis. That's fairly simple, right? There is one constraint in that you'd want your coefficients to be sparse, you'd want them to be mostly zero, right? So every data point is represented as a sparsely new combination of bases, right? So this is if you apply sparse coding to natural images, and this was originally when a lot of work developed at Stanford with Andrew Ng's group. So if you apply sparse coding to take little patches of images and learn these bases, these dictionaries, this is how they look like and they look really nice in terms of finding edge like structure. So if given a New example, I can say, well, this new example can be written as a linear combination of a few of these bases, right? And taking that representation, it turns out that particular representation, a sparse representation, is quite useful as a feature representation of your data, right? So it's quite useful to have it. And in general, oops, how do we fit these models? Well, if I give you a whole bunch of image patches, but these don't necessarily have to be image patches. This could be little speech signals or any kind of data you're working with, you'd want to learn a dictionary of basis. You have to form, you have to solve this optimization problem. So the first term here, you can think of it as a reconstruction error, which is to say, well, I take a linear combination of my basis, I want them to match my data. And then there's a second term which is, you can think of it as a sparse penalty term which essentially says try to penalize my coefficients so that most of them are zero. That way every single data point can be written as just a linear combination, sparse linear combination of the basis. And it turns out there is an easy optimization for doing that. If you fix your Dictionary of basis Phi 1 up to 5k and you solve for the activations, that becomes a standard lasso problem. There's a lot of solvers for solving that particular problem. That's a general very. It's a lasso problem which is fairly easy to optimize. And then if you fix the activations and you optimize for dictionary basis, then it's a well known quadratic programming problem. Each problem is convex, so you can sort of alternate between finding coefficients, finding basis and so forth so you can optimize this function. There's been a lot of recent work in the last 10 years of doing these things online and doing it more efficiently and so forth. At test time, given a new input or a new image patch and given a set of learned basis, once you have your dictionary, you can then just solve a lasso problem to find the right coefficients. So in this case, given a test sample or test batch, you can find, well, it's written as a linear combination of subset of the basis. And it turns out again that that particular representation is very useful, particularly if you're interested in classifying what you see in images. This is done in completely unsupervised way. There is no class labels, there is no specific supervisory signal. That's here. Back in 2006, there was work done again at Stanford that basically showed a very interesting result. So if I give you an input like this. And these are my learned bases, Remember these little edges, what happens is that you just convolve these bases. You can get these different feature maps, much like the feature maps that we've seen in convolutional neural networks. And then you take these feature maps and you can just do classification. And this was done on one of the older data sets, the Caltech 101, which is a data set that predates Imagenet. And if you look at some of the competing algorithms, if you do a simple logistic regression versus if you do PCA and then do logistic regression versus finding these features using sparse coding, you can get substantial improvements. So again, you see sparse coding popping up in a lot of different areas, not just in deep learning, but folks who are using, looking at the medical imaging domain in neuroscience, these are very popular models because they're easy, they're easy to fit, they're easy to deal with. So what's the interpretation of the sparse coding? Well, let's look at this equation again and we can think of sparse coding as finding an over complete representation of your data. Now the encoding function, we can think of as an encoding function which is, well, I give you an input, find me the features or sparse coefficients or basis that make up my image. We can think of encoding as an implicit and very nonlinear function of X, right? But it's an implicit function, we don't really specify it. And the decoder or the reconstruction is just a simple linear function and it's very explicit. Just take your coefficients and then multiply it by the find the right basis and get back the image or the data. And that sort of flows naturally into the ideas of autoencoders. The autoencoder is a general framework where if I give you an input data, let's say it's an input image, you encode it, you get some representation, some feature representation, and then you have a decoder. Given that representation, you're decoding it back into the image. So you can think of encoder as a feed forward, bottom up pass, much like in a convolutional neural network. Given the image you're doing a forward pass and then there is also feedback and generative or top down pass, right? Given features, you're reconstructing back the input image and the details of what's going inside the encoder decoder, they matter a lot. And obviously you need some form of constraints, you need some of constraints to avoid learning an identity. Because if you don't put these constraints what you could do is just take your input, copy it to your features and then reconstruct back. And that would be a trivial solution. So we need to introduce some additional constraints if you're dealing with binary features, if you want to extract binary features, for example, I'm going to show you later why you'd want to do that. You can pass your encoder through sigmoid nonlinearity, much like in the neural network. And then you have a linear decoder that reconstruct back the input. The way we optimize these little building blocks, or these little blocks is we can just have an encoder which takes your input, takes a linear combination, passes it through some non linearity, the sigmoid nonlinearity. It could be rectified linear units or could be 10h nonlinearity. And then there is a decoder where you reconstruct back your original input. So this is nothing more than a neural network with one hidden layer. And typically that hidden layer would have a small dimensionality than the input. So we can think of it as a bottleneck layer. And we can determine the network parameters, the parameters of the encoder and the parameters of the decoder by writing down the reconstruction error. And that's what the reconstruction error would look like given the input encode, decode and make sure whatever you're decoding is as close as possible to the original input. Then we can use back propagation algorithm to train. There is an interesting sort of relationship between autoencoders and principal component analysis. Many of you have probably heard about PCA as a practitioner, if you're dealing with large data and you want to see what's going on, PCA is the first thing to use, much like logistic regression. And the idea here is that if the parameters of encoder and decoder are shared and you actually have the hidden layer, which is a linear layer, so you don't introduce any nonlinearities. Then it turns out that the space, the latent space that the model will discover is going to be the same space as the space discovered by pca. It effectively will collapse the principal component analysis, right? Doing pca, which is sort of a nice connection because it basically says that autoencoders, you can think of them as nonlinear extensions of pca. So you can learn a little richer features if you are using autoencoders. So here's another model. If you're dealing with binary input, sometimes we're dealing with like Mnist, for example. Again, your encoder and decoder could use sigmoid nonlinearity. So, so given an input, you extract some binary features, binary features you reconstruct back the binary input. And that's actually relates to a model called the restricted Boltz machine, something that I'm going to tell you about later in the talk. There's also other classes of models where you can say, well I can also introduce some sparsity, much like in sparse coding to say that I need to constrain my latent features or my latent spot space to be sparse. And that actually allows you to learn quite reasonable features, nice features. Here's one particular model called predictive sparse decomposition where you effectively, if you look at the first part of the equation here, the decoder part, that pretty much looks like a sparse coding model. But in addition you have an encoding part that essentially says train an encoder such that it actually approximates what my latent code should be. So effectively you can think of this model as there is encoder, there is a decoder, but then you put the sparsity constraint on your latent representation and you can optimize for that model. And obviously the other thing that we've been doing in the last seven, eight and ten years is well, what you can do is you can actually stack these things together so you can learn low level features, try to learn high level features and so forth. So you're just building these blocks and perhaps at the top level if you're trying to solve a classification problem, you can do that. This is sometimes known as a greedy layer wise learning. And this is sometimes useful whenever you have lots and lots of unlabeled data and when you have a little labeled data. Right, a small sample of labeled data. Typically these models help you find meaningful representations such that you don't need a lot of labeled data to solve a particular task that you're trying to solve. And this is again you can remove the decoding part and then you end up with a standard or convolutional architecture. Again, your encoder and decoder could be convolutional and it depends on what problem you're tackling. And typically you can stack these things together and optimize for a particular task that you're trying to solve. Okay, here's an example of. Just wanted to show you some examples, some early examples Back in 2006 this was a way of trying to build these non linear autoencoders and you can sort of pre train these models using restricted bolt machines or autoencoders generally. And then you can stitch them together into this deep Autoencoder and back propagate through reconstruction loss. One thing I want to point out is that here's one particular example. The top row I show you real faces. The second row you're seeing faces reconstructed from a bottleneck of 30 dimensional real value bottlenecks. You can think of it as just a compression mechanism. Given the data, high dimensional data, you're compressing it down to 30 dimensional code and then from that 30 dimensional code you're reconstructing back the original data. So if you look at the first row, this is the data, the second row shows you reconstructed data and the last row shows you PCA solution. One thing I want to point out is that the solution here you have a much sharp representation, which means that it's capturing a little bit more structure in the data. It's also kind of interesting to see that sometimes these models tend to, how should I say it? They tend to regularize your data, right? Like for example, if you see this person with glasses removes the glasses and that generally has to do with the fact that there is only one person with glasses. So the model just basically says that's noise, get rid of it. Or it sort of gets rid of mustaches. Right. Like if you see there's no mustache. Right. And then again that has to do with the fact that there's enough capacity. So the model might think that that's just a noise. And if you're dealing with text type of data, this was done using a Reuters data set. You have about 800,000 stories. You take back and forth representation, something very simple, you can press it down to two dimensional space and then you see what that space looks like. And I always like to joke that the model basically discovers that European Community economic policies are just next to disasters and accidents. Right. This is done. This was back in, I think the data was collected in 96. Right. I think today it's probably going to become closer though those two things. But again, this is just a way. Typically autoencode is a way of compression or trying to do dimensionality reduction. But we'll see later that they don't have to be. Okay. There's another class of algorithm called semantic hashing, which is to say, well, what if you take your data and compress it down to binary representation? Wouldn't that be nice? Because if you have binary representation, you can search in the binary space very efficiently. Right. In fact, if you can compress your data down to 20 dimension, 20 dimensional binary code, 2 to the 20 is about 4 gigabytes. So you can Just store everything in memory and you can look at the, you know, just do memory lookups without actually doing any search at all. So this sort of representation sometimes have been used successfully in computer vision where you take your images and then you learn these binary representations, you know, 30 dimensional codes, 200 dimensional codes. And it turns out it's very efficient to search through large volumes of data using binary representation. So you can, you know, it takes a fraction of a millisecond to retrieve images from a set of millions and millions of images. Again, this is also an active area of research right now because people are trying to figure out, we have these large databases, how can you search through them efficiently? And learning a semantic hashing function that maps your data to the binary representation turns out to be quite useful. Now let me step back a little bit and say, let's now look at generative models and let's look at probabilistic models and how different they are. And I'm going to show you some examples of where they're applicable. Here's one example of a simple model trying to learn a distribution of these handwritten characters. So we have Sanskrit, we have Arabic, we have Cyrillic, and now we can build a model that says, well, can you actually generate me what a Sanskrit should look like? The flickering you see at the top, these are neurons, you can think of them as neurons firing. And what you're seeing at the bottom is you're seeing what the model generates, what it believes Sanskrit should look like. In some sense, when you think about generative models, you think about models that can generate, or they can sample the distribution or they can sample the data. This is a fairly simple model. You have about 25,000 characters coming from 50 different alphabets around the world. You have about 2 million parameters. This is one of the older models, but this is what the model believes Sanskrit should look like. And I think that I've asked a couple of people to say that. Does that really look like Sanskrit? Okay, great. Which can mean two things. It can mean that the model is actually generalizing or the model is overfitting, meaning that it's just memorizing what the training data looks like. And I'm just showing you examples from the training data. We'll come back to that point as we go through the talk. Here's. You know, you can also do conditional simulation. You know, given half of the image, can you complete the remaining half? Right. And more recently there's been a lot of advances, especially the last couple of years, for the conditional generations. And it's pretty amazing what you can do in terms of inpainting given half of the image, what the other half of the image should look like. This is sort of a simple example, but it does show you that it's trying to be consistent with what different strokes look like. So why is it so difficult? In the space of so called undirected graphical models of bolster machines, the difficulty really comes from the following fact. If I show you this image, which is a 28 by 28 image, it's a binary image. So some pixels are on, some pixels are off. There are two to the 28 by 28 possible images. So in fact there are two to the 784 possible configurations and that space is exponential. So how can you build models that figure out in the space of characters there's only a little tiny subspace in that space. If you start generating 200 by 200 images, that space is huge. And the space of real images is really, really tiny. So how do you find that space? How do you generalize to new images? That's a very difficult question in general to answer. One class of models is so called fully observed models. There's sort of been a stream of learning generative models that are tractable and they have very nice properties like you can compute the probabilities, you can do maximum likelihood estimation. Here's one example where if I try to model the image, I can write it down as taking the first pixel, modeling the first pixel, then modeling the second pixel, giving the first pixel and just writing it down in terms of product of the conditional probabilities. Each conditional probability can take a very complicated form. It could be a complicated neural network. Sorry. There's been a number of successful models. One of the early models called neural autoregressive density Estimator, actually developed by Hugo Real valued extension of these models. And more recently we start seeing these flavors of models. There were a couple of papers popped up actually this year from DeepMind where they sort of make these conditionals to be sophisticated RNNs, LSTMs or convolutional models. And they can actually generate remarkable images. And so this is just a pixel CNN generating I guess elephants. Yeah, it actually looks pretty, pretty interesting. Right. The drawback of these models is that we yet have to see how good of representations these models are learning so that we can use these representations for other tasks like classifying images or find similar images and such. Now let me jump into a class of models called restricted BOSS machines. So this is the class of models where we actually trying to learn some latent structure, some latent representation. These models belong to the class of so called graphical models. And graphical models, very powerful framework for representing dependency structure between random variables. This is an example where we have. You can think of this particular model, you have some pixels. These are stochastic binary, so called visible variables. You can think of pixels in your image and you have stochastic binary hidden variables. You can think of them as feature detectors. So detecting certain patterns that you see in the data. Much like sparse coding models, this has a bipartite structure. You can write down the probability, the joint distribution over all of these variables. You sort of have pairwise term, you have unary term, but it's not really important what they look like. The important thing here is that if I look at this conditional probability of the data, given the features, I can actually write down explicitly what it looks like. What does that mean? That basically means that if you tell me what features you see in the image, I can generate the data for you or can generate the corresponding input in terms of learning features. So what do these models learn? They learn something similar that we've seen in sparse coding. And so these classes of models are very similar to each other. So given a new image, I can say, well, this new image is made up by some combination of these learned weights or these learned bases. And the numbers here are given by the probabilities that each particular edge is present in the data. In terms of how we learn these models, one thing I want to make, another point I should make here is that given an input, I can actually quickly infer what features I'm seeing in the image. So that operation is very easy to do. Unlike in sparse coding models, it's a little bit more closer to an autoencoder. Given the data, I can actually tell you what features are present in my input, which is very important for things like information retrieval or classifying images, because you need to do it fast. How do we learn these models? Let me just give you an intuition, maybe a little bit of math behind how we learn these models. If I give you a set of training examples, and I want to learn model parameters, I can maximize the log likelihood objective, right? And you've probably seen that in these tutorials, the maximum likelihood objective is essentially nothing more than saying, I want to make sure that the probability of observing these images is as high as possible. So finding the parameters of the probability of observing what I'm seeing is high. And that's why you're maximizing the likelihood objective for the log of the likelihood objective, which is, you know, take a product into the sum, you take the derivative. There's a little bit of algebra. I promise you it's not very difficult. It's like second year college algebra. You differentiate and you basically have this learning rule which is the difference between two terms. The first term, you can think of it as looking at sufficient statistics, so called sufficient statistics driven by the data. And the second term is the sufficient statistics driven by the model. Maybe I can parse it out. What does that mean? Intuitively what that means is that you look at the correlations you see in the data and then you look at the correlations that the model is telling you it should be and you're trying to match the two. That's what the learning is trying to do. It's trying to match the correlations that you see in the data. So the model is actually respecting the statistics that you see in the data. Data. But it turns out that the second term is very difficult to compute. And it's precisely because the space of all possible images is so high dimensional that you need to figure out or use some kind of approximate learning algorithms to do that, right? So you have these difference between these two terms. The first term is easy to compute, it turns out, because of a particular structure of the model, and we can actually do it explicitly. The second term is the difficult one to compute, right? So it sort of requires summing over all possible configurations, all possible images that you could possibly see. So this term is intractable. And what a lot of different algorithms are doing, and we'll see that over and over again, is using so called Monte Carlo sampling or market chain Monte Carlo sampling or Monte Carlo estimation. So let me give you an intention what, what this term is doing, and that's a general trick for approximating exponential sums. There's a whole subfield in statistics that's basically dedicated to how do we approximate exponential sums. In fact, if you could do that, if you could solve that problem, you could solve a lot of problems in machine learning. And the idea is very simple actually. The idea is to say, well, you're going to be replacing the average by sampling. And there's something that's called gibbsampling Markov chain Monte Carlo, which essentially does something very simple. It basically says, well, start with the data, sample the states of the latent variables, sample the data, sample the states of the latent data, sample the data from these conditional distributions. Something that you can compute explicitly. And that's a general trick, much like in sparse coding, we're optimizing for the basis, when we're optimizing for the coefficients here, you're inferring the coefficients, then you infer what the data should look like, and so forth. And then you can just run a Markov chain and sort of approximate this exponential sum. So you start with the data, you sample the states of the hidden variables, you resample the data, and so forth. And the only problem with a lot of these methods is that you need to run them up to infinity to guarantee that you're sort of getting the right thing. And so obviously you will never run them infinite. You don't have time to do that. So there's a very clever algorithm that, a contrastive divergent algorithm that was developed by Hinton back in 2002. And it was very clever. It basically said, well, instead of running this thing up to infinity, run it for one step, right? And so you're just running it for one step. You start with a training vector, you update the hidden units, you update all the visible units again. So that's your reconstruction, much like an autoencoder. You reconstruct your data, you update the hidden units again, and then you just update the model parameters, which is just looking at empirically the statistics between the data and the model. Very similar to what the autoencoder is doing, but slight, slight differences. And implementation is basically takes about like 10 lines of MATLAB code. I suspect it's going to be, you know, two lines in TensorFlow, although I don't think TensorFlow folks implemented Boltzmann machines yet. That would be my request. But you can extend these models to dealing with real value data, right? So whenever you're dealing with images, for example, and it's just a little change to the definition of the model, and your conditional probabilities here are just going to be a bunch of Gaussians. So that basically means that given the features, sample me the space of images and I can sample you, give you real valued images. The structure of the model remains the same. If you train this model on these images, you sort of tend to find edges, something similar again to what you've seen sparse coding and ICA independent component analysis, model autoencoders and such. And again, you can sort of say, well, every single image is made up by some linear combination of these basis functions. You can also extend these models to dealing with count data, right? If you're dealing with documents, in this case, again, a slight change to the model. K here denotes your vocabulary size, and D here denotes number of words that that you're seeing in your document. So it's a bag of words representation. And the conditional here is given by so called softmax distribution, much like what you've seen in the previous classes. When the distribution of possible words and the parameters here W's you can think of them as something similar to as what word2vec embedding would do some combination of these edges in case of documents or web pages, you're saying it's the same thing. It's just made up some linear combination of these learned topics. Every single document is made up by some combination of these topics. You can also look at one step reconstruction. So you can basically say, well, how can I find similarity between the words? So if I show you chocolate cake, I infer the states of hidden units and then I reconstruct back the distribution of possible words. It tells me chocolate cake, cake, chocolate, sweet, dessert, cupcake, food, sugar and so forth. I particularly like the one about the flower high and then there is a Japanese sign. The model sort of generates flour, Japan, Sakura blossom, Tokyo. So it sort of picks up again on low level correlations that you see in your data. You can also apply these kinds of models to collaborative filtering, where every single observed variable you can model can represent a user rating for a particular movie. So every single user would rate a certain subset of movies. And so you can represent it as the state of visible vector. And your hidden states can represent user preferences, what they are. And on the Netflix data set, if you look at the latent space that the model is learning, some of these hidden variables are capturing specific movie genre. So for example, there's actually one hidden union dedicated to Michael Moore's movies. So it's sort of like very strong. I think it's sort of either people like it or hate it. So there are a few hidden units specifically dedicated to that, but it also finds interesting things like action movies and so forth. So it finds that particular structure in the data. So you can model different kinds of modality real value data, you can model count data, multinomials, people like it or hate it. So there are a few hidden units specifically dedicated to that. But it also finds interesting things like action movies and so forth, right? So it finds that particular structure in the data. So you can model different kinds of modality real value data, you can model count data, multinomials, and it's very easy to infer the states of the hidden variables. So that's given just the product of logistic functions and that's very important. And it's Very easy to infer the states of the hidden variables. So that's given just the product of logistic functions, and that's very important in a lot of different applications. Given the input, I can quickly tell you what topics I see in the data, right? One thing that I want to point out, and that's an important point, is a lot of these models can be viewed as product models. Sometimes people call them product of experts. And this is because of the following intuition. If I write down the joint distribution of my hidden ordinance in a lot of different applications, given the input, I can quickly tell you what topics I see in the data. One thing that I want to point out, and that's an important point, is a lot of these models can be viewed as product models. Sometimes people call them product of experts. And this is because of the four observed variables. I can write it down in this sort of log linear form. But if I sum out or integrate out the states of the hidden variables, I have a product of a whole bunch of functions. So what does it mean? What's the intuition here? So let me show you an example. Suppose the model finds these specific topics and suppose I'm going to be telling you that the document contains topic government corruption and mafia, then the word Silvio Berlusconi will have very high probability, right? I guess. Does anybody know? Everybody. The following intuition. If I write down the joint distribution of a mahedin knows who Silvio is, Silvio Berlusconi, he's had like he's in head of the government, he's connected to mafia, he's very corrupt, was corrupt and I guess I should add like a banga banga party. So here, right. Then observed variables, I can write it down in this sort of log linear form, right? But if I sum out or integrate out the states of the hidden variables, I have a product of a whole bunch of functions. So what does it mean? What's the intuition here? Will become completely clear what I'm talking about. But then one point I want to make here is that you can think of these models as a product. Each hidden variable defines a distribution of a possible words over possible topics. And once you take the intersection of these distributions, you can be very precise about what is it that you're modeling. So that's unlike generally topic models or latently allocation models, models where you actually using mixture like approach. And then typically these models do perform far better than traditional mixture based models. And this comes to the point of local versus global versus distributed representations, right? In a lot of different algorithms, even unsupervised learning algorithms. So let me show you an example. Suppose the model finds these specific topics, right? And separate things are just clustering. You typically have some you partitioning the space and you're finding local prototypes and the number of parameters for which you have baseballs. I'm going to be telling you that the document contains topic government corruption and mafia. Then the word sickly parameters for each region. The number of regions typically grow linearly with a number of parameters. But in models like factor models, PCA restricted Boltzmann machines, Silvio Berlusconi will have very high probability, I guess. Does anybody know? Everybody knows who Silvio is. Silvio Berlusconi, he's had like he's in head of the government, he's connected to mafia. Deep models, you typically have distributed representations, right? And what's the idea here? The idea here is that if I show you the two inputs, each particular neuron can differentiate between two parts of the plane. Given the second one, I can partition it again. Given the third hidden variable, you can partition it again. So you can see that every single neuron will be affecting lots of his very corrupt was corrupt. And I guess I should add like a bunga bunga parties here, right? Then it will become completely clear what I'm talking about. But then one point I want to make here is that different regions, and that's the idea behind distributed representations, because every single parameter is affecting many, many regions, not just the local region. And so the number of regions grow roughly exponentially with the number of parameters. So that's the differences between these two classes of models. Important to know about them. Now let me jump and quickly tell you a little bit of inspiration behind what can we build with? You can think of these models as a product. Each hidden variable defines a distribution of a possible words over possible topics. And once you take the intersect these models, as we've seen with convolutional networks, the first layer we typically learn some low level features like edges or if you're working with a word table, typically learn some low level structure. And the hope is that the high level features will start picking up some high level structure as you are building. And these kinds of models can be built in completely unsupervised way. Because what you're trying to do is you're trying to model the data. You're trying trying to model the distribution of the data. You can write down the probability distribution for this model. It's known as a Bolson machine model. You have dependencies between hidden variables. So now introducing Some extra layers and dependencies between those layers of these distributions. You can be very precise about what is it that you're modeling. So that's unlike generally topic models or latently allocation models, models where you actually using mixture like approach. And then typically these models do perform far better than traditional mixture based models. And this comes to the point of local versus global. And if we look at the equation, the first part of the equation is basically the same as what we had with restricted bolt machine. And then the second and third part of the equation essentially modeling dependencies between, you know, the first and the second hidden layer and the second hidden layer and the third hidden layer. Right. There is also a very natural notion of bottom up and top down. So if I want to see what's the probability of a particular unit being taking value one, it's really depend on what's coming from below and what's coming from above. So there has to be some consensus in the model to say ah yes, what I'm seeing in the image and what my model believes the overall structure should be, should be in agreement. And so in this case, of course, in this case hidden variables become dependent even when you condition on the data. So these kinds of models will see a lot versus distributed representations. Right. In a lot of different algorithms. Even unsupervised learning algorithms are just clustering. You typically have some. You're partitioning the space and you're finding local prototypes. And the number of parameters for which you have basically is. You're introducing more flexibility, you're introducing more structure. But then learning becomes much more difficult. You have to deal. How do you do inference in these models? Now let me give you an intuition of how can we learn these models? What's the maximum like parameters for each region? The number of regions typically grow linearly with the number of parameters you could estimate it doing here. Well, if I differentiate this model with respect to parameters, I basically run into the same learning rule. And it's the same learning rule you see whenever you're working with undirected graphical models, factor graphs, conditional random fields, you might have heard about those, those ones. It really is just trying to look at the statistics driven by the data correlations that you see in the data and the correlations that the model is telling you it's seeing in the data. And you're just trying to match the two.

Right?

That's exactly what's happening in that particular equation. Right. But then models like factor models, PCA restricted Boltzmann machines, deep models, you typically have distributed representations and what's the idea here. The idea here is that if I show you the two inputs, each particular neuron can differentiate between two parts of the plane. Given the second one, I can partition it. Again. Given the third, hidden. But the first term is no longer factorial. So you have to do some approximation with these models. But let me give you notation what each term is doing. Suppose I have some data and I get to observe these characters. Well, what I can do is I really want to tell the model this is real, these are real characters. So I want to put some probability mass around them and say these are real. And then there is some sort of data point that looks like this, just a bunch of pixels on and off. And I really want to tell my model that put almost zero probability on this. This is not real. Right? And so the first term is exactly trying to do that. The first term is just trying to say, put the probability mass where you see the data. And the second term is effectively trying to say, well, look at this entire exponential space and variable. You can partition it again. So you can see that every single neuron will be affecting lots of different regions. And that's the idea behind distributed representations, because every single parameter is affecting many, many regions, not just the local region. And so the number of regions grow roughly exponentially with the number of parameters. So that's the differences between these two classes of models. Important to know about them. Say no, everything else is not real. Just the real thing is what I'm seeing in my data. And so you can use sort of advanced techniques for doing that. There's a class of algorithms called variational inference. There's something that's called stochastic approximation, which is Monte Carlo based inference. I'm not going to go into these techniques, but in general you can train these models. So one question is, how good are they? Because a lot of approximations that go into these models. So what I'm going to do is, if you haven't seen it, I'm going to show you two panels. On one panel you will see the real data. On another panel you'll see data simulated by the model or the fake data. And you have to tell me which one is which. So again, these are handwritten characters coming from alphabets around the world. How many of you think this assimilated? And the other part was real? Honestly? Okay, some. What about the other way around? I get half and half, which is great. If you look at these images a little bit more carefully, you will see the difference. So you will see that. Now let me Jump. And quickly tell you a little bit of inspiration behind what can we build with these models? As we've seen with convolutional networks, the first layer we typically learn, this is simulated and this is real. Because if you look at the real data, it's much crisper, there's more diversity when you're simulating the data. There's a lot of structure in the simulated characters, but sometimes they look a little bit fuzzy and there isn't as much diversity. Right. And I have some low level features like edges or you know, if you're working with a word save will typically learn some low level structure. And the hope is that the high level features will start picking. Learn that trick from my neuroscience friends. If I show you quickly enough, you won't see the difference. And you know, if you using these models for classifying, you can do proper analysis, which is to say given a new character, you find further states of the latent variables, hidden variables. If I classify based on that, how good are they? And they're much better than some of the existing techniques. This is another example trying to generate 3D objects. This is sort of a toy data set. And later on I'll show you some bigger advances that's been happening in the last few years. Some high level structure as you are building. And these kinds of models can be built in completely unsupervised way. Because what you're trying to do is you're trying to model the data, you're trying to model the distribution of the data. You can write down the probability distribution for this model. It's known as a Bolson machine model. You have dependencies between hidden variables. So now introducing some extras. This was done a few years ago. If you look at the spirit space of generated samples, they sort of, obviously you can see the difference. Look at this particular image. This image looks like car with wings, don't you think? So there is. Sometimes it can sort of simulate things that are not necessarily realistic. And for some reason it just doesn't generate donkeys and elephants too often, but it generates people with guns more often. Like if you look at here and here and here, and that again has to do with the fact that you're exploring this exponential space of possible images. And sometimes it's very hard to assign the right probability through some extra layers and dependencies between those layers. And if we look at the equation, the first part of the equation is basically the same as what we had with the different parts of the spectrum space. And then obviously you can do things like pattern completion. So given half of the image, can you complete the remaining half? So the second one shows what the completions look like in the last machine and then the second and third part of the equation essentially modeling dependencies between the first and the second hidden layer and the second hidden layer and the third hidden layer there is also very natural, no one is what the truth is. So you can do these things. So where else can we use these models? These are sort of toyish examples, but where else? Let me show you one example where these models can potentially succeed, which is trying to model the space of the multimodal space, which is the space of images and text. Or generally if you look at the data, it's not just single source, it's a collection of different modalities. So how can we take all of these modalities into account? And this is really just the idea of given images and text. Can you actually find a concept that relates these two definition of bottom up and top down. So if I want to see what's the probability of a particular unit being taking value one, it's really depend on what's coming from below and what's coming from above. So there has to be some consensus in the model to say ah yes, what I'm seeing in the image and what my model believes, the overall structuring sources of sources of data. And there are a few challenges and that's why you should be, should be in agreement. Right. And so in this case, of course, in this case hidden there, you know, models like generative models, sometimes probabilistic models could be useful. General, one of the biggest challenges we've seen is that typically when you're working with images and text, these are very different modalities. Right. If you think about images and pixel representation, they're very dense. If you're looking at text, it's typically very sparse. So it's very difficult to learn these cross model features from low level representation. Perhaps a bigger challenge is that a lot of times we see data that's very noisy. Sometimes it's just non existent. Given an image there is no text. Or if you look at the first image, a lot of the tags about is what kind of camera was used to describe that particular image, which levels become dependent even when you condition on the data. So these kinds of models we'll see a lot is you're introducing more flexibility, you're introducing more structure, but then learning becomes doesn't really tell us anything about the image itself. And these would be the text generated by a version of a Boltzmann machine model sort of does samples what the tag should look like. Much more difficult you have to deal with. How do you do inference in these models? Now let me give you an intuition of how can we learn these models? What's the maximum like? The idea again is very simple. If you just build a simple representation given images and given text, you just try to find what the common representation is. It's very difficult to learn these cross model features. But if you actually built a hierarchical model, so you start with representation, you can build a Gaussian model, replicate a Softmax model, you can sort of build up that representation, then it turns out it's much more, it gives you much richer representation. There's also a notion of bottom up and top down, which means that low level images or estimator doing here? Well, if I differentiate this model with respect to parameters, I basically run into the same learning rule. And it's the same learning rule you see whenever you're working with undirected graphical models, factograph tags can effectively affect low level representation of images and the other way around. So information flows between images and text and sort of gets into some stable state. And this is what you know the text conditional random fields, you might have heard about those, those ones. It really is just trying to look at the statistics driven by the data correlations that you see in the data and the correlations that the model is telling you it's seeing in the data index generated from images. Looks like some of the examples. A lot of them look reasonable. But more recently with the advances of convnets, this is probably not that surprising. Here's some examples of the model that's not quite doing the right thing. I particularly like the second one. For some reason it sort of correlates with Barack Obama and such and we've sort of the features when we were using this model we didn't have at that time sort of imagenet features. Right now I don't think we'd be making these mistakes. But generally speaking, you know what we found in a lot of the data is that there aren't a lot of images of animals. Which brings us to the next problem is that if you don't see images of animals then you're just trying to match the two. That's exactly what's happening in that particular equation. But the first term is no longer factorial. So you have to do some approximation with these models. But let me give you an intuition what each term is doing. Suppose I have some data, right? And I get to observe these Characters. The model is confused because it sees a lot of Obama signs. And these are black and sort of white and blue sort of signs that are appearing a lot. You can also do images from text, given text or tags can retrieve this. Well, what I can do is I really want to tell the model this is real, right? Relevant images. So this is the data set itself at about a million images. It's a nice data set and you have very noisy tags. And the question is, can you actually learn some representation from those images? One thing that I want to highlight here is we've tried. There's a 25,000 labeled image. Somebody went and labeled what's going on in those images, what classes we see in those images. And you get some numbers, which is mean average precision. But what's important here is that we found that if we actually use unlabeled data and we pre train these channels separately using a million unlabeled data points, then we can actually get some performance improvements. So at least that was a little bit of happy sign for us to say that unlabeled data can help in the situations where these are real characters. So I want to put some probability mass around them and say these are real. And then there is some sort of data point that looks like this, just a bunch of pixels on and off. And I really want to tell my model that, you know, put almost zero probability on this. This is not real, don't have a lot of labeled examples. So here it was helping us, it was helping us a lot. And then. Right. And so the first term is exactly trying to do that. The first term is just trying to say put the probability. Once you get into this sort of representations dealing with text and images, this is one particular thing you can do. And I think Richard pointed out what happens in the space of linguistic where you see the data. And the second term is effectively trying to say, well, look at this entire exponential space and just say, no, everything else is not real. Just the real thing is what I'm seeing in my data. And so you can use sort of advanced techniques for doing that. There's a class of algorithms called variational inference, something that's called stochastic approximation, which is Monte Carlo based inference. I'm not going to go into these techniques, but in general you can train these models. One question is how good are they? Because a lot of approximation regularities, you can do the same thing with images. Just fun to do. They sometimes work, they don't work all the time. But here's one example. If I take that particular image at the top. And I say get the representation of this image, subtract the representation of day, add the night, and then find closes images. You get these images that go into these models. So what I'm going to do is if you haven't seen it, I'm going to show this and then you can do some interesting things like take these kittens and say minus ball plus box, you get kittens in the box. Or if you take this particular image and say minus box plus ball, you get kittens in the ball. Except for this thing that's a duck. So you can sort of get these interesting representations. Of course, these are all sort of fun things to look at, but they don't really mean much because we're not specifically optimizing for those things. Now let me spend some time also talking about another class of models. These are known as Helmholtz machines and variational automation encoders. These are the models that have been sort of popping up in our community. Two panels. On one panel you will see the real data. On another panel you'll see data simulated by the model or the fake data. And you have to tell me which one is which. Okay, so again, in the last two years, right, so what is a Helmholtz machine? A Helmholtz machine was developed back in 95 and it was developed by him and Peter Dayan and Brandon Frey and Radford Neal. And it has this particular architecture. You have a generative process given some latent state. It's a neural network, it's a stochastic neural network that generates the input data. And then you have so called approximate inference step, which is to say given the data, infer approximately what the latent states should look like. And again, it was developed in 95. There's something that's called ways of handwritten characters coming from alphabets around the world. How many of you think this is simulated in the other sleep algorithm and it never worked? Basically people just said it just doesn't work. Part was real, honestly. Okay, some. What about the other way around? I get half and half, which is great. If you look at these images a little bit more carefully, you will see the difference, right? So you will see that. And you know, then we started looking at restricted bolt machines and bolt machines because they're working a little bit better. And then two years ago people figured out how to make them work. And so now 10 years later, I'm going to show you the trick. Now these models are actually working pretty well. The difference between Helmholtz machines And deep Boltz machines is very subtle. They almost look identical. The big difference between the two is that in Helmholtz machines you have a generative process that generates the data and you have a separate recognition model that tries to recognize what you're seeing in the data. So you can think of this Q function as a convolutional neural network, given the data, tries to figure out what the feature should look like. And then there is a generative model, given the features, it generates the data. Boltzmann machine is sort of similar class of models, but it has undirected connections. So you can think of it as this is simulated and this is real. Because if you look at the real data, it's much generative and recognition connections are the same. So it's sort of a system that tries which crispr there is more diversity when you're simulating the data. There's a lot of structure in the simulated characters, but sometimes they look a little bit and there isn't as much diversity. And I've learned that trick from my neuroscience friends. If I show you quickly enough, you won't see the difference. An equilibrium state when you're running it. So it's a little bit. The semantics is a little bit different between these two models. So what is the variational autoencoder? Variational encoder is a Helmholtz machine. It defines a generative process in terms of sampling through cascades of stochastic layers. And if you look at it, there's just a bunch of conditional probability distributions that you're defining so you can generate the data. So theta here will denote the parameters of the variational autoencoders. You have a number of stochastic layers. And sampling from these conditional probability distributions, we're assuming that we can do it. It's a tractable. It has to be tractable. But the innovation here is that every single conditional probability can actually be a very complicated function. It can denote a nonlinear. You can model nonlinear relationships. It can be a multi layer nonlinear neural network, deterministic neural network. So it becomes fairly powerful. Here's an example of I have a stochastic layer. You have a deterministic layer, have a stochastic layer and then you generate the data. And if you use using these models, you can introduce these nonlinearities into these models. And this can for classifying, you can do proper analysis, which is to say given a new character, you find further states of the latent variables, hidden variables. If I classify based on that, how good are they? And they're much Better than some of the existing techniques. This is another example trying to generate 3D of conditional probability with denoting one layer neural net. Right. Now I'll show you some examples, but maybe I can just give you a little intuition behind what these equations do. In a lot of these kinds of models, learning is very hard to do. And there is a class of models called variational learning. And what the variational learning is trying to do is basically trying to do the following. Well, I want to maximize the probability of the data that I observe, but I cannot do it directly. So instead what I'm going to do is I'm going to maximize the so called variational low bound, which is this term here. And it's effectively saying, well, if I take the log of expectation, I can take the log and push it inside. And it turns out just logistically working in this representation is objects. This is sort of a toy dataset. And later on I'll show you some bigger advances that's been happening in the last few years. Much easier than working in this, this representation. If you go a little bit through the math, turns out that you can actually optimize this variational bound, but you can't really optimize this. This was done a few years ago. If you look at the space of generated samples, they sort of, obviously you can see the difference. Here's look at this particular image. This image looks like car with wings, don't you think? Particular likelihood objective. It's a little bit surprising for those of you who haven't seen variational learning, how it's done. But this one little trick, this one little so called Jensen's inequality, actually allows you to solve a lot of problems. The other way to write the lower bound is to say, well, there is a log likelihood function and something that's called KL divergence, which is the distance between your approximating distribution Q, which is your recognition model and the truth, the truth in these models would be the true posterior according to your model. And it's hard to optimize these kinds of models in general. You're trying to optimize your generative model, you're trying to optimize your recognition model. Back in 95, Hinton and his students, basically they developed this wake slip algorithm that was a bunch of different. So there's sometimes it can sort of simulate things that are not necessarily realistic. And for some reason it just doesn't generate donkeys and elephants too often. Right? But it generates people with guns more often. Like if you look at here and here and here and that again has to do with the fact that you're exploring this exponential space of possible images. And sometimes it's very hard to assign the right probabilities and things put together. But it was never quite the right algorithm because it wasn't really optimizing anything. It was just a bunch of things alternating. But in 2014 there was a beautiful trick introduced by King Mainwelling and there was a few other groups that came up with the same trick called repanetarization trick. So let me show you what repanetization trick does intuitively. So let's say your recognition distribution is a Gaussian. So a Gaussian, I can write it as a mean and a variance. So this is the different parts of the space.

Right?

And then obviously you can do things like pattern completion. So given half of the image, can you complete the remaining half? So the second one shows what the completions look like. And the last, the mean. This is the variance. Notice that my mean depends on the layer below. Could be one is what the truth is. So you can do, you can do these things. So where else can we use these models? These are sort of a very nonlinear function. The variance also depends on the layer below, so it could also be a nonlinear function. Toyish examples, but where else? Let me show you one example where these models can potentially. But what I can do is I can actually do the following. I can express this particular Gaussian in terms of auxiliary variables. So I can say, well, if I sample this epsilon from normal 0, 1, a Gaussian distribution, then I can write this particular H, right? My style state in a deterministic way. It's just mean plus essentially standard deviation or variance, square root of the variance times this epsilon. So this is just a simple parameterization of the Gaussians, right? I'm just pulling out the mean and the variance exceed, which is trying to model the space of the multimodal space, which is the space of no sort of surprises here. So I can write my recognition model as this Gaussian or I can write it in terms of noise plus the deterministic part, right? So the recognition distribute images and text or you know, generally if you look at the data, it's not just single sources. A collection can be represented as a deterministic mapping. And that's, that's the beauty because it turns out that you can collection of different modalities, right? So how can we take all of these modalities into account, lapse these complicated models effectively into autoencoders and we know how to deal with autoencoders. And this is really just the idea of given images and text. Can you actually find a concept that relates these two different back propagate through the entire model? So we have a deterministic encoder and then the distribution of these auxiliary variables really don't depend on parameters. So sort of it's almost like taking a stochastic system and separating the stochastic part and deterministic part. In deterministic part you can do backpropagation, so you can do learning. And the stochastic part you can do sampling. So just think of it as a separation between the two pieces. So now if I take the gradient of the variational bound or variational objective with respect to parameters, this is something that we couldn't do back in 95 and we couldn't do it. In the last 10 years people have tried using reinforce algorithm or some of the sources of data and there are a few challenges and that's why approximations to it, it never worked. But here what we can do is we can do the following. We can say, well, I can write models like generative models. Sometimes probabilistic models could be useful. In general, one of the biggest challenges we've seen is that typically when you're working with images and text, these are very different modalities. If you think about images and pixel representation, they're very dense. If you're looking at text, it's typically this expression because it's a Gaussian is sampling bunch of these auxiliary variables and then this log, I can just inject the noise in here. The whole thing here becomes deterministic and that's where the beauty comes in. You take this gradient here and you push it inside the expectation. So before, if you take the gradient of expectations, like taking the gradient of averages, you compute a bunch of averages and you're taking the gradient. What you're doing now with randomization trick is you're taking the gradients and then taking the average. It turns out that hugely reduces the variance in your training and actually allows you to learn these models quite efficiently. So the mapping edge here is completely deterministic and gradients here can be computed by back propagation. It's very sparse, right? So it's very difficult to learn these cross model features from low level representation. Perhaps a bigger challenge is that a lot of times we see data that's very noisy, right? Sometimes it's just non existent. Given an image there is no text or if you look at the first image, a lot of the Tags about is what kind of camera was used to describe that particular image with deterministic system. And you can think of this thing inside is just an autoencoder that you are optimizing. And obviously there are other extensions of these models that we've looked at and a bunch of other teams looked at where you can say, well, maybe we can improve these models by drawing multiple samples. These are so called case samples, importance weighting bounds. And so if you can make them a little bit better, a little bit more precise, you can model a little bit more complicated distributions over the posteriors. But now let me sort of step back a little bit and say, why am I telling you about this? What's the point? There are a bunch of equations. You're injecting noise. Why do we need noise? Why do we need stochastic systems in general? Here's a motivating example. We wanted to build a model that given capture, doesn't really tell us anything about the image itself. And these would be the text generated by a version of a Boltzmann machine model. It sort of does samples what the tag should look like. We want to generate the image. And my student was very ambitious and basically said, I want to be able. And the idea again is very simple. If you just build a simple representation, given images and given text, you just try to find what the common representation is. It's very difficult to learn these, to just give you any sentence. And I want to be able to generate image, kind of like an artificial paint. I want to paint what. What's in my caption in the most general way. So this is one example of a Helmholtz machine where you have a generative model, which is a stochastic recurrent network. It's just a chain sequence of variational autoencoders. And there's a recognition model which is. You can think of it as deterministic system, like a convolutional system that tries to approximate what the latent states are. But why do I need, why do I need stochasticity here? Why do I need variational encoders here? And the reason is very simple. Suppose I give you the following task. I say a stop sign is flying in blue sky, cross model features. But if you actually built a hierarchical model, so you start with representation. Now if you were using a deterministic system like an autoencoder, you can build a Gaussian model, replicate a Softmax model, you can sort of build up that representation. Then it turns out it's much more. It gives you much richer representation. It's also generate one image because it's a deterministic system. Give an input, I give you output. Once you have stochastic system, you inject this noise, this latent noise that allows you to actually generate a whole space of possible images. So for example, it tends to generate this top sign and this top sign. They look very different bottom up and top down. Which means that, you know, low level images. And there's a car here, so maybe it's not really flying. It's just can't draw the pole here. This one looks like they're clouds. Here's this yellow school bus is flying in blue skies, right? So here we wanted to test the system to see does it understand something about what's in the sentence. Here's a herd of elephants is flying in blue skies. Now we cannot generate elephants or although there are now tags can effectively affect low level representation of images. And the other way around. So information flows between inks that are getting better, but sometimes it generates two of them. Right. And it's a commercial plane flying in blue skies. But this is where we need stochasticity because we want to be able to generate the whole distribution of possible outcomes, not necessarily just one particular point. We can basically do things like a yellow school bus parked in the parking lot versus a red school bus parked in the parking lot. Images in text and sort of gets into some stable state. And this is what the text generated from images looks like. Some of the examples, a lot of them look reasonable. But more recently with the advances of convnets this is probably not versus a green school bus parked in the parking lots. It's sort of in a blue school bus, right? So it's sort of. We can't quite generate blue school buses, but we've seen blue cars and we've seen blue buses. So it can sort of make an association to draw these different things. They look a little bit fuzzy. But in terms of comparing to different models, if I give you a group of people on the beach with surfboards, this is what we can generate. There is another model called labgap model which is a model based on adversarial neural network. Something I'll talk as the last part of this talk. And there is these models convolutional deconvolutional variational autoencoders which is again convolutional deconvolutional autoencoders. Just with some noise. And you can certainly see that generally we found it's very hard to be able to generate that Surprising. Here's some examples of the model that's not quite doing the right thing. I particularly like the second one for some reason it's sort of correlates with Barack Obama and such and we've sort of the features when we were using this model we didn't have at that time sort of ImageNet features. Right now I don't think we'd be making these mistakes but generally speaking, you know what we found in a lot of the Danes with arbitrary inputs as a text, right? Here's my favorite one. A toilet seat sits open in the bathroom. I don't know if you can see toilet seats here maybe, but you can say toilets. It sits open in the grass field. That was a little bit better. At least the colors were quite right. And when we put this paper on Arxiv, one of the students basically came to me and said well this is really bad because you can always ask Google and if you type that particular query into Google search it gives you that. Which was a little bit disappointing. But now if you actually Google or if you actually put this query into Google, this image comes before this image it is that there aren't a lot of images of animals. Which brings us to the next problem is that if you don't see images of animals then the model is confused because it sees a lot of Obama signs and these are black and white and blue sort of. And generally because what's happening is that people are just clicking on that image all the time to figure out what's going on in that image. So we got bumped up before that other movie. So now size that appearing a lot. You can also do images from text given text or tags can retrieve relevant images. So this is the data set itself at about million images is a nice. Although I can say that according to Google this is a much better representation for that sentence than this one. Here's another sort of interesting model which is a model where you're trying to build a recurrent neural network. Again, it's a generative model, but it's a generative model of text. This model was trained on about 7,000 romance novels. And you take a caption model and you hook it up to the caption generation system. So you're basically saying the model here's an image generate me in the style of romantic books what you'd see here and it generates something interesting. We're barely able to catch the breeze on the beach and so forth. She's beautiful. But the truth is I don't know what to do. The sun was just starting to fade away, leaving people scattered around the Atlantic Ocean. And there are a bunch of different things that you can do. Obviously we're not there yet in terms of generating romantic stories, but here's one example where it's a generative model. It seems like syntactically we can actually generate reasonable things. Semantically, we're not there yet. And actually that particular work was inspired a little bit by Baidu's system that was given an image. I think it would generate poems, but the poems were predefined. So it was mostly the selecting the right poem for the image. Here we actually were trying to, you know, generate something nice data set and you have, you know, very noisy tags. And the question is, can you actually learn some representation from those images? One thing that I want to highlight here is, you know, we've tried. There's still a lot of work to do in that space because, you know, syntactically we can get there. Semantically, we are nowhere near getting the right structure. Here's another last example that I want to show you. You know, there's a 25,000 labeled image. Somebody went and labeled what's going on in those images, what classes we see in those images. And you get some numbers, which is mean average precision. But what's important here is that we found that if we actually, this was done in the case of one shot learning, which is, can you build generative model of characters? Right. That's a very defined domain. Very well defined domain. It's a very simple domain, but it's also very hard. Right? Here's one example. We've shown this example to people and to the algorithm and we can say, well, can you draw me this example? And on one panel humans would draw how they believe this example should look like. And then on the other panel we have machines drawing it. This is really just a generative model based on a single example. You show a new example and you're trying to generate what it is. And so quick question for you. How many of you think this was machine generated and this was human, human genome unlabeled data? And we pre train these channels separately using a million. Ah, what about the other way around? More unlabeled data points. Then we can actually get some performance improvements. So at least that was a little bit of happy sign for us to say that unlabeled data can help in the situations where you don't have a lot of labeled examples. So here it was helping us a lot. And then more aha. So there's a vote that this is. What about this one? How many of you think this machine generated and this human generated a few? What about the other way around? Ah, great, great. Well, the truth is I don't really know which one was generated by which machine because that was done. I should actually ask Brendan Lake, who designed the experiments for this particular model. But I can tell you that there's been a lot of studies. He's done a lot of studies. And it's almost 50 50. So in sort of this kind of small carved domain, we can actually compete with people trying to generate these characters. Now let me step back a little bit. And once you get into these sort of representations dealing with text and images, this is one particular thing you can do. And I think, Richard, point it out what happens in the space of linguistic and tell you about a different class of models. These are models known as regularities. You can do the same thing with images. Just kind of fun to do. They sometimes work, they don't work all the time. But here's one example. If I take that particular image at the top and I say, get the generative adversarial networks, and they've been gaining a lot of attraction in our community because they seem to produce remarkable results. So here's the idea. We're not going to be really defining explicitly the density, but we need to be able to sample from the model. And the interesting thing is that there's no variational learning, there's no maximum likelihood estimation, there is no Markov Chain Monte Carlo. There's no sampling. How do you do that? How do you learn these models? And it turns out that you can learn these models by playing a game. And that's a very clever strategy. And the idea is the following. You're going to be setting up a game between two players. You're going to have a discriminator D. Think of it as a representation of this image. Subtract the representation of day, add the night, and then find closes images. You get these images and then you can do some interesting things like take these kittens and say mine is bold plus evolutional network, evolutional neural network. And then you're going to have a generator G. Maybe you can think of it as a box. You get kittens in the box, right? If you take this particular image and say minus box plus ball, you get kittens in the ball. Except for this thing that's a duck. So that's a variational thing called or a Helmholtz machine or something that gives you samples from the data, the discriminated D is going to be discriminating between a sample from the data distribution and a sample from the generator. So the goal of the discriminator is to basically say, is this a fake sample or is this a real sample? Fake sample is a sample generated by the model. Real sample is what you see in your data. Can you tell the difference between the two? The generator is going to be trying to fool the discriminator by trying to generate samples that are hard for discriminated to discriminate. So my goal as a generator would be to generate really nice. You can sort of get these interesting representations. Of course, these are all sort of fun things to look at, but they don't really mean much because we're not specifically optimizing for those things right now, looking digits so that the discriminator wouldn't be able to tell the difference between simulated and the real. Let me spend some time also talking about another class of models. These are known as Helmholtz machines and variational autoencoders. These are the models that have been sort of popping up in our community. That's the key idea. And so here is intuitively what that looks like. Let's say you have some data. So images of faces. They give you an image of a face. And now I have a discriminator basically says, well, if I get a real face, I push it through some function, some differentiable function. Think of it as a convolutional neural network or another differentiable function. And here I'm outputting one. So I want to output one. If it's a real sample, then you have a generator. And generator is. You have some noise, some input noise. Think of it as a Gaussian distribution. Think about Helmholtz machines given some noise in the last two years. So what is a Helmholtz machine? A Helmholtz machine was developed back in 95 and it was developed by Hinton and Peter Dayan and Brandon Frey. I go through differentiable function, which is your generator, and I generate a sample. This is how my sample in Redford, Neal, and it has this particular architecture look like, right? And then on top of it, I take this sample, I put it into my discriminator and I say for my discriminator I want to output zero, right? Because you have a generative process. So given some latent state, you just. It's a neural network discriminator will have to say, well, this is fake and this is real, right? That's the goal. And the generator basically says, well, how can I Get a sample such that my discriminator is going to be confused. Such that the discriminator always outputs one here because it believes it's a true sample, believes it's coming from the true data. Now you have these systems, so what's the objective? The objective is a min max value function. It's a very intuitive objective function that has the following structure. You have a discriminator that says well this is an expectation with respect to distribution data distribution. So this is a stochastic neural network that generates the input data, right? And then you have so called the proxy saying I want to classify any data points that I get from my data as being real. So I want this output to be 1, because if it's 1, the whole thing is going to be 0. If it's less than 1, it's going to be submit inference which is to say given the data infer approximately what the latent state negative and I really want to maximize it and then discriminate the sayings. Well, anytime should look like, right? And again it was developed in 95. There's something that's called wake sleep algorithm and it never worked. Basically people just said it just doesn't work. My generator sample whatever samples comes out from my generator, I want to classify it as being fake. That's the goal of the discriminator. And then there is a generator. The generator is sort of the other. You're trying to minimize this function which essentially says well, generate samples that discriminated with classifieds real. So I really am going to try to change the parameters of my generator such that this would produce zero, the discriminator would produce one. Right. So trying to fool the discriminator, right? And it turns out the optimal strategy for discriminator is this ratio which is probability of the data divided by probability. And you know, then we started looking at restricted bolt machines and Bolst machines because they're working a little bit better the data by plus probability of the model. And in general, if you succeed in building a good generative model and then two years ago people figured out how to make them work. And so now 10 years later I'm going to show model. Then probability of the data would be the same as probability of the model. So discriminator will always be confused as to 1/2. And here's one particular it seems like a simple idea, but it turns out to work remarkably well. Here's an architecture called deconvolutional do the trick. Now these models are actually working pretty well. The difference between Helmholtz Machines and Diebbolt's machines is very subtle. They almost look identical. The big difference between the two is that in Helmholtz generative adversarial network architecture that takes the code, this is a random code, it's a Gaussian code. It passes through a sequence of convolutions, also a sequence of deconvolutions. Given the code, you sort of deconvolve it back to high dimensional image and you train it using adversarial setting. This is your sampling, you generate the image and then there is a discriminator which is just a convolutional neural network that's trying to say is that the real, is that a fake? And if you train these models machines, you have a generative process that generates the data and you have a separate recognition model that tries to recognize what you're seeing in the data. So you can think of this Q function as a convolutional neural network. Given the data tries to figure out what the feature should look like. And then there is a generative model. Given the features, it generates the data. Also on bedrooms, these are called LSUN data sets, a bunch of bedrooms. This is how samples from the model would look like, which is pretty impressive. In fact, when I look at these samples I'm also sort of thinking, well maybe the model is memorizing the data because these samples look remarkably impressive. Boltzmann machine is sort of similar class of models, but it has undirected connections. So you can think of it as. Then there was a follow up work. These are samples from the CIFAR data set. So here you're seeing training samples and here you're seeing samples generated from the model, which is again very impressive. If you look at the structure in these samples, it's quite remarkable that you can generate samples that look generative and recognition connections are the same. So it's sort of a system that tries rich. Very realistic. Actually this is what's done again this was done by team, Team Salomon and his collaborators. If you look at the ImageNet and you look at the training data on the ImageNet equilibrium state when you're running it. So it's a little bit, the semantics is a little bit different between. And looking at the samples again you look this horse, there's is like there's some animal that is an airplane and so forth is like some kind of a truck and such, right? So in these two models, so what is the variational autoencoder? Variational encoder is a Helmholtz machine. It defines a generative process in terms of sampling through cascades of stochastic layers. When I look at these images, I was very impressed by the quality of these images because generally it's very, very hard to generate realistic looking images. And the last thing I want to point out, this was picked up by Ian Goodfellow. If you cherry pick some of the examples, this is what generated images look like. So you can sort of like see there is a little bit of interesting structure that you're seeing in these samples, right? And if you look at it, there's just bunch of conditional probability distributions that you're defining so you can generate the data. And one question still remains, what with these models is how can we evaluate these models properly, right? Is the model really learning a space of all possible images and how images, what's the coherency in those images? Or is the model mostly kind of like blurring things around and just making some small changes to the data? So the question that I would really want, would like to answer, to get an answer to, is that if I show you a new example, a new testing image, a new kind of a horse, would the model say, yes, this is a likely image, this is very probable images I've seen. So theta here will denote the parameters of the variational encoders. You have a number of stochastic layers, similar images before, or something like that or not. So that still remains an open question. But again, this is the class of models which steps away from maximum likelihood estimation, sort of sets it up in game theoretic framework, which is a really nice set of work. And in computer vision community, a lot of people are showing a lot of progress in using these kinds of models because they tend to generate much more realistic looking images. So let me just summarize to say that and sampling from these conditional probability distributions, we're assuming that we can do it, it's attractable, it has to be tractable. But the innovation here is that showing you hopefully a set of learning algorithms for deep unsupervised models. There's a lot of space in these models, a lot of excitement in that space. And just wanted to point out that these models, the deep models, they improve upon current state of the art in a lot of different application domains. And as I mentioned before, there's been a lot of progress in discriminative models, convolutional models using recurrent neural networks for solving action recognition models, dealing with videos and unsupervised learning still remains sort of a field where we've Made some progress, but there's still a lot of progress to be made and Let me stop there. So thank you. Every single conditional probability can actually be conditioned. Can be very. Question 20 minutes. Go to the mics. Oh, sorry. Complicated function. It can denote a nonlinear. You can model nonlinear relationships with. It can be a multi layer nonlinear neural. So as a Bayesian guy, I'm pretty depressed by the fact that GAN can generate clearer image than the variation in autoencoder. So my question is, do you think there could be an energy based framework or a probabilistic interpretation of why GAN is so successful other than it's just a Mimax game? I think that generally if you look at neural network, deterministic neural network, so it becomes fairly powerful. Here's an example of. I have a stochastic layer, you have a deterministic layer, you have a stochastic layer and then you generate the data. I sort of go back and forth between variational encoders because some of my friends at OpenAI are saying that they can actually generate really nice looking images using variational autoencoders. I'm looking at Peter here, but I think that one of the problems with image generation today is that with variational auto encoders there is this notion of Gaussian loss functions. You can introduce these nonlinearities into these models and this can. Right. And what it does is it basically says, well, never produce crystal clear images because if you're wrong, if you put the edge in the wrong place, you're going to be penalized a lot because of the L2 loss function. Right? What the GANS are doing. Gans are basically saying, well, I don't really care where I put the edge as long as it looks realistic so that I can fool my conditional probability with denoting one layer neural network. Right? Now I'll show you some examples, but maybe I can just give you a little intuition behind what these equations do. In a lot of these kinds of models, learning is very hard to classify. So what tends to happen in practice and a lot of times if you actually look at the images generated by Gans, sometimes they have a lot of artifacts like these specific things that pop up. Whereas in variational encoders you don't see that. But again, the problem with variational encoders is they tend to produce images that are much more diffused or not as sharp or not as clear as what GANS is doing. There's been some work on trying to sharpen the images, which is you're using variational tone coders to generate the globally coherent scene and then you're using generative adversarial nets to maybe sharpen it. Again, it depends what loss function you're using and Gans seem to be able to deal with. And there's a class of models called variational learning. And what the variational learning is trying to do is basically trying to do that problem implicitly because they don't really care whether you get the edge quite right or not as long as it fools your classifier. Thank you. Do the following. Well, I want to maximize the probability of the data that I observe, but I cannot do it directly. So instead. Hi, thank you very much for the interesting talk. I have a question about the variational autoencoder for the more challenging data set like the street view house number. I noticed that many implementation they use a PCA to. Instead what I'm going to do is I'm going to maximize the so called variational low bound, which is this term here. And it's effectively saying, well, if I take the log of expectation, I can take the log and push it inside. Process the data before they train the model. What is your thought on that pre processing state step? Why is it necessary to do that? Why don't we just learn from the raw pixel? I actually don't know. My experience has been that we don't really do a lot of pre processing. I mean what you can do is you can do ZCA preprocessing is you can take the mean, you can take sort of the second order covariance structure. Right. And it turns out just logistically working in this representation is from the data that sometimes helps, sometimes it doesn't. But I don't see any particular reason why you'd want to do PCA pre processing. Right? I mean it's just one of just like we've seen a lot in our field, people just doing XY and then later on they figure out that they don't really need X and Y. So maybe it was working better for their implementation for their particular task. Much easier than working in this representation. Right. If you go a little bit through the math, turns out that you can actually optimize this variational bound, but you can't really optimize this particular likelihood objective. It's a little bit surprising for those of you who haven't seen variational learning how it's done. Generally I haven't seen people doing a lot of pre processing using PCA for training variational autoencoders. Any more questions?

Yes, there's one queries regarding binary RBMs. So if you look at, you know.

But this one little trick, this one little so called Jensen's inequality actually allows you to solve a lot of the.

Literature for, let us say estimation of the partition function for easing models, right? You will see that the literature is a lot more rich compared to the variational inference literature for restricted Boltzmann machines, especially in the binary context. Is there like a cultural reason for this? Because specifically, you know, like you have for the strictly ferromagnetic case, you have a fully polynomial time approximation scheme for estimating the log partition function, right? So, but then I don't see usage.

Of these F problems, right? And the other way to write the lower bound is to say, well, first.

Kind of algorithms in the RBM space. So when you kind of juxtapose with the literature for the easing models compared to binary RBMs, you'll find like a very stark asymmetry. Is there like a reason for this?

Yeah, so the thing, the thing about easing models is that, you know, if you're like in ferromagnetic case or if you are, there is a log likelihood function and something is called KL divergence, which is the distance between your approximating distribution Q, which is your recognition model, and the truth, the truth in these models would be the true posterior. You have certain particular structure to the easing models. You can use a lot of techniques. Even if you use techniques like coupling from the past, you can draw exact samples from the models, you can compute the log partition function, the polynomial time. You have specific structure. The thing about RBMS is that generally those assumptions don't apply. You cannot learn a model which is a ferromagnetic model with RBMS just where all your weights are positive. That's a lot of constraints to put on these classes of models according to your model. And it's hard to optimize these kinds of models in general. So that's why. And once you get outside of these assumptions, then the problem becomes NP hard for estimating the partition function. And obviously for learning these systems, you need the gradient of the log of the partition function. And that's where all the problems come in. I don't think there is a solution for that. And unfortunately variational methods are also not working as well. You know, you're trying to optimize your generative model, you're trying to optimize your recognition model. And back in 80, back in 95, Hinton and his students, basically they developed this wake slip algorithm that was a bunch of different things put together. But it was never quite the right algorithm because it wasn't really optimizing anything. It was just a bunch of things, you know, approximations like contrastive divergence or something based on. Based on sampling. People have looked at better approximations and using sort of more sophisticated techniques, but it hasn't really popped up yet. Practically it just doesn't work as well. But it's a good question. Hello.

My question is about using autocoder.

To.

Get semantic hash, especially in text. Do we need to any special representation? Text. Text representation like word to vector as input for the. As for our text sequence.

So, you know, I've talked about the model, which is very simple model, just modeling bag of words. Yes, right. Obviously you can take word 2vec and initialize the model. Right, because alternating. But in 2014 there was a beautiful trick introduced by. Because it's a way of just taking your words and projecting them into the semantic space. There has been a lot of technique, recent techniques using like Richard was mentioning grus as a way, you know, if you want to, if you want to work with sentences or if you want to embed the entire document into the semantic space and if you want to make it binary, you know, you can use GRUs, bidirectional GRUs to sort of get the representation of King main welling. And there was a few other groups that came up with the same trick called reparameterization trick. Right. So let me show you what the repanetomezation trick does intuitively. So let's say your recognition distribution is a Gaussian. So I Gaussian, I can write it as a mean and a variance. So this is the document. I think that would probably work better than using Word2Vec and then just adding things up. And then based on that, you can learn a hashing function that maps that particular representation to the binary space, in which case you can do searching fairly efficiently. So, so as an input representation, there are lots of choices. You can use bidirectional grus, which is the method of choice. Right now you can use GLUF as the mean. This is the variance. Notice that my mean depends on the layer below. Or you can use word 2vec and sum up the representations of the words.

Okay, so using only back of word, we use only normal network, neural network. There is no recurrence network or.

Yeah, yeah, that's right.

Just simple.

That's right. But again, your representation can be whatever that representation is, as long as differentiable. Right. Because in this case you can sort of back propagate through the very nonlinear function. The variance also depends on the layer below. So it could also be a nonlinear function. Bidirectional grus. And get the, you know, learn what there is. What is the presentation.

Thank you so much.

Okay, let's thank Russ again. But what I can do is I can actually do the following. I can express this particular Gaussian in terms of auxiliary variables. So I can say, well, if I sample this epsilon from normal 01, a Gaussian distribution, then I can write this particular H, right? My state in a deterministic way. It's just mean plus, essentially standard deviation or variance, square root of the variance times this epsilon. Right? So this is just a simple parameterization of the Gaussians. Right? I'm just pulling out the mean and the variance. There's no sort of surprises here. So I can.