Scientist Stories: Andreas Krug, Training Neural Networks with Cognitive Neuroscience

This is the Data Science Seminar. Good morning, everyone. For the livestream audience, please use the chat button below the livestream window. Insert your name there and paste comments and questions. They will be moderated after the talk. A warm welcome to our speaker, Andreas Krug. Andreas Krug is a PhD student on explainable AI at the Artificial Intelligence Lab at the Otto-von-Guericke University Magdeburg. He has a background in natural science and holds a master's degree in bioinformatics. We are really happy to have you here. So today he will talk about neuroscience-inspired techniques for explainable AI. It's your stage, Andreas.

Thank you, Doreen, and also a very warm welcome from my side. I'm Andreas and very good introduction by Doreen already. I will talk today about my research in the field of explainable AI, in particular about training and understanding artificial neural networks. Networks with cognitive neuroscience-inspired methods. So in my talk, I will briefly give an introduction to deep learning model analysis, and then I will spend most of the time actually introducing you to some of the methods we developed about explainable AI using neuroscience-inspired methods. So in particular, I'm going to talk about two different approaches of ours, which is neuron activations profiles, which is an event-related potentials technique to analyze neural networks. And I will talk about activation topo maps to visualize artificial neural networks. So, but you will see about this later. So first of all, the motivation for our research. You probably all know about deep neural networks, which function as a black box. So you have kind of some input, you pipe it through black box network and there's an output. So for example, you can classify those cat images as a cat. And there are already some introspection techniques, so model analysis techniques out there, which usually use the opportunity to visualize the output predictions or the model internals in the input space. So we somehow project this back on the input. And two main methods which are used there are the saliency maps. So this simply takes a prediction in the end of the network and maps it back to the input space to figure out what are the important regions. So in this particular example, you can see a heat map which tells like this is the important part of the image in this case for the prediction. So these are saliency maps. And the other common techniques to visualize also internals of the network is activation maximization. So the idea there is simply to optimize an input which maximally activates some parts of the network. So I think a lot of you already have heard about the Google Deep Dream thing, which is part of this type of technique. So, but there are also a lot of problems with these techniques. So here's like some overview about the main troubles with this. So with saliency maps, although they are like very easy to visually interpret, it's not really global. So it's only looking at a single example and also it's only possible when you have an idea about the input data. So in an image, you have some expectation on where are like the important points, but in other data, this might be more complicated. So I will talk about speech later, but we will come to this. And there are also plenty of saliency map methods out there, and each method gives a different heatmap, so it's not too reliable, and sometimes it can also be misleading, as some studies showed. And in the feature visualization, this is more global because it's independent of the input example, and it also shows some model internals. But in order to look natural, it needs a lot of regularization, and also it's highly subjective to actually find some information in there. So what I mean by this is when you look at those patterns here, in order to evaluate whether this is like a useful pattern for something, you can mostly just visually look at it and think, yeah, does this make sense? And this is, in my opinion, mainly some like looking in shapes or looking for shapes in clouds, which is not like too objective. Mostly this is applied to data which is visually interpretable, as I said. But what happens now when we move over to an example which is not that easily visually interpretable? So for this, we are going to work with a convolutional speech recognition model in this talk. So about this model, I will use it for all the, or like many results in this talk. So that's why I will also introduce the architecture so you have an idea of what we actually analyze in the coming slides. What we use is here a fully convolutional speech recognition architecture by the Facebook people. So it's called Wave2Letter, and it's an 11-layer convolution network which only uses 1D convolutions, has about 25 million parameters, and a frequency of 50 letters per second when it predicts. So the idea of this whole thing is that you pipe in a spectrogram, which is a representation of time on the x-axis, frequency of the audio on the y-axis, and then the intensity of each point tells you how strong was this particular frequency and like intense was it at this particular timestamp. Then you pipe in a complete sentence of input in this network and let it predict the complete character sequence. It uses a CTC loss for this. So this is like not this typical scenario of supervised one-to-one mapping, but really having a sliding window for this convolution network and it predicts letters in the end, but as a sequence as well. So although the network itself is like not an input-to-one-output classifier, as soon as it's fully trained, we can treat the model basically as a classifier by taking snippets of the spectrogram which are exactly the receptive field size, and this is then predicted as one letter. So this doesn't make sense during training, but we use it for the analysis. So this is the model which we mainly use for the analysis here. Okay, so now let's see whether it's actually applicable to use the introspection method from computer vision, so what we saw for the images also in this example, because in the end the spectrogram also looks like an image, right? And then for the saliency map approach, again, we have now this classifier approach. So we have a particular spectrogram snippet, it's predicted as a particular letter, and we want to project it back to the input. And typical Heat maps, so saliency maps, would look like this. So as you might see, you don't see a lot. So you have a few points at some particular frequencies and some particular positions in the time, but it's not that nicely interpretable. I mean, if you're expert in the field, you could maybe get some hypothesis on what it looks at, but it's not at all as interpretable as for the image. So in general, those saliency maps on the input are applicable but not that nicely interpretable. What about feature visualization? They are dissimilar, like in early layers, you could still argue that the patterns look quite nice and someone who is familiar with phonetics and spectrograms would be able to make sense of these classes. So in this case, it's an optimal input for an A and an optimal input for T. So it actually makes also a little sense, but when you go to the deeper layers, which learn more complex patterns, then it looks approximately like this, and this is not interpretable at all anymore. Okay, to be fair, in this case there was not too much regularization here, but even when we would get like more natural patterns, it's way more difficult to do this looking for shape in clouds scenario like in the images because there's like less intuition about how how do patterns in the spectrogram look like. So this means also the optimal inputs approach is not really applicable in our case. Okay, and yeah, because of these problems with the regular computer vision scenarios and to have also an opportunity to have a more generalized approach on analyzing neural networks, our research tries to find a new way of actually doing this. Kind of research. And for this, we have a project called CoXAI, which is Cognitive Neuroscience Inspired Techniques for Explainable AI. So this is a BMBF-funded project for actually finding new ways of doing this kind of analysis. And our approach is to actually get inspiration from the cognitive neuroscience to make the aforementioned methods either better applicable to all the scenarios, but also to develop new methods for analyzing neural networks and to train them in a way which is easier to interpret. So the motivation for this in neuroscience, like real brains have been studied already over decades. And I mean, in the end, the brain is also like super opaque system which we can only measure from outside and We have plenty of years of experience from this to basically look into a black box. And this rich treasure of experiences is a good opportunity to also make use in the deep learning scenario. So in our project, what we basically do is we look at what are the neuroscientists doing when they analyze brains and how can we make use when we when we analyze neural networks or when we design neural networks to understand better what they do. And today I brought two of these methods, as I already mentioned before. So one is about how can we characterize artificial neuron functions with an event-related potentials-like analysis, and how can we identify functional regions in a deep neural network with activation topomaps. So the neuroscience equivalents of this both come from the EEG analysis. So if you don't know about EEG, basically what you have there is that you measure neuronal activity in the real brain by putting an electrode cap on the head. So in this right image here, you can see each dot would be a location of an electrode and it measures the activity of the neurons. But as it's measured from outside, this has a lot of noise. And so each of these points is also like a channel. So you have a lot of these measurements. And so you have the activation of the neurons on the time axis and this line plot here. And you can project this in a nice visualization as a topographic map. When you plot the activations like in this heat map on the top view of the head. So these are the methods from neuroscience which we adapted here for neural networks. Okay, but step by step, let's start with the first thing, which is event-related potentials, so ERP-like analysis of artificial neural networks. So this is this part in our setting. Okay, so event-related potentials are a method from cognitive neuroscience which are used to actually get meaningful information out of the measurements when you have EEG recordings. So the thing is that because the measurements from these electrodes are so noisy, the information is not clear from only one measurement. So what they do is they take a lot of measurements which are done at the exact same stimulus. So let's say a person just hears always the same word and then the neural activity is tracked afterwards. And each of these individual measurements is very noisy, but when you average them all together, you get a signal which is consistent. So the idea here is that the noise signal is always different, so this stuff is averaged out when you have n different measurements. But if you have this alignment point at the exact same stimulus, and there's always the same reaction in some small form in these measurements, then those are not averaged out and you can get through averaging a consistent signal in the end. So this is the idea here. And we are adapting this to neural networks. And to understand how we do this, we will first start by more intuitive example, namely by averaging inputs in general. So like averaging the measurements of this EEG recordings. Let's look at a more pictographic example of averaging faces, for example. So here, that's just an art project of someone that averages particular faces. So in this case, it was like 9 paintings and By simply taking all these inputs and computing the average, you can get kind of a characteristic representation of the face. So you can see that like from individual pictures, there may be some leftovers because it's only 9 paintings. So this, whatever tiara or whatever this is, but some consistent face features like a nose or an eye, they are on all pictures and so they stand out in the end. So that's the idea here. But in this case, instead of like in the ERP for EEG data, we don't remove noise in measurements. So this is what they did. Instead, we do not have noisy information here because the inputs are clear and we also know all the activations in our neural networks. But we can remove the variation which comes from feeding different input data. So in this case, as I said, like consistent features of this face, so all images of a particular face are thrown in and you get the consistent features out of it. Okay, so yeah, the same we can now also apply to speech. So the idea here is we now simply take all train and/or test data of our model prediction for a particular class and we compute the average of all the inputs. So this would look like this. So again, this is only the idea for now. So we take all the examples of a particular letter. Let's say we take A, then we compute the average and we would get a common pattern which has consistent information for this particular letter. But unfortunately, this does not work in one step, and that's because of the alignment of input data. So let's go back to this. Other image for a sec. So you might imagine that when you take a lot of faces and they are not aligned, so let's say the nose is at different positions in each, in each of the images, then the average would not be meaningful, right? You would just get a messed up average of all the faces and there are no consistent information. So that's why you first need to like align these faces at the nose point, for example. And this we would also need to do in speech because otherwise it wouldn't work that nicely. So especially like in speech, the stuff is not aligned, which you can imagine because there are 2 seconds of audio for one letter prediction. So you can from the prediction not really say where is the location in the speech thing. So here's an example, but I won't go through it in detail here. Just know that it's not aligned at this point. And we also don't know the true alignment because for the inputs we only know the complete sequence but not the actual position. And yeah, how do we get this alignment if we don't have it? Our idea for this is that we use the saliency maps here. Although they are not like directly interpretable, they still tell us where is the position in spectrogram which is important. So how does this look like? We have here the spectrograms and the corresponding saliency maps coming from sensitivity analysis in this case. And what we can now say is, because the sensitivity is based on the prediction, then the most important points based on the saliency map are used as an alignment point because we assume that the network actually puts the importance on this occurrence of the particular letter. So we can even take like the sensitivity maximum or minimum, or maybe also the maximum of the absolute value. So those are options, but that's the idea here. And with this, we create like an estimate of the alignment if there is no ground truth alignment available. What we do here is centering the inputs by centering. So we take the maximum and center them to make the alignment. But what is also possible is to use this as a starting point, like in the ERP analysis, because there you have the stimulus and afterwards the signals come. This is also possible here, but in our case, this doesn't make much sense because information from before and after can be included in the network predictions. Okay, so yeah, now we have the averages without alignment here, and in the first column it's without alignment, in the other columns it's with alignment, but still you see that this doesn't really look like a pattern, right? So this is still just some fuzzy average and doesn't make really sense. The reason for this is that usually the letter-specific information, or in general the group-specific information, is overshadowed by information which is consistent across all data. So for example, it could be in this case that it looks like a spectrogram at all, for example, or in the activation setting it could be that there are some neurons which have a baseline activation. So we want to extract this as well. So Instead of only aligning, we also subtract the average over the complete dataset. So kind of a normalizer, which is the expected average input in this case. And if we now subtract this normalizer, we can actually get interesting and meaningful patterns for the inputs. So without alignment, this doesn't work. That's why we need alignment. But when we align and subtract the normalizer, we can actually get meaningful patterns, which is for letter A and letter T. For those who are familiar with the frequencies for particular sounds, this also makes sense. It's not perfectly fitting, but it makes sense. Also like this sound for T, it also makes sense that it has this sharp edge between high frequencies and low frequencies. We get letter-specific patterns due to the normalization. It's not a spectrogram anymore, But yeah, that's just because of the normalization and we can use it to learn something about the model. So here we could actually say that the patterns make sense in terms of the letters that they are supposed to learn. So this is a method which we introduced in 2018 actually, which is called Mavai, but this is just a special case of the stuff which follows afterwards. So I will not go into detail here again. So yeah, we talked about inputs and I hope that you now have an impression of like an idea of how this is meant to work, but we can also generalize this to the activations. So instead of using the inputs on the left side here, we use activations somewhere in the middle of the network. So in this case, it was a ReLU network. So you only see some positive activations here and we can now apply it in the exact same way. So we take all the activations of a particular layer here, and we average them over a particular group. So again, let's say all the activations in the particular layer for letter A. Then we subtract the average over the complete dataset, which is this normalizer, and we end up with an average which is normalized for this particular— an average of the group activations normalized for this particular letter in this one layer. So this concept we call neuron activation profile because it characteristically describes the activations of the artificial neurons in this network. Again, for a particular letter in a particular layer in this case. And yeah, in this version, there is a lot of information also in outer regions, which is a little weird because usually networks tend to not look at the complete area of the input or the complete feature maps, for example. So that's why we also add an additional step here. So these neuron activation profiles, we also multiply with a gradient mask. So again, averaged over the gradients which we already took for the alignment, we can now also get a mask for where's the network actually looking at in terms of the gradient in this case, but it could also be another saliency method. And what we get out of this is then what we call a GradNAP, so a gradient-adjusted neural activation profile. So this combines both the good information here. So we have on the one side this consistent information from the neural activation profile,, but also we only show those information where the network actually kind of attended to. So here's no attention mechanism, so it's just like where the importance based on the gradient is placed. Okay, so, and that this actually makes a difference also like in the input neuron activation profiles, you can see here. So this is the average input which we saw before for t, So a little different because it comes from another experiment, but yeah, so you have the T which comes from the thing without the gradient mask. And when you apply the gradient mask, you have additional information that like this complete outer region is not really used. So all the information which are there maybe are consistent, but are not used by the network for the prediction, only to a very small extent. Okay, now you know about like the concept of neuron activation profiles, how to compute them, but you probably wonder how do we actually use them, which is far more interesting, I assume. So here's an overview of some examples how you can actually use neuron activation profiles. So let's start with one example to use or to inspect the model without using neuron activation profiles. So what you usually are interested in is to figure out why are errors made in the network. So what you can do is in the regular scenario without our method, you can compare what are correctly classified inputs. So here we have our beloved MNIST example, which probably we all worked with already. So correctly classified ones and misclassified ones in this case. And it would be even here a little possible to say, okay, We see a lot of these ones with a stroke on top and on bottom. They are misclassified. Also, we see very thick ones and very rotated ones. But on the one hand, this needs visually interpretable inputs, and it's also difficult to see very general behavior. So now we can actually use our neuron activation profiles on the input. So this input averaging approach to make this a bit more structured. So what you can see here is actually inspecting input layer neuron activation profiles for correctly classified examples in the first row and misclassified examples in the second row. And in this case, we again subdivided into different groups, for example, so to have like a better spread. So, and then we can see that correctly classified examples look like this. So they are typically straight lines with different rotations, and the misclassified examples are sometimes also straight lines but with weird rotations or extreme rotations or with a very strong thickness, or if they are not straight lines but have these artifacts like these strokes on top and bottom. So this is quite interesting because from inspecting here, we can see more general behavior. Still, we need visually interpretable data, but This is already a nice application. So yeah, to actually find those subgroups, so for which groups we would compute these neuron activation profiles in the input, this highly depends on the variation of the class itself or the group, more generally spoken. So in MNIST, we have relatively little variation. So a 1 always looks kind of like a 1, but when you think of a regular image classifier, all the different cats, for example, there's a lot more variation. So we would need to also use more subgroups for this. Yeah, but this is relatively straightforward because we can simply cluster the activations here to find similarly active or similar reactions on the inputs here and then group them automatically. Okay, so we can apply the same thing also again for speech. So we can inspect the input neuron activation profiles here with gradient adjustment for particular phonemes. So phonemes are basically sounds and compare whether they are or whether they show frequencies at the correct position. So with the /t/ sound and the /r/ sound, for example. So we can do this as well here. And we are not limited to only using the classes. So in this case, you might remember that this model which we used was trained on the characters or the letters, but we can also investigate a grouping across a phoneme. So yeah, a particular letter can be pronounced differently and also a particular sound can be composed out of different letters. So here we can use a grouping which has nothing to do with the original labels, but we can actually use any group which we are interested in. I mean, this error analysis is similar. We compare the group of misclassified and correctly classified, which is also not part of the labels. So this grouping is very, very flexible indeed. But yeah, you only saw stuff for the inputs until now, but what can we do with deeper layer neuron activation profiles? Because they are not visually interpretable. So in our 1D convolutional scenario, it's really not possible at all to see something. If you would work with 2D convolutional architecture, it might be a bit easier because the feature maps to a certain point are also visually interpretable. So then you would also be able to inspect the hidden layer neuron activation profiles by a visual inspection. Okay, so yeah, what do we do with the activations if we cannot interpret them? So for example, we can see which ones are most active. So let's just convert the thing in a line plot so it looks nicer and looks more like the EEG data which we saw before. So what we can see is then here we have some high peaking activations of particular neurons, or in this case feature maps. Again, this visualization is possible because we have 1D convolutions. In a 2D convolution, you would have like a 3D peak here, so it doesn't make much sense. But in any case, you can find like the most responsive neurons for a particular group. Can visualize them, can identify them, and also find jointly activated neurons for a particular group. And for those, you can, for example, do targeted feature visualization. So instead of only blindly optimizing input for some arbitrary neuron or for an output class or whatever, you can also take now those most group important neurons and optimize input for those particular neurons. And this has the advantage that you can now also see like jointly learned features because in neural networks we have distributed representations and rarely it's only like one neuron or filter which performs a particular task, but mostly it's a combination of many. And so here you can identify neurons or filters which work together and then optimize input for those. So yeah, in this case, we now can have different neuron activation profile for this particular phoneme, for this particular phoneme, and optimize inputs for those responsive neurons. And we can actually see that when we only optimize for this set of particular neurons, we can get an interesting and meaningful feature out of it. So as opposed to the feature visualization you saw before for speech, because they look kind of similar, but here it's really targeting important nodes in the first or second layer. I think it's the second layer, while the other thing was only optimizing for one single node. And here you can actually visualize for pattern detectors that work together. Okay, so this you can do. What you can also do is compare representations. So this is a very flexible approach to investigate whether the representations of the groups have some similarities. And for example, we can then compare to human intuition. So to show whether this representation is something which we as a human would consider as meaningful. So what do I mean by that? Let's just go through for an example. So what seems intuitive to you is when you have like letters like A and E, they should, or like they are more similar to you than A and E are to T. So because those are vowels, so A and E are vowels and T is a consonant. So conceptually, they should be similar, so A and E should be similar. And this is not only true for letters, but it's also like for sound. So this is like phonemes for /au/ and /or/. They should be more similar than to /i/. So from sound, they are different vowels, but still there are conceptual similarities. So, and using our neural activation profiles, so when we compute for each of the groups, so each for each letter in this case, or for each phoneme in this case, and compute the neural activation profile, we can now use the similarity between those neural activation profiles to compare them on a very comprehensive level. So for example, like this. So here we have in one layer of the network, the comparison of the neural activation profiles with respect to their like the similarity of those for all the possible groups. So this is all the letters, but in the 8th layer of the network. And we can then by clustering, like by similarity analysis in this case, through clustering, can identify which of the classes are similarly represented by the network. So it's not a similarity based on like the inputs or whatever, really by the reactions of the network. So, and In this case, okay, we have a group of vowel letters including the Y because it's mostly predicted similarly in English. Then we have a bunch of consonant letters and some of the letters like T and D are closer to each other than to others, which makes sense. So we can really globally compare this complete letter with respect to all the possible letters. And we can also extend this more to comparing groups and looking into it in different layers. So this gets a little smaller, but the idea here is that we can, on the one hand, compare it for the letters and on the one hand for the phonemes. Then we do the exact same thing for different layers. And what we can then see is where are particular concepts represented. So where are phonemes represented, for example? So is the particular sounds learned earlier than the actual letter prediction, which we would expect, but it's not guaranteed that this happens. And here we can, for example, see that some concepts based on the phonetic similarities can be learned already here in the 9th layer of our network, while the similarity with respect to the letters is not that strong. It exists, but it's not that strong. And then in the following layer, also the letters are enough to distinguish, and the similarities get more contrasted in the phoneme level as well. So as you can see, because the colors are more contrasted here. Okay, so this is again visual inspection, but this can also be quantified by measuring the quality of the clustering, for example, with silhouette scores, for example. Or what you can also do is add another method for quantification, like adding a linear classifier probe, which is then regression-based, to find out whether the representation is nice. So Linear classifier probes would simply take the representation in this particular layer and try to predict the class directly from it. So this is possible for quantification as well. Okay, so yeah, you saw some applications now of neural activation profiles, but in general they are very flexible because they simply give you a characteristic response of the network and it has plenty of uses. So to I summarize those neuron activation profiles just quickly, a few strengths and weak spots which it still has. So neuron activation profiles are global because they are data-driven and don't work on a single example, but really take as much as possible into account. And it's not necessary to have interpretable input data because you can also analytically compare the reactions, like with this clustering approach, for example. Neuron activation profiles are like in the core, just averages of activations. So you can obtain them for any architecture. So you're not limited to CNN or MLP, whatever. So you can also apply it to any architecture. And yeah, as it's just a characteristic response of the network to a particular group of inputs, you can use it then for a variety of analysis depending on your needs. So That's very handy and there are also some weak spots. So as you might imagine, computing all the activations for every example is quite computation expensive and also averaging over all of those. And if you might want to reuse the stuff, you might, you need to store all the activations and saliency maps and alignments and whatever on your hard drive, which is quite storage intensive. I mean, you could simply do estimates by just picking a random example, but then the estimate is worse and you also need to introduce like tests for statistical significance, for example. As you saw with the image examples, it's less informative if there's like higher intragroup variance. So then you need to subgroup what you have, but that's a good workaround and not the problem actually. It can help you in figuring out what are like different subgroups in your group, which is thought as only one. And another weak spot is that the alignment relies on the saliency map method. So in the end, when the saliency map is like completely wrong, your results might also get a little weird. But it isn't important that this alignment is at the exact right position. It just needs to be consistent. So it's a little less sensitive than the saliency map method itself. Okay. Also, in a lot of cases, we also have already aligned data. For example, when we have a camera always pointing at the exact same location, then we do not need alignment. So in this case, this is also not the problem concerning this weak spot. Okay, good. Yeah, then there is the second method, which will be shorter than the first one. It's about activation topomaps for artificial neural networks. So this is the second part in the overview figure. So, and what we are implementing there is kind of a visualization which works like this topomap on EEG data. So having these activations of real neurons in the brain on top of the projected, on the top view image of the head. This is not that straightforward in neural networks because we have no ordering of the neurons or the filters in the network. I mean, in this EEG data, you have all these electrodes which are located somewhere in the head. So they have like an implicit neighborhood in there, but we don't have this in neural networks. So this particular red neuron, we could put at any position here. We could simply change the index and still the network works the same. So there's no implicit ordering in the neurons. So as an idea for obtaining this possibility to visualize it in this way, we are arranging the neurons to visualize them in a similar way. So in a regular convolution network, so again, our speech recognizer in this case, when we look in the filter space of one layer, then the filters which are learned look a little like this. So this is just like a zoom in 9 different positions. So this grid which you see here has no meaning. It's just all the possible filters in the space, and they do not have an ordering, right? They are learned at some particular space, and this doesn't have a meaning. And in the neighborhood, they are not similar. It's just the position does not convey information here. So what we do is, for example, we want to have like activation similarities in the plot. And for this, we are first trying to make the filters themselves similar to go from something like this to an ordered filter space. So this way, for example, uses some self-organized map which arranges the filters in a 2D grid such that the neighbors are similar. I mean, there are plenty of methods to do that. You could also simply let it arrange with similarity measures and then arrange it on the grid. But yeah, SOM is a nice machine learning method to do this like flexibly, but we also investigated other variants. So the idea of the SOM, for those who don't know SOM, You have like a grid of positions in the SOM and you train it in a way that you pick a particular data point, you search for the closest position in the self-organizing map, and you move this particular position a little closer to the data point. And it's not only this data, this position in the map which is pushed towards that point, but also the neighboring ones to a certain extent. And this way you update the closest one to be similar to this data point, but also the neighboring ones. And by this, you learn a representation of this whole input space where the neighbors have similar representations of data points. So in this scenario, the grid would then in the end look like this. And when we visualize this on some grid, it would be possible to again use this as a grid and then show this particular data point it's at this position in the song. So that's the idea here. When we apply this for our neural network for speech recognition, again, we can get such an ordering. So we have then groups of similar inputs. If you zoom in, you can see it better. There will be a picture in a few slides. For zooming in, but here you can then have regions of similar filters that work in a similar manner. So here, like rising pitches or plosive sounds, for example, here. So yeah, then they are similar. So what we can have now, so basically we have a location in our filter space, which is like simply similar regionality here so that positions of filters are similar like here. And then when we have the activation, so here we would in the EEG equivalent plot the activations of each of the electrodes here, and here we can plot the activations of each of the filters in this case. And then we can see that we have some kind of smoothness here. It's not so strongly visible because this was an early result from a few years ago. And yeah, this is the idea for ordering the filters afterwards. But if we can also apply this method to directly include in the training, so we don't need necessarily to reorder the filters afterwards, but we can directly include it as a training constraint. So what we can do is already define the script, so not with a self-organizing map, but simply assigning coordinates, so to say, to have a positional grid. And then we add to the loss function an additional regularizer which takes those filters. If they are like in the neighborhood, they should be similar. So it's simply a penalty which we add which computes the cosine similarity from a filter to its neighbors, and this should be minimized for all the neighborhoods. So there's a lot of weighting going into play and a lot of experiments on how to actually end handling the edges and so on, but it's not too important. But what we can then obtain instead of having these unordered filters, we can then get an ordering of similar kernels. So this is really included in training and we can see that neighboring filters now really have a similarity to each other. If you would look at another edge of this complete filter space, then the filters would of course look differently, similar to this sum ordering. Okay, and now when we have ordered those filters, we can also plot the activations for particular groups. So here, each of the filters we now reduce to like one activation point per feature map, and we can plot our activation maps here. So what you can see is actually for a complete layer, each of the points is one feature map and this scale averaged activation. And you can see that some of the regions really are working together and depending on the class. So again, this is different sound examples from before. And you can see that for some sounds there are active regions like here or inactive regions like here, and we can actually distinguish between different examples here. Exactly. And by doing this, we can actually have a visual cue on how similar are the groups and where are particular active regions. Another alternative, which I won't go into detail now, But instead of using the weight similarity, because it's kind of a proxy for this activation similarity, we can also encourage the activation topography explicitly. So instead of saying that the filters should be similar if they are in the neighborhood, we could also directly say like the activations should be similar if they come from a filter in a neighborhood. And we already had some experience on this. Again, I won't go into these for now. We also need something for Philip to talk afterwards. But a key point is that the activation topography learning would require feature maps of many examples. And that's actually like a huge difficulty because in order to regularize the activations, you can't simply look at one example, but the activations should be similar for a lot of examples for this particular group. So this makes the optimization approach actually a lot more complicated. But yeah, there are different methods which we are currently researching in this direction. And also we can of course combine neuron activation profiles and topographic maps. So what we can do is really use the neuron activation profiles values and plot them on this topographic map which we trained. So what we can then really see is that we have characteristic activation patterns on a regularized topographic map here. So, and can actually see that there are active regions for particular classes and can now also investigate, do they make sense? So for example, we have this neuron activation profiles values plotted on the topographic map for truck in the bottom right corner. It has like an active region here which spans over to the top side here because it's continuous. And the same region is also part of the automobile here, which makes sense because they're kind of similar. But also there is a difference here, like this central region of activation is different to here. So, and yeah, this is here an early result which I wanted to show just to demonstrate the capability of activation topo maps and neuron activation profiles working together. And exactly, so yeah, with this I also like to summarize the stuff which I showed you in the talk. So basically what I showed you is our methods on a data-driven and global deep learning model analysis, and in particular I showed you our method of neuron activation profiles with some variations of it, which function as characteristic network responses for arbitrary groupings of inputs. So whatever you choose for this, and also techniques for easier visual interpretability. So in particular, which are not only working on the input space, but also give you a visual interpretation possibility for particular layers comparing groups to each other. And here I showed you only a few of the use cases, especially about investigating the learned representations for self-defined groups or automatically gathered groups and comparing their similarities. And also that it's nicely possible to analyze prediction errors with this, with our tools. But other use cases, for example, might also be for model compression. So you can identify in the neural activation profiles which neurons basically do the same thing and you can compress them away afterwards. Or you can also do targeted fine-tuning to when you identify the group responsive neurons, then you could target the fine-tune those which are making a mistake, for example. So seeing the ones which are responsible for a particular classification error and then fine-tuning those in particular. So there are a lot of possible use cases as well. So exactly, so in terms of the next steps in research which you can expect in the future, like in the near future, is that there will be a lot of more evaluations for more types of datasets because So far we mostly showed everything for speech, but we are also working on different evaluations for image recognition where you already saw some early results, for example. But yeah, I'm always very interested to also see this in like your datasets. So if there's like interest in collaboration, then I'd be also interested to start a collaboration project. And also we are working on making a general tool box for applying our neuron activation profiles to really allow you to simply take your model from TensorFlow or PyTorch or whatever, give it into toolbox and let it automatically produce neuron activation profiles for you. And we're working on this to make it as general as possible to avoid problems with this at the moment. Okay. Yeah. So In our project which we are doing, there are also a lot of other tracks, so also by the colleagues which are working on it, which I don't go into detail here. For example, it also includes predictive coding and active learning and a lot of other more things. If you want to hear more about the project or about my research in particular, I'm very happy to get in touch with you. There are contact information everywhere spread around. And yeah, with this, I'd like to thank you for your attention, and I'm happy to get into discussion with you, hearing your questions. And yeah, here's also like the main references for the resources and a selection of our research which shows the most recent paper which are published about. Neuron activation profiles and the electrophysiological maps. Okay, so thank you all and looking forward to hear your questions.

Yeah, thank you, Andreas, for giving us insight into your research focus. So now we have some time for your questions. You can ask them via the chat window below the livestream and Now it's time for Lena. Maybe you want to ask some questions also to Andreas. Yeah, many, many thanks, Andreas, also for the offer for collaboration. And yeah, obviously your topic of research is very, very relevant. And I think there's actually two tracks of applications. I think on the one hand, you need the explainability for method development to understand what your method is doing, right? But I think there's also increasing pressure to get such explainability into actual products, especially on the clinical side. So there's actually new laws on this. So I was wondering, do you think to bring this actually such methods into the clinics, do we need to collaborate maybe with other disciplines such as designers? To actually bring the message across? I mean, I guess for a developer, I think what you're showing is totally fine because developers are experts in the field. But do we need to go beyond this?

What do you think? That's a very good question. And also, like, we are thinking about this a lot and also included this, like, in the planning for this project because, yeah, as you said, that the current state is in particular important for the developer to, like, make error analysis and to to make the model better. But like also one goal of this project or like my research in particular, my work is to make the explanations or like have an abstraction level of these visualizations which are communicable to people who have never seen AI, so to say. Have a communications tool also with the general public to tell, yeah, here we have this and this plot visualization, whatever. And trying to convey like the working of this method without going into detail so far. And so personally, I do a lot of like science communication to like test also my visualizations on the general public and trying to like explain to those people like what does it show and see like whether they already understand it. And if not, then I can also take back this feedback into my research. So I also do this like on a personal level and Yeah, this is not like part of the current project, but it would be like really nice to actually do also like user studies on what do the people actually want to see when they want explanations. Also like in the clinical setting, this would be very interesting to see like what do the patients want to see when they have some kind of treatment recommendation based on an AI system, right? So Yeah, it would be interesting to have also this combination and I find this very important and exciting.

And maybe now that we have you here as an expert, I would also have to admit that I find this topic really interesting, but we really haven't worked on it besides, you know, beyond the standard attention maps or activation maps, etc. Do you see any new trends in the field that we should be aware of in explainability, machine learning in general?

Okay, so do you mean like methodological or also like political?

Well, I was talking about methodological, but feel free to answer as you think.

Because you all were also like mentioning like law stuff before, so I was wondering. But yeah, so I mean, from the methods, I think first of all, It's getting more and more important to always have explainability of models on the side. So also like on the— when you publish stuff, people want to know why it works. This was a long time not at all the case, but now also the reviewers want to have some form of explainability in there. That's kind of nice. I don't like to elaborate at the moment, but this starts to happen. So it's getting more and more important. I guess also in your field, so we also have some medical corporations in other fields, they also ask always about why is this model now performing better, for example. This is at least a requirement also. Trend-wise, I think still we are currently more focusing as an explainability community on this whole saliency map stuff. Also because a lot of the data is actually visually interpretable. So when you think about like fMRI scans, like, or yeah, I mean, that's the most typical thing which I can imagine right now, like, or CT, whatever, then you have kind of a scan which everyone already kind of has seen. And so it's sufficient to really just highlight regions. And so when we want to convey that it works, it's kind of nice for a person to look at it. But yeah, I mean, what I think will come like very soon is to have some methods which also like focus more on the quantification of the quality of the models. So people are working on this. I mean, it's a super tough thing, but we actually want to quantify the AI quality, not only in terms of the prediction quality, but really in terms of is it reliable, where are like the decision boundaries of the decisions made, not only the why is it predicted this way, but also what would make it like another prediction class like this. So I mean, that's super tough because these models are extremely complicated functions with, yeah, where only like a very, very tiny portion of the actual possible space is captured by the dataset. So yeah, but still, if we really want to employ this in like critical applications, and I think like politics also, like notice like policymaking, then we really need some kind of AI audit. So, and there's also a lot of interest in the community both and the politics in doing so. Hope this answers your question.

Yeah, I was gonna ask that almost next because I think it's very hard to validate methods for explainability, right?

But you already— yeah, we rarely have like a kind of a ground truth of what we actually expect there, right? So what is explainable? What is— do we actually want to see there? And that's actually at the moment the biggest issue I see. So no matter which method you develop, you can't really compare it against something which you want to have. So you have to develop your own expectations and you formulate them in a way which are not consistent everywhere. But yeah, at least like the xAI community is aware of this, but it's like a tough nut to break.

Thank you very much again for your talk. This was the last talk before the summer break. So the next seminar, data science seminar, will start in October then. Have a nice summer break. Thank you very much. Bye-bye, everyone.