Scientist Stories: Robert Geirhos, Human and Machine Vision

Welcome all to the Data Science Seminar. It's my great pleasure to welcome Robert Jiros from the University of Tübingen and the International Max Planck Research School for Intelligent Systems. And also welcome to Paul Jaeger, and thank you for taking over the introduction.

Yeah, hi everyone. I'll try to make this short because we want to start with the talk, I guess. Just very quickly, I'm Paul Jaeger. I'm heading the Interactive Machine Learning Group here at DKFC and taking over for Lena today, who's sick. Just very quickly, we're looking for PhD candidates, so if you're looking for a PhD position or know anyone who's looking for it, please talk to me. All right, but now we're gonna introduce our guest today, which is Robert Gaiaros. I hope I pronounced it correctly. And as Doreen mentioned, he did his PhD at the MPI in Tübingen. Where he's now working as a postdoc, I guess, with Matthias Goethe and others. And I have to say, I've admired your work for a long time. So I guess most of the people here, or many people, know the Texture Bias paper, which was discussed here in several journal clubs, and also your perspective on shortcut learning. I actually used that paper during my recruiting process for my students as a journal discussion. So I really like your perspectives. And I think you also have some other fanboys here in the crowd today. Just the guy across, sitting across from me, said this morning, ah, when I heard that Robert is coming today, that made my day. And yeah, so I think people are just rooting for you because it's like a really high-quality lab from Germany that is also academia and takes a stand against the large corporate organizations dominating the machine learning community. So yes, welcome, Robert.

Thanks, Paul, for the very kind introduction. It's my pleasure to be here today. And thanks everyone for hanging in there with my Zoom problems. Glad we managed to find a solution. I'm going to take over and share my screen here. And yeah, today I'll be talking about how we might be able to narrow the gap between human and machine vision. And before going into any details, I wanted to say that at this point, I think we're at this point where we can see both the potential and the dangers of deep learning quite clearly. And this is a really unique and interesting point because on one hand, we have seen so much progress. For example, AlphaFold promising to help develop medicine through predicting protein structures. Also the promise of autonomous driving potentially saving thousands of traffic accidents, avoiding them through self-driving cars. So the potential is quite clearly visible at this point. At the same time, the dangers have also become apparent. So staying with the example of autonomous driving, for example.

If you've all heard those, you know, seen those examples of what happens when something goes wrong, for example, here a Tesla on autopilot failed to distinguish an overturned white truck from the bright clear sky.

So both the potential and the dangers are quite clearly visible, and so I think at this point One of the most crucial aspects really is whether understanding of machine decision-making is going to be able to keep pace with applications of machine learning. And in order to better understand machine decision-making, I think usually deep neural networks are getting extremely proficient at most tasks that they are trained on. The classic task would be just getting an image, outputting a prediction like Tabby Cat here for images. But then again, arriving at an understanding of what happened in between and what the different processing steps were and how those millions of parameters work together to form an understanding of what a cat looks like, that is a pretty challenging task. And I think there are many different aspects to understanding deep neural network decision making. Personally, I've specifically focused on, or we've looked at, understanding machine decision making through comparing it with human decision making to have a bit of a reference point for getting a better understanding of machine decision making. And the hope is that ultimately a better understanding of machine decision making is going to enable us to improve machine decision-making as well, which in turn might again inform a better understanding. All right, for the purpose of the talk today, we'll be looking at two comparisons between humans and machines. And the first one is going to be a bit more high-level on the topic of shortcut learning, which we think is a common unifying pattern behind many perhaps unintuitive machine learning failures. And then the second comparison is going to be more specific, specifically to out-of-distribution generalization, an area where we can compare human and machine perception directly on exactly the same images and see how we might be able to make progress.

Towards a better alignment of human perception.

All right, starting with shortcut learning.

As Paul has mentioned, this is more of a perspective. This is not a traditional research article where there's like a lot of novelty. This is more a certain way of looking at things, and hopefully looking at them through a lens that helps us to see certain patterns behind perhaps seemingly disconnected machine learning failures.

Deep learning, as we know, can work very well, but it can also fail unexpectedly. And one really interesting example here is this example by Zechin and colleagues where they trained a neural network to recognize pneumonia from X-ray scans. So they just went through the standard deep learning pipeline.

So they started by collecting lots of X-ray scans from different hospitals, getting a big data set, and then training a neural network on this task. Now what they noticed is that somehow there was an unexpected failure.

The neural network was getting really well on those specific hospitals that it was trained on, but when presented with scans from a different hospital, different scanner, the network was getting back to chance level quite quickly and showing sometimes really complete failures.

So at this point, when the researchers thought about this, they said, well, what's going on here?

And they started to investigate this and discovered that actually the network wasn't really looking at the lung that much. It rather had learned to identify a hospital-specific token, as you can see in the top right corner, that the radiologists have placed on the images in a specific hospital.

And then the network had learned that, you know, it just so happened that in this hospital the pneumonia prevalence rate was much higher than in the other hospitals. Maybe it was specialized for pneumonia.

So the network had essentially learned to identify the hospital rather than pneumonia.

And then obviously it had no idea what to do with scans from a novel hospital that it hadn't seen before. So in deep learning, there can be many such unexpected failures, and usually an unexpected failure in deep learning reveals a discrepancy between the intended learning outcome, in this case pneumonia recognition, and perhaps the actual learning outcome, what the model had indeed learned when they really started to investigate this, in this case just hospital token recognition. And this discrepancy between intended and actual learning outcome is a common pattern, a threat that runs throughout many areas where deep learning is applied, and potentially beyond, as we will see in a second.

So staying within the broader area of computer vision, working with images, another very prominent example is the so-called cow on the beach. So neural network trained on standard cow images gets extremely high accuracies, but then unexpected failure— the network fails when presented with images of a cow on the beach. And in this case, what was going on is that actually in the training data, it just so happened that All the cows were usually photographed in a standard setting.

If you go to the Swiss Alps or maybe the Allgäu or somewhere else, you can see that usually cows are standing on a maybe green landscape, maybe some mountains in the back.

So the network had just learned to take grass as a shortcut for predicting cow, and it was getting really high accuracies on the training data, obviously this failed when going to an unexpected setting like a cow walking on the beach where the grass is missing. So again, a striking discrepancy between intended learning outcome, object recognition, which really sounds like a very high-level task, and the actual learning outcome, which is maybe a few steps below, which is more closely related to background detection instead.

In this case. And in fact, examples of shortcut learning, of neural networks exploiting certain patterns in the training data that we don't really want them to learn about, happens also in other applications like natural language processing.

Here the task is usually called natural language understanding. The task is to look at a short paragraph of text.

In this case, it's just a paragraph about some Super Bowl. And then given this paragraph plus a question, what is the name of the quarterback who was aged 38 in a certain Super Bowl, the network has to make a prediction about the correct answer based on the paragraph and the question.

And usually everything works fine.

The original prediction, John Elway, is indeed the player who was aged 38 in the Super Bowl.

So everything is looking great, but then as soon as you add another irrelevant sentence about some other player with a, you know, completely unrelated with a jersey number 37, if you add this sentence at the end, the network's prediction changes. And this is again a striking failure, because the sentence has nothing to do with the question at all. But the network had learned that usually in this task, questions refer to something that happened at the end of a paragraph. So whoever was doing something at the end of this paragraph was just going to be outputted as the prediction here.

And this shortcut usually works well, but then it can also fail quite strikingly and reveal this discrepancy between intended and actual. Learning outcome.

Finally, just as a last example here of shortcut learning, I think there are also cases where this happens beyond machine learning, for example in biological perception, when humans think about how animals behave, for example. So in this case, researchers were training rats to navigate a complex maze.

And the only way they could be successful in this task was by distinguishing subtle color differences on the walls of the maze, like a light blue versus a slightly darker shade of blue.

And the rats were getting extremely proficient at this task, in fact much better than they should be able, given the constraints of their actual biological hardware there.

Their color perception.

So they were getting much better than they should be able to do on this task, which also sometimes can be an indicator of shortcut learning if something is unexpectedly failing or if something is unexpectedly much better than it should be, looking too good to be true. And in this case, actually, what the researchers discovered was that, well, the rats weren't actually using their eyes at all for the task.

They were just smelling the different types of paint on the walls of the maze, and then once the smell was controlled for, the rats were back to walking around randomly essentially. So here again, a discrepancy between the intended learning outcome, subtle color discrimination, and the actual learning outcome, which was more closely related to just smelling. In all of those cases, we can see shortcuts are being learned.

And in order to really pin it down to the most simple setting imaginable, or maybe like a very reduced toy example, we came up with this simple setting just to distill what's going on and really understand where this comes about. So given a very simple task, stars labeled A's, moons labeled B's, and a new test set, the task is to categorize those novel images that you haven't seen during training into A's and B's. And any 2-year-old could do this and label those images.

What is interesting here is that a standard fully connected neural network trained on this task with exactly those images in the training set and exactly those test images, it outputs BBAA, so completely the opposite prediction from what a standard, like you and me, would label those. We would still think like 2 stars followed by 2 moons.

And the reason being here is that we artificially induced a shortcut in the training data.

This means that stars were always presented either in the bottom left or in the top right corner, and for moons vice versa. And essentially this indicates that there's at least two ways now of solving this task.

You can either look at the shape, stars versus moons, or you can look at the location, bottom left versus top right, and so on. And then your network can actually just use the location here. So even if we just look at a very reduced setting, stars versus moons, there's, there's examples of what can go wrong when we just have such a simple setting but more than one way to solve the same problem.

And this revealed the discrepancy between how typically humans would categorize those images and how neural networks would categorize those images.

The crucial aspect is really that we had to go to an out-of-distribution test set to notice this difference. Out-of-distribution means that essentially the test data now comes from a somewhat different distribution than training data. The standard testing procedure is we take a large dataset and we split it at some point, either it's 50/50 or like 80/20, it doesn't matter.

But essentially we start from a big dataset and we split it somewhere, use part of it as training data, part of it as the test data, And this test data is then called IID test data because it's coming from the same distribution as the training data. The problem is that when we only look at IID test performance, IID test performance is useful, but it's not sufficient to understand what's going on because many of the correlations that are present in training data, they're also, by design, they're guaranteed to be present in the test data as well.

If cows happen to be on grass all the time during training, that's also going to be the case for IID test data. If there's a correlation with location, as in this example, that's also going to be the case for IID test data.

So we need to go this additional step and go to out-of-distribution test settings to really understand the difference between a shortcut solution and the intended solution, to be able to understand what's what's going on and really make the task harder for neural networks. Go the extra mile and maybe test them on cows on the beach instead of the standard setting or like a randomized location setting like here. Because only looking at the test set, we will still have exactly those correlations present and we wouldn't be able to tell this difference between human and neural network perception and strategies. In this case. So it's just a toy example to show that even in a simple setting this can happen, and the chances that this happens in a more complex setting are probably even higher, because then it can be infinitely many ways to solve a task in some cases. So when we just look at the set of possible decision rules, just as a conceptual framework for thinking about this, I think it's important to recognize that a subset of this set of possible decision rules is a set of training solutions.

All of those solutions are going to perform well on the training set. And then we usually know that with overfitting and so on, it's important to look at an ID test set.

However, this is not guaranteed to give us the intended solution because even with this set of solutions that performs well on the training set and on the IID test set, there's still usually many shortcut solutions that are possible. So only by narrowing it down with out-of-distribution tests, we're able to really tell apart whether a model has learned something close to the intended solution, as opposed to just a shortcut solution. And the intended solution is the solution that performs well on the training set, on an ID test set, and on relevant out-of-distribution test sets.

All right, so I think it's important to mention at this point that obviously this is a very general problem and we don't have any solution, any general purpose solution to propose. I wish we had.

I hope you have, or that we can as a field figure out something that works in general.

I think at this point we're still not there yet.

So in the absence of a general purpose solution, we came up with a set of 3 recommendations of what we can do at the moment to at least take the problem into account and then perhaps mitigate some of the pitfalls.

So first of all, I think it's important to connect the dots. We've seen examples of shortcut learning in many different areas, so hopefully recognizing that problem as being very general might also help us in coming up with a general solution, or maybe transfer something that works well in natural language processing over to computer vision.

Because if we're actually looking at the same problem, this might enable us to harness synergies.

Also, I've advocated for being very careful with results that only come from IID test performance and applying an extra layer of caution there.

Good performance on a standard dataset, if you're looking at IID test data, doesn't necessarily equal good performance on acquiring an underlying ability. Many of the tasks that we're using, like object recognition or natural language understanding, those are really high-level terms and abilities but then sometimes the solution can be found in a much lower, much more simple setting. So being careful here is certainly something that doesn't hurt. Last but not least, we think that testing out-of-distribution generalization performance is really important, and as standard benchmarking on IID data is often insufficient to detect shortcut learning.

This wraps up the first high-level part of the talk. We've seen that neural networks are prone to shortcut learning, which is a unifying pattern behind many failures.

And the question that follows now, obviously, is given that we have such many striking discrepancies between human and machine perception, often through shortcut learning, whether there's anything that we can do to narrow the human-machine gap here. And this brings us to the second project: how can we narrow the human-machine gap?

In particular, we're going to look at out-of-distribution testing.

We've seen that out-of-distribution testing is very important to tell apart shortcut solutions from intended solutions, So for the purpose of this project, we are specifically focusing on out-of-distribution distortion robustness. So applying some sort of distortion like noise to images and then checking how this affects object recognition accuracies. So in order to narrow the gap, we first need to measure the gap and really quantify it. And for this purpose, we started by collecting lots of human classification decisions in the lab.

So just humans seeing an image, whether it's a clean image or a noisy image, and then clicking on the category, like cat, dog, car, and so on.

And in terms of images, we used a set of 17 out-of-distribution datasets intended to cover a broad variety of different distortion types, both parametric transformations and also non-parametric transformations like sketches or silhouettes and so on, just to get a broad understanding of how distribution distortion robustness is affected and develops the more and more noise we apply to an image.

And having collected this set of human responses, we can then compare this to models evaluated on exactly the same images.

And in terms of models, we used both our standard deep neural networks from the Torchvision model zoo, but also a bunch of recent exciting new models and developments like self-supervised models, for example, models that cope without human-provided labels, CLIP, which is jointly trained on image and text, vision transformers, adversarially trained models, or just models trained on extremely large-scale datasets ranging from 1 million images, like standard ImageNet, all the way to 1 billion images, some of the more recent models, just to try out many different directions and see whether any of those is going to narrow the gap towards robust human perception.

And then all of this is brought together in a single comprehensive toolbox and benchmark, which we called Model vs.

Human, that's available on GitHub and hopefully enables others if they have their own model and want to test it against given responses. That's something that would be possible with this toolbox.

As a disclaimer before we start looking at the data, I think it might be important to mention that robustness as a term is very broad and general, and there are many interesting angles to studying robustness.

Here we specifically just focus on out-of-distribution distortion robustness.

Also, humans and machines can and should be compared in many different ways. We just pick robust visual object recognition here as an example, but obviously there's many, many different aspects to this comparison that are also very interesting and relevant.

And also, this overall, this can be characterized as a functional, as a behavioral approach of comparing humans and machines, which essentially means that we're going to look at input-output relationships. Given the same image, do a human observer and the model output the same prediction? This is in contrast to comparing internal representations, for example, whether the representation at an intermediate layer of a model shows similarities to some measure of brain activity at an intermediate processing stage or something.

Here we just focus on input-output relationships.

Given the same input, do we get the same response prediction? And arguably, I would say this is a level of comparison that has maybe the most direct relevance for applications, because in the end, machine learning is increasingly being applied in many different areas. And Usually users of a system, they're going to be affected by the final machine decision, not necessarily by some intermediate processing step, even though that obviously constrains the decision in some ways. But the decision at the end has the most direct practical relevance, I would say. And so it's maybe a useful level of comparison to human responses.

With that being said, let's dive right into the data and look at some results here.

So we've collected a large set of human responses, and now we can compare that to model decision-making.

The question here is simply, how does object recognition accuracy—

how fast does it drop the more and more noise we apply to images? So here in this example, you can see uniform noise being applied to an image of a biker and recording classification accuracy on the y-axis as a function of the noise strength.

We can see in the top left, obviously, no noise means high accuracies.

On the right, a lot of noise means chance-level accuracies, but in between, That's the really interesting part, how fast does recognition accuracy drop when you apply noise to images.

And comparing Human Observers in red to a standard ResNet-50 trained on standard ImageNet, a million images, we can see a substantial gap in performance.

And this gap is not just the case for this particular network, ResNet-50, but that's also a big gap for many other standard models from the TorchVision model zoo. So all of those models, they might outperform humans on noise-free images, but as soon as noise is being applied, they rapidly drop. So it seems that standard neural networks— I think this is something that's already well known— they're not super noise robust. There's still a big gap to human robust perception.

But then again, we also wanted to look at some of the more recent exciting models and see whether any of those new models is going to make progress and narrow the gap towards robust perception.

And indeed, what we found is that actually it doesn't really necessarily take a different architecture.

Many standard deep networks that are just trained on 10 times, 100 times, or maybe 1,000 times more data compared to standard 1 million images in standard ImageNet, this alone is sufficient to reach a very decent level of robustness. And it's important to mention that, you know, none of those training datasets contain exactly those types of distortions or like uniform noise.

This is really out-of-distribution generalization. And some of the more recent models are really making a lot of progress, especially models trained on large-scale datasets.

And so this is data for just a single experiment, uniform noise. I've previously mentioned that we looked at 17 different datasets, so we can also look at the overall accuracy across all of those datasets to get a more general understanding of what's going on. And here, human out-of-distribution robustness is at a fairly high level.

There's also a big gap between standard models and humans, and this gap isn't closed by adversarially trained models, for example, in blue here. They still seem to have a big gap towards human perception.

And also, this gap isn't really closed by self-supervised networks in orange.

Also, those models that learn without labels, it's a great way to get rid of the reliance on human-provided labels during training, but it doesn't seem super effective in increasing distortion robustness here. So there's still a big gap, but then again, many of the more recent models trained on large-scale datasets are narrowing and in some cases even closing or surpassing the gap to human perception. In fact, all of those models with red arrow on top, they are trained on large-scale datasets, and this seems to be one of the most important factors in making progress here, even though obviously architecture and loss function and so on and the degree of detail of model training still play an important role, but the single most important factor when looking at those results appears to be the size of the training dataset rather than anything else. All right, so I'd say that's great news in terms of being actually able at this point to have models that are reaching or even surpassing human out-of-distribution distortion robustness.

At this point, we thought, well, maybe it's a good point to take it a step further and look more closely at human versus machine decision patterns and error patterns. So out-of-distribution accuracy is still a very coarse measure.

This is just a single scalar for a large set of images. But this doesn't really tell us whether humans and models are making errors on the same images or on different images. And this is really the last aspect that we looked at, error patterns. Do humans and models actually agree which images are easy? Are they making errors on the same images on different images. And this is something that goes beyond just counting the number of failures and errors. This is hopefully going to tell us more about which models are implementing a somewhat more human-like strategy compared to other models.

So in terms of quantifying this, we're using the metric called error consistency. This is based on Cohen's kappa. I'll try to give you a rough intuition on how this works.

So error consistency is a score that is going to be zero if a model and a human observer are making completely independent errors, and it's going to be positive if a model and a human observer are making errors on the same images.

And just as an example, if we have a human with 80% accuracy on the dataset And we evaluate the model on exactly the same set of images.

And the model has now 90% accuracy on this dataset.

In order to understand whether there's actually consistency in terms of the errors that they're making, we need to compare this to how much consistency are we getting just by chance alone, even if there's human and model, if they're completely independent. And this is something that we can calculate quite easily. So modeling them as independent binomial observers, we can just assume they're independent and ask, well, how much agreement would we expect just by chance?

And this is the probability that they're both correct at the same time, 0.8 times 0.9, plus the probability that they're both wrong at the same time, which is the reverse setting, 0.2 times 0.1. So this gives us 74% agreement just by chance, and this is what we can use for calculating our consistency.

If the consistency that we actually see in our data, if that's larger than 74%, we can see that there's agreement beyond what we can expect by chance.

And error consistency would be positive.

If error consistency would be—

if the consistency that we see in data, if that's lower than 74%, error consistency would be negative. And if it's exactly 74%, would be completely independent errors, just at the level expected by chance, and error consistency would be zero.

Just to give you a rough idea of how this score works.

That we're going to see now.

So now we can use this to compute consistency between any two decision makers, for example, between humans and other humans, or between models and humans. Human-to-human consistency across those datasets is at a, again, at a fairly high level.

Humans essentially agree which images are easy and which images are difficult. When it comes to standard models versus humans, we can see that it's on one hand clearly beyond chance, it's not zero, it's positive, so there's a certain degree of consistency. But then again, there's still a wide gap. And interestingly, some of the more recent models again make progress towards narrowing this gap towards human-to-human consistency. Again, many of those models are trained on large-scale datasets, the models with a red arrow on top, and they're really getting closer towards human-to-human consistency.

The best model here in this case was CLIP, which is jointly trained on images and text and also on a large-scale dataset.

But even the best model, as you can see, still has some way to go left. So in a certain way, humans and models increasingly agree which images are easy, but not as much as humans would agree with other humans. And zooming out a little, we started with this question of which approach would narrow the human-machine gap.

And conceptually, you know, these axes here don't actually correspond to anything. This is just a conceptual figure here. Standard CNNs, they're neither particularly human-like nor are they particularly robust.

On the other hand, there's a great deal of recent exciting new models, and we wanted to find out whether any of those directions would narrow the gap towards human-like models towards robust models, as robust as human perception, hopefully.

And what we've seen is that perhaps, at least contrary to my expectation, some of the models that we tested, like adversarially trained models or self-supervised models, they don't really come with that much of a benefit on the axis. On the other hand, CLIP and vision transformers are much more robust, also a bit more human-like, and especially training on large-scale datasets seems to be the single most important factor when it comes to making progress in this direction. And I think in terms of what this means, or as a personal interpretation, I think On one hand, I'm really excited about those results and see that the gap is finally narrowing down.

For a long time, there was a big, big robustness gap between humans and models. And now the best models are outperforming humans, which is great, I think.

And that's going to be very, very useful. On the other hand, I'd say there's also a certain deal of curiosity.

What's going on with those large-scale datasets?

Why are they so effective and perhaps more effective sometimes than exchanging the architecture? And finally, I think there's also a certain degree of disappointment when looking at those results and maybe a bitter lesson to be learned from this or think about because many of those directions that were actually successful, like for example the size of the training dataset, they're successful because of scale and computation, not really successful because of maybe like human ingenuity or some neuroscience-inspired layer that provides much better robustness or divisive normalization. People have tried many different aspects, but really scale and the size of the training dataset is perhaps in this case really the most crucial factor. And this kind of reminds me of Rich Sutton's Bitter Lesson that I'm sure many of you heard of this lesson before.

Rich Sutton argued that general methods that leverage computation they're ultimately the most effective and by a large margin. And building in how we think we think might not work in the long run.

And I think this is certainly something to think about. At least I'm thinking about this question quite a bit. And this kind of raises the question of whether robustness it's essentially just going to be a function of a very simple scale factor like dataset size. Whether there might be such a relationship, out-of-distribution test error is related to dataset size with a scaling law.

This might be something to perhaps watch out for the future. I'd be interested to see it like a very controlled study, perhaps the same network trained at multiple orders of magnitude, more and more data, to see how this develops and whether that's a general pattern. Or whether perhaps also we would be able to beat the bitter lesson and come up with a really smart solution that doesn't require just pouring more and more data on top of models. That would also be fantastic, I think.

And with that, I'd like to just wrap up. We've seen in the first part that comparing neural networks to human decision-making, that your networks are prone to shortcut learning, which might be seen as a unifying pattern behind many failures.

However, recent models, especially models trained on large-scale datasets, narrowing the gap towards robust visual perception.

And with that, I would like to thank you all for your attention and for staying with me and working through the Zoom issues and everything. And I look forward to any questions you might have.

Yeah, thanks so much, Robert, for these interesting insights into your work. I would start with one question. I've been wondering, so the definition of these shortcuts in contrast to, for example, overfitting or other relying on spurious correlations, if you want to frame it in a general way, is that they are only detectable on this OOD test data, right? In contrast to overfitting, which would be detectable on IID test data already or other spurious correlations. And I'm struggling with this a little bit because it relies on this binary discretization between IID and OOD, which in reality is often maybe not so useful because you only have a very limited set of test samples. And yeah, then it's a bit artificial to say, okay, this sample is still a data variation that we want to be robust against, and that is basically part of the machine learning generalization. Data variations to just generalize across data in distribution versus this sample. We call it now OOD because it crossed the line, you know. And do you— well, what's your perspective on this? Is that problematic, this discretization? I mean, also for when we think about what's the secret of big data set, you know, maybe this is related that the larger data set you have, then again, also the concept of IID expands because you have more sample of your distribution and you will find samples that are more outside, or this artificial line shifts around, I don't know, and you have samples that are much more similar to these OOD cases.

Yeah, that's a great point. I think in many settings it makes sense to distinguish between overfitting, which is revealed by a discrepancy between just training and IID test accuracy, and shortcut learning on the other hand, which is characterized by good training, good IID performance, but then an unexpected failure in some other setting. But I completely see your point that in some cases maybe there's not a discrete boundary where you can say this is IID and at this point we're on completely the other side. Maybe it's more of a, like, can be thought of as a continuous version of this, where in some cases it's really clear that this is out of distribution, in some cases it's really clear that it is in distribution, but then also there's a lot that happens in between. And this is where all the, yeah, but there's a all the trouble comes in.

Yeah, in terms of, you know, is this still IID, is it OOD?

But then I guess it depends a bit on the application, whether we can find examples that are clearly out of distribution, or whether we should perhaps be looking at something more in the middle.

I think, yeah, that's it. Like, and do you think this is related to also your last observation? I mean, did you If you have this continuous OOD scores, let's say, did you also look at if maybe the gap continues to be large even though you have a larger IID training set for the very OOD cases? Like, is the gap overcome with more training data only in the near-OOD cases? Like, did you look at this OOD kind of continuous scale when looking at where more data helps?

Right, so I think it's hard to measure really. We don't have a good measure of how close is this dataset to a different dataset. We did look at some of the models trained on large-scale datasets because we were a bit suspicious obviously large-scale datasets scraped from the web contain maybe a lot more variation than ImageNet.

So for example, aspects like blurry images are likely to occur somewhere in the map.

But we also included in the benchmark a set of distortions that were specifically designed for research purposes, like either long distortions, for example. Those were distortions that are created with a specifically like recent research purpose toolbox, and they're not occurring anywhere on the web. And I think the positive finding was that models that were good on, for example, blurry images, large-scale models, they were like good on images that are likely to occur on the web, they were also consistently better on those distortions that are not included in any of the web scraped datasets. So in this case, I think something that the models are learning appears to be quite general.

Okay, thanks.

Are there more questions? I saw Alexandra raising their hand.

Yes, thank you. Nice meeting you, Robert, in the seminar. Thanks for the talk. I wonder if like this out-of-distribution issue, like deciding if something is in distribution or out of distribution, maybe relies on— I mean, there must be some dimensions on what does it mean to be out of distribution. You will have different dimensions. So by learning and analyzing different tasks where things go wrong, you actually start recognizing what goes wrong. With these mice, the smell is the case. And with the label on the like X-rays, it's the label. So is there like, are you currently working on kind of a guideline on collecting things that are dimensions one should check depending on the task you have? Like where could be things go wrong?

Yeah, that's a great point. So I wish there was such a set of guidelines that we would have. I'm not like currently working on this, but I think it would be great to have such a set that we could, like a checklist that we could go through. Unfortunately, it's my impression that neural networks are usually one step ahead of us. Like they are very good at finding intricate smart solutions that we haven't really anticipated. So perhaps my worry would be that if we have a set of things to check, that they would find like yet another solution that we haven't anticipated. But for practical purposes, well, like a set of standard things to check could still be very useful, I think. And that's, yeah, I think.

Yeah, I think to be also ahead of the neural networks at one point, maybe we should all work together. Cause I like, I work with CT scans and I've actually done like an outer distribution check by giving it not a head and neck cancer patient, but a healthy pelvis patient. And I recognize like my segmentation interpretation is also my human interpretation. Like I say, okay, it's orientating towards the spine. But maybe it does not because the network sees something different than I do. And I think collecting examples where things go wrong or right, that would— might help building that image in the end, finding what networks actually interpret.

Thanks.

Right. Okay. Hey, thanks for your nice presentation. We at DKFZ are working mainly in the medical domain, and the majority of your research, of course, is based in the natural imaging domain. And I've been wondering if you believe that, because you're the expert, how it would generalize or translate to a different setting, regarding to maybe shape biases. Because personally, I was wondering if one could translate some experiments done to like a medical setting where it's mostly focused on detection and localization of objects, and how like the change in the domain would like influence current findings of yours.

Right, um, so I'm not an expert on medical aspects, obviously. Um, I think From what I've seen, sort of the classification and detection seem to have very similar patterns. So for example, if you think about like, I don't know, texture biases in classification, you can make the task harder for neural networks by using like a random texture on top of the image.. And that makes it harder for classification networks. And we've seen exactly the same patterns also for object detection.

If you just apply a random texture where the shapes are still the same, that makes it harder for the neural networks. So they seem to be exploiting some of that information.

I think in general, But that's like a more general answer. I think for a more specific answer, I would probably need to have more knowledge on the exact problems and how this would pan out. I think usually trying as hard as one can to come up with something that's out of distribution can be helpful in many cases, like whether it's a data augmentation, just applying augmentations on top of images, or whether it's maybe something more subtle like recording data from a different scanner, a different machine, to see whether it's machine-specific what the model learns.

There might be perhaps ways to approach this, but Yeah, I don't know.

Do you have ideas on what could be helpful?

I think the most challenging task is collecting the data in the first place because unfortunately, we cannot have easy access to human subjects because usually you need radiologists or experts to collect the data. Therefore, it's really hard to reproduce or emulate a setting like this. And also looking at the setup of the psychophysical experiments you do, it's kind of difficult to, I think, go to detection because scrolling through scans is part of the job there. And then giving short— The translation seems like to have some problems, but maybe we'll find a nice way.

Right. Thank you. I imagine it's also difficult to collect datasets large enough to have such a large-scale setting. In many cases, there's probably going to be much smaller datasets.

Yeah, definitely.

So yeah, thanks. So we have one maybe last question from the chat. So Marco's asking about the human inconsistency. What's your feeling about this? He says in your 2020 paper, there was still a lot of inconsistency between humans. Do you think they're still very inconsistent, surprisingly, or—

Right, I think it depends a bit on the dataset. Like on some datasets, the human-to-human consistency is like extremely high. On some datasets, it's closer to chance level. We're actually using the same, like some of the same human data as well. In addition to new data. I think the sort of the variance within a single dataset wasn't super large. So if you go to, I don't know, silhouettes, human-to-human consistency hovers around a certain number. And if you go to a different dataset like, I don't know, style transfer images, it might be a different value.

But then again, the variance within a single dataset seems to be smaller than the variance across datasets.

So this might indicate that it has something to do with the dataset.

That would be my answer.

Yeah, I think that makes intuitive sense. I hope that answers your question, Marco. All right, so yeah, let's, let's thank Robert again for his time and for visiting us here at DKFC. And I hand over to Doreen.

Maybe you want to close the meeting. Yeah, so yeah, thank you everybody. Thank you, Robert, for being here, for getting over these technical problems, but in the end we managed it perfectly. So thanks everyone for watching and listening, and the record will be available after the talk, it seems, till the end of the week, available on our YouTube channel. So yeah, if there are further questions, please visit our website and you can then contact Robert directly. So yeah, thanks everyone, goodbye!

Bye everyone, have a nice day!