Scientist Stories: Jenia Jitsev, Transfer Learning with Large Scale Models

Welcome to the Data Science Seminar. It's my great pleasure to welcome our speaker today, Jenya Jitsev from Jülich Supercomputing Center.

Jenya Jitsev.

A warm welcome also to Ralf Floker, who will take over the introduction.

Oh, thank you very much for the kind words, Doreen, and great to have you here, Jenya. Jenya comes from the research center Jülich and working there in the division of federated systems and data, heading the group of cross-sectional deep learning. And our first encounter we both had, for example, was for a project pitch in last year's summer for the pandemic in Helmholtz. It certainly did not work out with the pitch, but it was very interesting to work with Daniel and we see a lot of synergy and potential as he's working on very interesting topics in the context of maybe unsupervised learning, generative models, also things like one-shot learning or transfer learning, how to make it most efficiently if you want to conduct and apply research in adaptive multi-layer neural networks. And so we directly had the feeling there's a lot of music into it and synergies between the work he does and things that are done, for example, in the German Research Center, which is more applied. So I'm very happy that we could arrange then also a presentation of yours, and I'm really looking forward to. And so without further ado, I leave the stage now because I'm curious what you have to show. Thank you very much for being here, Dania.

Okay, thanks a lot for kind of introduction. Yeah, I cannot promise to play any music, but I would like to share the screen. This is okay now, right?

So let's see.

Yes, that works.

That hopefully works fine.

Okay, so, uh, hello, welcome everybody. Thanks a lot for inviting me and for joining in for this talk. In the talk, I would like to elaborate on systematic improvements that pre-training of large network models on large datasets may generate if transferring those large pre-trained models across different and eventually domain-specific datasets of much smaller size than those used during large-scale pre-training. And before we turn to transfer learning, a few remarks about the non-transferable learning which was and still arguably is quite a large fraction of machine learning studied and exercised in practice. There we have, as depicted here, for distinct datasets that may belong to different domains D1, D2, and D3 with corresponding tasks to solve D1, D2, and D3, each a separate model M1, M2, and M3 being trained on respective dataset task from scratch, providing then solutions only in the frame of a particular scenario the training was carried out in. So the strong assumption reassuring here model quality on unseen data is that training and tests are coming from roughly the same data distribution. This limits model reuse across distinct datasets strongly, so that for each new dataset and task, an entirely new model has to be then trained from scratch again, which is of course resource-consuming and slow.

Especially for deep learning where data has to be large enough to obtain a model of a decent quality.

So even more importantly, for those cases where you have only few data available for training, there is often just no way to train a good model from data when it's scarce, right? And here where the transfer learning enters the stage and puts forward methods that allow to pre-train one model on a particular large, rather generic dataset, the collection of those such that The pre-trained model can be then taken as a blueprint and used for efficient transfer to a broad range of different downstream datasets and tasks. Efficient means that it can be better on the downstream dataset and task if compared to training same model there from scratch without pre-training. Better being in terms of faster learning convergence, or in terms of data efficiency requiring less data to train, or just having better final performance than training from scratch would deliver. So importantly, Those methods allow to reuse model representations obtained during pre-training in such a way that it becomes possible to train a good model even for those problems where available data is scarce.

So classical transfer learning is still though a restricted setting, like a restricted frame, restricted setting.

As for each target domain, you end up with different models depicted here with M12, M13, originating from the common pretrained model M1 following retraining or fine-tuning, and those models do only care about performance on their respective target setting. Ideally, and this is what arguably everyone in machine learning community is dreaming of, one would like to have this kind of universal learner which digests incoming datasets and continually updates one single unified generic model that is also able to cope with those new incoming datasets and tasks in efficient manner without losing the ability to perform well on all the tasks seen before. So one single model for all, and sure enough, it's known in the community how hard it is to get to a learning system like this. So right now, this way more simple post-transfer learning is able to provide some initial segments of long shaking bridge towards that kind of a generic self-organizing and transferable learning. So let's have a look on the usual transfer methodology employed in deep learning nowadays. The usual scenario is we have two domains, the source and the targets.

So pre-training is done first on the large generic source dataset D1, like for instance ImageNet.

Network M1 is randomly initialized for pre-training. So training done from scratch, and source dataset being large makes this procedure slow. So the pretrained model M1 is then taken to be retrained or fine-tuned on a usually smaller dataset D2, which can be a much faster procedure, not only due to a smaller dataset size, but also sometimes due to using only a small fraction of parameters as compared to the whole pretrained network.

For instance, in extreme case, when freezing all of it and just fine-tuning topmost linear classification layer of the network, or when using a few-shot transfer methods where only a few examples per class are provided during transfer.

So this fine-tuned network model can be then compared in terms of different performance aspects on transfer to a network that is trained from scratch directly on the target dataset. So in the pre-training phase, which can be understood just as a training in conventional sense.

We can, on the one hand, execute different forms of learning that rely either on labeled source dataset, like still most widespread supervised case, or using a fraction of labels while dealing with large unlabeled parts, like in semi-supervised case, or entirely working without external labels by doing unsupervised learning that can again have different forms. For instance, self-supervised where pseudo labels can be generated automatically from the available source data. And these different learning forms can be used either within a standard single-level optimization, which is still most often the case, or they can be also part of a bi-level optimization scheme with inner and outer optimization loops, for instance, like in case of meta-learning approaches to transfer.

So once pre-trained, the network model with pre-trained weights is exposed to a target dataset and task where fine-tuning is performed. For the fine-tuning, there are gradations to which extent the network stays trainable.

So it can be anything in between freezing the whole network and training only final task-specific layer on the top of it, for instance, linear classification head, or as another extreme, doing full fine-tuning involving all the network weights. When doing fine-tuning, hyperparameters like learning rate schedules and such are usually different from pre-training. And target data is often smaller than the pre-training data source. And in so-called few-shot transfer, in addition, there is only a few number of examples on the few per class unsupervised case that is shown during fine-tuning for transfer.

For fine-tuning pre-trained model, we just run the usual gradient descent optimization based on loss defined on target task starting from pre-trained weights indicated here with theta, a part of which may be frozen and not subject to further change. So fine-tuning a pre-trained model during transfer is conceptually just training in usual sense. We just start not from random initialized weights but from pre-trained ones. So this is the overall methodology for transfer learning.

Here, an example where source is something like ImageNet and target datasets can be X-ray chest medical imaging.

The task can be in both cases classification, and we're interested in whether the model pre-trained on source gives us any benefits on target task if compared to the same model randomly initialized and trained on target dataset directly from scratch.

So transfer learning of this kind was applied already in the very early days when deep neural networks were rising using architectures like Overfit or VGG dating all the way back to 2013, '14.

So those networks were pre-trained on ImageNet and then taken as off-the-shelf generic feature reservoirs that could be reused in a multitude of different downstream tasks, mostly by freezing the whole network and retraining on its top, for example, a simple linear support vector machine classifier.

And it turned out that across all different tasks, those transferred pre-trained generic models were as good or even better than specialized state-of-the-art back then, solutions depicted here in violet color in the bars that were based on carefully hand-engineered processing pipelines and features as a strong constraint to off-the-shelf generic convolutional network obtained by learning from image data.

Okay, so since then, as all the non-deep learning-based methods were wiped out from the field, transfer learning performance is no longer assessed by comparing deep nets to other methods but by comparing deep nets to themselves.

To the trained-from-scratch versions. And there are networks like ResNet-50 that serve very often as a standalone standard backbone architecture for pre-training a model for transfer. Meanwhile, there is plenty of other choices for a network model backbone for pre-training, like for instance more recent EfficientNet variants as shown on the right here,, and this choice is usually driven by the performance networks show on standard ImageNet training and by its computational cost. So for systematically evaluating transfer performance, there are benchmark datasets for transfer learning that span a variety of domains, like for instance Visual Task Adaptation Benchmark here, VTAB, that contains different datasets that also belong to standards in machine vision like CIFAR-10 or CIFAR-100, also subdividing the domains here into 3 groups: natural, specialized, and structured, loosely corresponding to domain type.

With such transfer learning benchmarking datasets, it becomes then possible to make somewhat systematic evaluation and compare transfer learning across different conditions.

Like for instance shown here, where VTAB is used to check average performance on each of those different domain groups when either doing supervised training from scratch directly on target dataset or using transfer learning with different types of pre-training as indicated here. So this provides here some evidence that transfer learning with either supervised, semi-supervised, or unsupervised pre-trained model and fully fine-tuned then on the target outperforms, especially on natural domain supervised training from scratch in both low and high data regime, where either a fraction of 1,000 here or all available samples are used during transfer. So here, another work that takes object detection and semantic segmentation tasks to test transfer learning benefits. Using again a vast collection of different datasets as targets. And here one special move is that instead of pre-training on ImageNet only like it is usually done, each of those datasets can also serve as a source for pre-training, with the rest being targets for transfer. And in this way, one can also assess differences in transfer learning when using different source data for pre-training. So in this score matrix, we see then the model performance after transfer on each dataset when using different source datasets for pre-training relative to the transfer learning performance with model pre-trained on ImageNet. So in some cases, we see that pre-training with a different source can be better than pre-training on ImageNet, as indicated here in green, and in some cases it can be worse, indicated here in red. But what is very clearly seen then in the rightmost column that training from scratch without any pre-training, without any transfer directly on target datasets, is always worse than doing transfer learning with an ImageNet pre-trained model. So again, evidence states that it's worth doing pre-training and transfer instead of just training from scratch. And this evidence, though, we saw comes from scenarios where source data and target data are still somewhat of similar kind, being similarly distributed, of similar domain. What is about the cases where target is quite different from source, like here in the depicted case of pre-training on natural images, ImageNet again, and transferring to different medical imaging datasets? And so this work here, for instance, compared performance on medical imaging datasets like Retina and Checkspert by either training there from scratch or by taking ImageNet pre-trained model for transfer. So in there we see for Retina it shows no benefit for transfer for any of the studied networks, and actually also indicates that smaller networks like mentioned here with CBR can be almost as good as their larger counterparts when trained from scratch.

For CheckSpert dataset, across different classification tasks, there is like mixed results, but also seemingly no clear significant benefit for transfer learning as compared to training from scratch.

And CBR smaller networks again seem to perform on par with larger ones. Yet another study had a look on CheckSpert as a transfer target while pre-training on ImageNet And here, when looking on average performance across many different network architectures used as backbone during pre-training, there is some evidence for better performance when using transfer learning compared to training from scratch again. However, its significance, well, it's quite modest, small, right?

So while this gives quite inconclusive overall picture, there is often one interesting point that reoccurs in such studies but is often overlooked, which is while often where no clear benefits can be found in full data regime during transfer, using only small fraction of data creates interesting phenomena showing significant improvement for transfer learning, but only when taking larger network models like seen here for ResNet-50 opposed to smaller CBR networks when transferring only a small fraction of retina dataset using 5,000 samples than we saw before.

So there is the point where we have to talk explicitly about model and data size used during pre-training. Common characteristic of transfer learning studies we saw is that the largest source dataset they usually use is ImageNet-1K with about 1.4 million images inside, and utilized network models like ResNets are mostly of modest size. And this is in strong contrast to what happens, for instance, and language modeling, where very large transformer networks are trained in self-supervised manner as auto-regressive generative models on very large datasets of text. So between language and vision domains, there are 3-4 orders of magnitude in model and data size used during pre-training, a big difference that is in addition to difference between supervised and unsupervised scheme utilized for learning.

So some concrete numbers here for contrast.

Large language models like GPT-3 family may contain billions or even 100 billion, so meanwhile trillions actually, of parameters. While still recently, well, still dominant convolutional architectures, although transformers are also coming there for vision, have around a few hundred million weights or even less, like in case of EfficientNet. And while ImageNet-1K that is still widely used for pre-training and transfer has around this 1.4 million image samples, the text corpora used for language modeling contain tens of billions of text sentences. Keeping in mind this strong contrast that still exists between language and vision domains in model and data size used for training, we may question whether differences observed in performance also in transfer learning may have to do something with it. So, and there is a large number of works coming out from language modeling that provides very strong evidence, as presented here as one example, that model size has very strong influence on transfer performance observed, especially when performing this kind of zero or few-shot transfer in very low data regime. So here we see that there is a huge difference in accuracy measured on transfer language modeling task after self-supervised pre-training of different sized GPT models, with performance strongly increasing when increasing model size. So in general, this line of works from language modeling suggests that there is strong improvement of model capability to generalize, as measured in decrease of loss on test dataset, when increasing model size or data size during training. As shown here on the plot on the left, steady reduction in test loss is possible when increasing a model size, however only when provided with a dataset large enough.

And similar phenomena emerges when increasing dataset size.

Here on the right, test loss decreases steadily only when using a large enough model size for training.

And it becomes here clear that you lose a lot of potential test loss performance when limiting yourself with either too small model or too small data.

In the pre-training. Here on the right, also some evidence provided that along with generalization, transfer performance improves as well when continuously increasing model size, if fine-tuning then on different target datasets.

So when measuring test loss systematically across different size conditions during training, like shown here, There seemed to be quite a generic story that poses scaling laws of power law form for test loss depending on compute time invested during training, data size, model size.

It showed that increasing all of them hand in hand without creating bottlenecks in either of those will result in steady improvement of generalization ability, holding across many orders of magnitude, with test loss going further down without any hints for saturation. So given sufficient compute budget, increasing both model size and data size is the way to strongly boost generalization, which also explains tremendous success of such huge pre-trained language models like GPT-3 when transferring on very different downstream target datasets. And as shown on the plot below, this generalization boost relies mostly on increasing model size followed by an increase in data size for training, which looks the modest in comparison to necessary substantial increase in the model capacity if we are aiming to further decrease the test loss and boost generalization.

So now, according to conventional view in machine learning theory, increasing model size further and further was always considered to be a very bad idea, leading to overfitting, poor generalization, and high error on the test set.

Since recently, this conventional view, although, has been radically revised, indicating that error on the test set has a behavior described by this kind of a double descent curve. In this revised view, increasing model size further and further leads to a transition in test error behavior such that with ever-increasing model size, test error goes further and further down in the so-called over-parameterized regime after passing interpolation threshold where training error drops to zero and stays there. So according to this view, making model larger and larger should make generalization stronger and stronger. So such double descent curves showing test error going down when making a model very large were obtained meanwhile not only for large deep neural nets but also for a plethora of different also classical machine learning approaches like kernel machines, decision trees, ensemble methods.

Here though, an example with a deep neural net, ResNet-18, trained on CIFAR-10, where increasing model size by increasing the width in the network leads to the test error going further and further down, while training error, shown here on the right, also saturates at zero and doesn't show any further sign of change, as opposed to ongoing changes in the test error.

This revised view on large models being good for generalization was not really surprising for deep learning on image data as this is what was already usually observed when training large deep neural nets in the community. So here, a rather older standard example with ResNets and DenseNets, where increasing in model size and compute during training clearly also leads to improvement on top-1 classification test error on standard ImageNet-1K. Beyond improving generalization on test set There is also evidence obtained for image domain that larger models pre-trained on larger source data also improve transfer to different target datasets, as shown here by this work known as BigTransfer. So plots here show experiments where network models of different size, indicated by circles, were pre-trained on datasets of different size, here along x-axis being standard ImageNet-1K the larger ImageNet-21K, and the very large GFT-300, and then transferred on different target datasets being again ImageNet-1K, Oxford Pets, and CIFAR-100 by fine-tuning pre-trained models and measuring accuracy performance on the targets. So what can be seen here is that while comparatively modest-sized models like ResNet do not show consistent improvement on transfer when increasing data set size during pre-training, the two largest tested networks consistently become better and better in transferring when increasing the data size in the pre-training. So for larger models, also transfer performance goes up if you feed them with more and more data during pre-training, while smaller models do not show this consistent transfer improvement. And this kind of pattern was also observed in different further works like this one here that again compares different, well, transfer across different target datasets after pre-training with different size networks, ordered here on the x-axis from small to large on ImageNet-1K or 21K, with consistent outcome showing that increasing either dataset size or model size during pre-training improves transfer performance on the target. There is also evidence that few-shot transfer learning is strongly improved when model and data size during pre-training are increased, improvement being especially pronounced when both model and data size increase hand in hand, as for instance shown here. Pre-training on smaller ImageNet-1K, indicated in green, may not consistently improve few-shot transfer when increasing the model size on x-axis. But if source dataset for pre-training is large, like GFT in red, then growing model always helps to further improve performance on transfer. So same another way around, taking a small model like ResNet-50 for pre-training may not provide benefits if growing dataset size, but taking a large model results in transfer improvements when increasing size of pre-training data. Clearly, so this has strong resemblance with what we saw from scaling laws for language modeling, where generalization and transfer benefit from larger model and larger data sizes used during pre-training. One important remark here is that it is essential to train long enough when increasing model and dataset size. Spending too short time for training may actually create the impression that models pre-trained on larger data are worse on transfer than those pre-trained with smaller data. As indicated here on the left, our plots, the difference between a smaller red circle and the larger blue one standing for models pre-trained with either ImageNet 1K or 21K. So only if you train long enough you indeed get to see the transfer improvements also shown in the table below. And in the case here, even after training for as long as 8 single GPU weeks with a large network, improvements may still not be visible yet. So this makes training experiments like those very compute intensive and also uncertain. So one full training run with a large model on ImageNet-21k, for instance, requires still around 81 hours, even if using large amount of 256 GPUs for pre-training. So until now, we saw results based on supervised forms in pre-training, and a large fraction of deep learning in image domain is still done in supervised way using data with labels. So most transfer learning studies are also dealing rather with supervised learning. However, This is changing rapidly since some time, and different types of unsupervised learning are used for training on large amounts of unlabeled image data, creating, for instance, generative models that can produce very realistic image samples from underlying data distribution. So these are actually generated samples here, indicating that representations that they learn may be also very useful for transfer. Many of those methods are termed self-supervised because in their laws they define kind of auxiliary tasks that work with something that might be called pseudo-labels during training. So such methods as generative adversarial networks or variational autoencoders or hybrid of those or non-generative models that use contrastive losses all rely on doing learning by creating pseudo-labels from available data in some automatic generic way, mostly independent of the nature of particular dataset the training is done on. And here the strong hope is that the representations created by those models can be useful for transferring a broad range of downstream tasks and have even more generic character than those from models pre-trained in a supervised way. So one example can be using a generative model like a BigGAN depicted here that contains an encoder part which can be used after unsupervised training as a pre-trained model for transfer learning downstream. Another example is taking base encoder f from a non-generative network model that pre-trains by self-supervised learning with contrastive loss, and this method is quite often used in transfer. So I have a closer look on that in the following. And the particular work called SimCLR, we can do training without labels on any image dataset by using a loss for image pairs that is minimized when two transformed versions of the same image get assigned high similarity, while two transformed versions of different images get assigned low similarity in a pair. Similarity measure in this case can be just simple cosine similarity measure. So network takes pairs of transformed images, um, uses Bayesian coder f to generate representations h that are mapped further by projection head g to vector, um, to feature vector z, and similarities are computed then based on those. And enter the cross-entropy loss. So a simple scheme that turned out to be powerful in creating generic representations in self-supervised manner. So after training, we can take Bayesian encoder f and use it as a pre-trained model for further fine-tuning and transfer.

This is what was exercised in the work by Zhang et al., simpler work again, where after self-supervised Pre-training with contrastive loss without labels.

The pre-trained model was fine-tuned on ImageNet-1K in supervised way using only linear classifier on the top of the convolutional base encoder. And also here, evidence was found, shown on the right, that performance on target task improves when increasing the model size during unsupervised pre-training. And also here in self-supervised pre-training, there is again evidence that increasing model size improves performance, especially when looking in low data transfer regimes. So when using different fractions of labels here, taking 1%, 10%, or all labels available during fine-tuning on ImageNet-1K, the larger the pre-trained model size, the larger we see the improvement, especially when using very small fraction of labels when compared to standard supervised training from scratch. So, and again, to conduct such pre-training experiments, computational demand is very high. For a single pre-training of this kind, you need 128 of last-generation specialized TPUs running for 1.5 hours if taking just the smallest ResNet-50 as a backbone for pre-training. And this computational cost can be though even much higher for other unsupervised methods used for transfer, like for instance the self-training approach taken in the recent work by Xi et al. So there you have a teacher network first pre-trained and supervised on ImageNet-1K, and teacher network is used to generate pseudo-labels on much larger unlabeled dataset like GFT or YFCC100. And those pseudo labels are taken then to train a new larger student network on the combined data, and this distillation procedure is iterated again and again. So the resulting large pre-trained model can be then reused for transfer with performance shown here for fine-tuning on ImageNet-1K again. So in here, just for single pre-training using a large model on very large GFT dataset, you need 6 days using the incredible number of 2048 TPUs. All right, so, um, so it seems that increasing model and dataset size during pre-training shows transfer improvement. However, most of the studies again look at transfer between similar domains that contain natural images mostly. Question arises then, would we find hints for similar kind of scaling laws if we were to perform transfer on domains are less similar to pre-trained source. Again, a specific scenario of source being natural images and a target being medical images is of high practical interest, and some very recent studies make first attempts to look into it on large scale, like the work here by Mustafa et al. from Google Health Lab. They use procedures and networks models from Big Transfer paper we saw before, so I pre-trained different sized ResNets on ImageNet-1K, ImageNet-21K, and GFT, and look at transfer performance on 3 medical target sets coming from mammography, dermatology, and in X-ray chest imaging. So again, checkspurt. For transfer experiments, the outcome seems to indicate that taking a larger model and larger dataset size for pre-training again somewhat improves transfer performance across all the sets. Especially when comparing the smallest and largest sizes. The improvement is clearly there, and transfer always outperforms training from scratch using standard ResNet-50 as a baseline, indicated by the dashed line here. However, difference in performance shown across pre-training conditions is sometimes rather small. So if you look here on the result in the mammography, comparing smallest network pre-trained on smallest data and largest network pre-trained on largest data much larger, so it's hardly a 0.5% apart. So it's not so clearly pronounced as in case of intradomain transfer we had before. Also, zero-shot transfer was examined in the setting. A pre-trained network is not allowed to fine-tune at all on the targets. And here, pre-training with larger models and larger data also hints some benefits, and some cases improvement strong. Like here on the chest X-ray transfer, where when comparing, for instance, small ResNet-50 with the largest network, and in some cases, like again in mammography, it's rather small when, for instance, here dataset size is increased for the largest network. There is hardly a benefit. Transfer performance was also assessed here in terms of data efficiency, measuring what fraction of available data is needed to reach a certain performance level. And we see that in most cases there is a trend to use less data for transfer if pre-training with a larger model on larger data, with some inconsistencies in between. So while there is some evidence again showing up that large models and large data lead to transfer improvement, it is less so clearly pronounced as for the intradomain transfer cases we had before. So from all these studies, it becomes clear that there is a certain lack of systematic approach to test transfer learning in the same manner how it was demonstrated for language modeling and scaling law cases we saw, namely by varying model size, data size, compute time for training, and eventually also by controlling other relevant factors like alignment between the source and target domain, testing influence of different learning forms, of different data regimes being low or high during transfer, amount of fine-tuning applied. And so all of this calls for a frame to systematically study transfer performance and its dependence on pre-training and fine-tuning conditions. This then may allow us to create such insightful plots we saw before from language modeling community that would provide hints on how generalization behaves dependent, for instance, on model and data size used during pre-training, and whether the improvements seen for generalization also appear, then in different transfer scenarios, eventually leading to our postulation of scaling laws for generalization and transfer that are valid for image domain. So as one modest step in this direction of studying large-scale transfer learning systematically, we framed the so-called COVIDNetX initiative, in frame of which we also met together, me and Ralph. Where we'd like to study all those effects of varying model data size and other factors I mentioned before on transfer performance in a specific frame given by a real-world problem of detecting COVID-19 relevant patterns and possibly predicting vital patient states from conventional X-ray chest images. So here pre-training is meant to be done on large source data being either natural or X-ray medical images or combinations of both, while varying model network and dataset size, type of learning during pre-training, and network architectures. And transfer is attempted on different target datasets that can be also natural or medical imaging sets of smaller size, where one particular dataset, COVID-X, contains X-ray chest images of COVID-19 patients to probe transfer learning also there. So COVID-X is an openly available dataset that has around 2,300 X-ray images of COVID-19 positive patients. It is composed by the efforts of Canadian-based labs from Montreal and Waterloo from different sources across different facilities. And so the idea is then in pre-training we can go quite large scale on natural image datasets using ImageNet-21K and larger ones like YFCC-100 that can be also used for self-supervised pre-training. And we can introduce systematic gradual variations in dataset size by choosing just subsets of those to carefully study influence of dataset size variation on transfer performance. So other obvious Targets for transfer are larger X-ray chest datasets like Checkspert or Mimic-CXR that contain different non-COVID-related pneumonia cases, and these datasets can be also used during pre-training alone or in combination with large natural image data to study impact on intra- or inter-domain transfer in such a setting. So current status of the COVIDnetX initiative is that it is in ramping up phase after being initiated at GSC back in April 2020 by myself from cross-sectional team deep learning and by Mehdi Chaherty from Helmholtz AI high-level support team. We intended this to become a frame for studying the kind of large-scale transfer learning where pre-training large models on large data is required. And hope that insights we win from those experiments will be also relevant for using transfer learning on other pathologies with more clinical relevance indeed. So also preparing grounds for quickly adaptable diagnostic aid tools in case of future pandemic events with yet unknown pathogens. So we hope to get collaborators on this track from different disciplines, and to make it happen even easier, We created a kind of a spin-off challenge hosted on Jülich Data Challenges platform, which is a collaborative data science platform driven by Research Center Jülich and local Helmholtz AI that also sets up hackathons. First one we already had this spring, and the hope is that besides attracting people, this challenge will foster a reproducible, easily extendable code base for performing such experiments, ensuring better collaboration and encouraging further development. As hinted many, many times during the talk, especially the large-scale pre-training procedure is computationally very demanding, and therefore we have no way but to make use of distributed training on supercomputers to speed up the experiment cycles for such studies. And luckily we do have this new machine at hand at JSC that is a supercomputer called JUWELS Booster. Tailored for large-scale deep learning and equipped with a fair amount of about 3,700 of A100 NVIDIA GPUs, which made this machine being number 1 in Europe in November 2020 where it was actually installed. So we got compute time of 1.9 million core hours granted for the first phase of the initiative by GALS supercomputing center organization and plan extension soon in August where collaborators are very welcome to join in. Just talk to us timely if there is interest. So using this kind of machines allows us to do computationally intensive model pre-training across a large number of GPUs, and we indeed obtain very good scaling when doing distributed training, shown here in images per second throughput. And here's a scaling efficiency compared to ideal linear scaling., which gives us close to linear speedup for training to accuracy time and enables to run such experiments in somewhat reasonable time frame. So this number of 81 hours when using 256 GPUs indeed corresponds to what we have measured for average full training run on ImageNet 21K with a large big transfer ResNet 152 times 4 in width on Jewel's booster. So with One single GPU, one would require roughly 2.4 years to finish such a training run. So being able to run large-scale pre-training in reasonable amount of time is necessary. So there's substantially less than months or weeks or years required otherwise with smaller machines. We were able then to obtain first preliminary results replicating transfer behavior after a large model, large data pre-training, where we used large ResNets from big transfer work pre-trained on either ImageNet 1K or 21K. And we see indeed that larger data, ImageNet 21K here in blue, improves transfer across different target datasets. And interestingly, again, the improvement is especially strongly pronounced on few-shot transfer regime where only few examples, like just 1 or 5 here, are shown per class during fine-tuning on transfer. So this all is still of course very far from the kind of one model, uh, for all construct I mentioned earlier, uh, that just continually digests any incoming tasks and datasets, solving those as they come. Learning from them and never further improving the generic model in open-end learning fashion. But it allows to prepare the steps by looking on how transfer in a more simpler setting actually depends on model and data properties and on type of losses used during training, what conditions may help and what may rather hurt transfer. So I would like to summarize with stating that the special use case of transfer learning applied to pattern detection in X-ray chest imaging domain is not only dealing with the current real-world problem, but is also well suited to actually study properties of large-scale transfer learning in general. For instance, an attempt to derive the scaling laws showing dependency of transfer performance on model size, data size, and forms of learning used during pre-training. This scenario also makes possible to study transfer for different types of tasks— classification, segmentation— and we saw that not only supervised pre-training is possible there, but also more interestingly different unsupervised modes that do not require labels. Another interesting direction in this frame is given by existence of different medical imaging modalities, so not only conventional 2D X-ray but also 3D CT and ultrasound datasets that are publicly available. Although their size is still very, very modest, it already opens perspectives towards multimodal learning and may allow to compare more systematically the differences in intra- and inter-domain transfer. A further perspective I would like to mention is establishing this kind of a golden repository where validated models obtained by large-scale pre-training are made openly available to allow for further reuse, validation, and boosting reproducible research in this area. So in this sense, again, everybody who is interested to collaborate on any of those directions, please feel welcome to reach out and join in also for the upcoming compute time application on Jewel's booster supercomputer. So here I would like to finish and thank the people that are involved in this line of work, all foremost my colleague Mehdi Cherti, who is main collaborator on COVIDNetX. Doing large-scale transfer learning experiments I was describing here. He's also main developer and maintainer of Ulrich Data Challenges platform where COVIDnetX also offers a challenge to participate on, among others. A big gratitude to Alex Struber for making supercomputers are working good enough for us to run those experiments at all, to Helmut's AI consultant team at our place that offers a different kind of support here and there. And to GSC Supercomputing Facility together with Dell Supercomputing Center for making all those machines available. And of course, a thank you goes out to the audience for their attention. Last but not least, thanks a lot for being here, and I'm looking forward to any questions and comments to come from your side.

Thanks. Yeah, thank you very much for your talk, for giving us insight in how you improved from Transfer Learning via Large-Scale Model Pre-Training. So, and before we start the questions and answers section, I will announce that we have Eric Medivan from the Helmholtz AI Consultant Team, and they will advertise their roadshow on 10th of June, and this will be after the question and answer section. So now there are some questions in the chat.. And I will moderate them for you. So the first was asked by Jens Petersen. What's your opinion on mixture of experts for vision? Could something like Switch Transformer work for vision models, particularly in a way that new data can be integrated via new experts and only routing needs to be tuned?

Yeah, yeah. So I mean, in general, transformers of course architectures that are coming strongly in Envision. And particularly so in the mixture of expert settings, this routing that is already intrinsically there in Transformers is very suitable to mediate between the different subnetworks that play the role of experts, switching between the tasks, between the domains. I think this is a very fruitful area.

We should definitely have a look on that.

Although the computational demand is further increasing, of course. So the switch transformers, I think, are on the range of trillion of parameters even. So for that, you definitely need to go to a larger machine to conduct experiments like this.

Thank you for this answer. So the next question asks, can transfer learning be performed on 3D segmentation tasks? And if that is the case, which dataset can be used for pre-training in this scenario?

Yeah, that's a very good question because it mentions explicitly 3D, which is not so easily available as a large dataset for pre-training. If it would be for 2D, I would immediately say, well, we take generic datasets for 2D segmentation that are really large by now and try to get some benefit from pre-training there. For 3D, honestly, I am not aware of any dataset that is large enough to serve as a pre-training base.

Maybe somebody else has a hint.

Um, so I mean, there are upcoming models like, um, NeRF, which deal with volumetric rendering, that provide— could provide a lot of ground rules in 3D space on the object boundaries, on the object structure in general. Of course, you could try to work with synthetic data that provides you a true 3D space and segmentations in it and do something like a domain adaptation afterwards and hoping that you get the sim-to-real transfer. That is possible, but natural datasets that are taken in real environment dealing with 3D, I'm not aware of.

Okay, thank you very much. So the next question comes from Abdul Nuit. He asks, regarding Double Descent, why does the test loss not decrease monotonically when increasing model or data size?

Is there some theoretical justification for this? Hold on, maybe I'm trying to go to the slide where we have this. So let me check here. Did I find it? Should be here. Okay, can you see this? Okay, so the question is why the test loss does not decrease monotonically. I mean, here in this study by Belkin in 2019, they were dealing mostly with model size. They were not looking on data size. And what they see here, if you increase model size monotonically to a very, very large model, then you indeed observe a monotonic decrease of test loss. So this is also then shown here on a specific example, which is though a bit of a toy example, one have to say, with a ResNet-18 that is actually a small model. So, but you see the monotonic decrease with increasing the model size, and it actually decreases stronger than the so-called sweet spot state in the classical under-parameterized regime. So you go way beyond the generalization capability in the classical regime. With data size, this kind of double descent or triple or whatever descent curves has to be yet obtained properly. Maybe there will be a similar or a different picture, but for the model size increase, there is this kind of monotonic decrease then observed. So maybe I haven't understood the question then properly.

Maybe you can exchange afterwards. Yeah. Yeah, thank you. So the next question comes from Patrick Godot. He says, first, thank you for the nice talk, and then you mentioned the vision of a continual learning model. What are your expectations on this in the near future? And despite of brute force, any combination of pre-training, fine-tuning, and the general rule bigger is better, what strategy do you know of for choosing ideal transfer scenarios?

Well, I mean, there are a lot of works that go into direction of bi-level optimization that I mentioned at the very beginning of the talk, where you have meta-learning approaches so at least two different loops, the inner and the outer loop, where from the beginning you try to optimize your model for the scenario of the unseen data. So it, in this sense, it learns to learn. So you really impose changes on the core learning algorithm during the training. I think this is one very fruitful direction, although computationally very, very expensive. But then again, you know, So further take on this is people were also convinced that in vision you can solve the problems before 2006 or so only by constructing very clever specialized architectures that use something very magical. It turned out then that if you just increase the scale of the model, of the data, and then all of a sudden things begin to work. And I mean, there is a view advocated by, for example, Richard Sutton, who says that this is the lecture we have to learn. So first you have to increase the scale to invest compute and see whether you indeed cannot solve the problem before going to very, like, intricate architectures and trying ideas in that direction. So I think we should do also these experiments with scaling up. And see what happens, whether all of a sudden multitasking and multi-domain and like meta-learning just emerges from a huge capacity of the network. Like the vision, the object recognition capabilities emerged from increasing the data and the network size in the vision domain.

Thanks for this answer. So then we have last question from Olaf Zimmermann. He asked what insights and evidences exist regarding the type of abstraction that is happening in the interpolation regime, and he's meant second valley.

So let me switch to the, to the valley. So what insights exist, what happens when the arrow begins to drop? Okay, so I think here one can refer to this work by Belkin et al. In Proceedings of National Academy of Sciences. They tried to argue that what happens beyond the interpolation threshold is some sort of automated model selection, so that the huge capacity of the network allows it actually to contain a lot of simple Occam's razor compatible models that are switched based on the current text task affordance. So you incorporate an ensemble of such in one big network, and therefore you still follow Occam's razor, just simply depending on the context you are on. This is the explanation I'm aware of, and there is a pile of work now following this double descent curve behavior appearing in NeurIPS and ICLR, and people are trying to find their more solid, I think, theoretical foundations why this is happening. But to my knowledge, the most, uh, the most evidence obtained is still experimental. So there is no very clear explanation why this is the case so far.

So thanks again, thanks for the questions.

I was hoping that somebody would ask about what is in the last picture, but yeah, nobody did.

Do you want to explain?

Well, yeah, I mean, explaining is a big word. It was just a Dalmatian dog. And I mean, I'm pretty sure that our current vision networks large enough could even identify it, although the scene is completely highland beaches.

But who knows, actually, one should try.

Great.

And again, thank you very much, Dania, for, for being today and being our speaker today. And yeah, if there are further questions, please visit our website. There are all the contact details. And yeah, for, for any question regarding the seminar, please come back to me. So thank you very much, everybody, and we'll see us in 2 weeks for the next Data Science Seminar, and be aware to be there on 10th of June for the Helmholtz Roadshow. So thank you again, goodbye, and stay healthy, everyone.

Thanks a lot for inviting me again.

Thanks for attending.

Bye-bye, folks.