Scientist Stories: Ewan Birney, Genomics, Imaging and AI

Well, good afternoon, everyone. Thanks for those of you who are here in the room, but we know from the registration process that there are hundreds of you who are watching via Zoom, which we did open up to anyone who wanted to join. But I can't think of a better way for having a first seminar of 2023, at least for me, than to to be able to introduce Ewan Birney, a good friend and colleague of both NHGRI, really the whole NIH. And Ewan is here meeting with various people at NIH for 3 days on a variety of topics, and he was eager and willing to also give a talk, and we were eager and willing to host such a talk. So those— and if we could just put up his first slide while I'm introducing him. There you go. As you can read, Ewan's current position is the Deputy Director General of the European Molecular Biology Laboratory, or EMBL. He's also the director of EMBL's EBI, or the European Bioinformatics Institute, with Rolf Altweiler. And he, in addition to those major leadership responsibilities, is still an active researcher and runs his own research group. Historically, way back when, Ewan completed his PhD at the Wellcome Sanger Institute working with Richard Durbin, and actually in 2000 already he took a— first leadership position becoming head of nucleotide data at EMBL-EBI. I first got to know Ewan, I don't know if it was 25 or 26 or 27 years ago, somewhere, and so probably about the midway point of the Human Genome Project where Ewan came in on the scene at just the right time and made seminal contributions to helping make sense of all the data being generated, especially about the human genome sequence. And really becoming really apparent at that time he was going to be a major leader in the field for not just years to come, but literally for decades to come. Now, that was— Genome Project ended, as you know, in 2013, and— or I mean 2003, but then what I can say is it was not surprising, therefore, that Ewan over the years has since picked up all sorts of leadership roles. 2012, he took over as Associate Director of the EMBL-EBI. He became Director then in 2015, and then in 2020, he became the Deputy Director General of EMBL. And in this role, he assists the EMBL Director General in relation to engagement with all the EMBL member states and external representation, really crisscrossing Europe on a regular basis. Some analogies, as we've been meeting today, between crisscrossing the NIH at the different institutes versus crisscrossing Europe, different countries, some things are similar, some things are different, but it is interesting, some of the similarities. Research interest-wise, Ewan has always been very interested since the end of the Human Genome Project in functional genomics, DNA algorithms, statistical methods to analyze genomic information. These skills were applied in particular shortly after the Genome Project ended on the NHGRI's ENCODE project, Encyclopedia of DNA Elements, where Ewan played a very important role not only in contributing data but also in really leading the effort and especially writing up some of the key papers at the early stages and mid-stages of that important project. And if he didn't have already enough to do, Ewan is now a non-executive director of Genomics England, and he's also a consultant advisor to a number of companies. I'm not Ewan's only fan. There's thousands of people that think highly of him, and he's, as a result, received many awards. He, as an example, he was elected to become an EMBO member in 2012, a fellow of the Royal Society in 2014. And a fellow of the Academy of Medical Sciences in 2015, all huge honors for an investigator in the UK. We're delighted that he's here today. I will tell for those who are in the room, when we're done with the seminar, come to microphones that are halfway up the stairs and to the two aisles. But in addition, if you are listening in by Zoom, feel free. Do we want the questions in the Q&A, Adam, or in the— Q&A. If you could please put your questions in the Q&A on Zoom,. And we have somebody in the room who will be monitoring those and will be relaying those through the microphone as— at the end of the talk. So with that, Ewan's going to talk about genomics, imaging, and AI, three technologies that are changing biological research through to clinical practice. Ewan.

So thank you very much, Eric. It's a real pleasure to be here at NIH. My science has always been very collaborative with colleagues here in America. I have fond memories of Rock Bottom, which I hope is still going, and the beer here. I have less fond memories of windowless ballrooms hired by NIH institutes, which I have sat in at hideous hours for my time clock. But, you know, international science, you have to make some sacrifices. So I could have talked about lots of different things. I've decided here to give a very strategic talk. Come on Friday to Cold Spring Harbor where I'll give my research talk. And for those people who were at Raleigh, you heard my research talk already. But this is going to be quite strategic. And I'm going to try and get through a lot. So apologies for the speed. I just want to introduce EMBL first. This is the organization I am Deputy Director General of. It is Europe's only intergovernment laboratory for the life science research. We're a kind of sister of CERN. CERN does high-energy physics. We do molecular biology. We have 5 different missions: perform excellent research, to deliver scientific services, to provide advanced training, which really means PhDs and postdocs, to do innovation translation and integrate the life sciences. Not unambitious projects. And we're headquartered in the lovely town of Heidelberg, Germany. It's on the Neckar, which is a tributary of the Rhine. Beautiful German Rhineland town. And that is our main laboratory. And our second biggest site is at EMBL-EBI, which I'm director of. And that is completely dry. It's bioinformatics only there. And kind of occupies a similar role to NCBI over here. We also have sites which are dedicated— which have a stronger service delivery side around the synchrotrons of Grenoble in France and in Hamburg in Germany. And then our two latest sites is in Rome that focuses on epigenetics and neurobiology. And our very latest site is in Barcelona. This is on the beach. It is beautiful. And they study tissue biology. And do disease modeling and mesoscale imaging. And with my boss, Edith Hurd, she came in as the new director general. And she has really challenged us to step out, to keep one foot very much in the lab and very much anchored to the atomic and molecular and cellular understanding of life, but to step out with the other foot outside of the lab and think about life in context. And there's one aspect here about thinking about the diversity of life, which I'll come back to, and species in different places. But also thinking particularly about humans and the way humans interact with the environment, the way the environment impacts humans, and the way humans are an environment. We're an environment for many species that live in our guts and alongside us. And that's been a tremendous kind of renewal and reimagining of where the endpoints of molecular biology are. It's quite exciting. But I want to focus here strategically on what you might think of as data science. And there's a great paper by Kim Naismith, who was the head of the Institute for Molecular Pathology in Vienna, very close to Bruno, about Mendel. He gave a review about Mendel. And he said that Mendel was the first data scientist in biology. He was the first scientist that collected data and then from that data derived a scientific insight rather than doing it through observations of just sort of individuals as Darwin did. So for Mendel, this was, of course, his beloved peas in his different ways. His data collection was about the phenotypes in the F1 and the F2 generation. And his model was Mendelian genetics and dominance and recessive. And what I'm going to claim here is that in some sense, we are still walking the same path. It's just that we're generating high-dimensional multivariate multimodal datasets. We're analyzing them in more sophisticated ways than writing down the tables in a variety of different ways. But we are still doing this fundamental process of deriving models or understanding making accurate predictions of the world. And this world has been changing. And I just want to step you through— many of you will know about these changes. But I want to step you through those changes, one of the two technologies, and then AI. So this is a busy slide about the successive technology innovation around DNA sequencing. On the left-hand side, there's Sanger dideoxy sequencing. Perhaps only Eric and myself have ever done this in this room. Maybe some others have. The old-school way of doing it where you generated a large number of molecules and you separated them on the gel with radioactivity. It's great fun. This got automated in the 1990s by Lee Hood and founding the company ABI. —where they used fluorescence, and rather than a slab gel, eventually capillary gels. And that was the bedrock for a lot of the Human Genome Project. And I've skipped now a couple of technologies, which were really good and really powerful. A lot of love for 454, soft spot for Helicase. But really, the dominant technology was Solexa, which became— was bought up by Illumina, which was reversible termination. Of pretty much the same process as Sanger dideoxy, but you could reverse the top thing and extend the strand by one more. The other trick that Solexa worked out, and then obviously now in Illumina, was that rather than doing a one-dimensional or single-lane interrogation, you could do that interrogation on a 2D plane with imaging. And that 2D plane has become very, very dense. And so you get many, many spots. And so you can sequence many things at the same time. And as many of you know, that's been a tremendous boon to all sorts of aspects of genomics. But that innovation has absolutely not stopped. And I've put these two new technologies— both work off single molecules. So the first one is Nanopore sequencing, where a single strand of nucleic acid goes through a pore, and you measure the ionic current which is being changed by the transit of the sequence through there. This was working conceptually in the 1990s. It's just that the strand of DNA went way too fast through the pore. So you could tell that it went through, that it was complex, and you had no hope in reading the sequence. And really, one of the big breakthroughs was to slow down the progression of the DNA sequence through the pore using what is kind of counterintuitively called a motor. It's really a brake. And it's a unidirectional brake of the nucleic acid going through that. So it steps one nucleotide at a time at a slow enough rate that you can measure the change in current. So that's Oxford Nanopore. The second one is equally whizzy in technology. Which is PacBio. I love this phrase from PacBio, which is the only moving part in our machine is the polymerase, which is a cute way of saying that they are measuring individual pieces of DNA being synthesized by the polymerase and using this optics trick that there's a quantum effect if you pass a very strong laser just underneath the surface. It will only excite the fluorophores which are right tethered just above it. So it's called zero-guide wavelengths. So, and that— both of those technologies have been stretching nucleic acid technology in a way. And just to note here, I have a honking great big conflict of interest. I'm a consultant and shareholder to Oxford Nanopore. So the animation has not quite worked out here, but I just wanted to go through the axes of improvement over these different years. So the first axis is the number of simultaneous molecules you can measure in, say, a fixed window of time. And that's gone from 1 to, would you believe, 1 times 10 to the 9. And the length of reads, actually, we went backwards with Solexa. And I remember processing the very first 19-base-pair Solexa data and working out whether it was useful or not. And it was, indeed, useful. And now it's 1 or even more than 1, so 4 times 10 to the 6th base pairs of a single molecule. And if you had told me 10, 20 years ago that we would get megabase reads out of a sequencing machine, I would be saying that you were fantasizing. One interesting thing about Oxford Nanopore is that it can see many different types of nucleic acid. So you notice I've been using the word nucleic acid, not DNA, about this. And so it can sense both DNA and RNA. And for the DNA, it can sense different types of modification on the DNA just as well as it can sense the 4 bases. And so we've got a diversity of molecules these technologies can sense. One of the big drivers is naturally the cost, or more accurately, the price for base pairs. And that has dropped precipitously over the years, faster than perhaps any other technology over the last 20 years. And the error rate has gone up and down, this arrow, but it does range from 1 in 10 which is pretty eye-watering at some level, but doable. It's still got signal. But now down, or in some scenarios being down in 1 in 10 to the 4, so 1 in 10,000 letters are incorrect in some scenarios. And then finally, there's the kind of all-in time to result. And just to stress, the all-in here doesn't mean total number of base pairs divided by total amount of time. It really means, If I have a sample on my desk now, how quickly do I get a result out in the computer or on the device? And that has gone from months, or a month or so, down to— would you believe it— 2 hours. And in Norway, they did a testing of a gene fusion during an operation for glioma. Using Oxford Nanopore technology. A lot of people feel there's some kind of inherent trade-off in this space. There actually isn't. There isn't some law of physics that doesn't allow you to be on the edge of all of these things. Sort of remarkable fact, and people often forget about it. Now, of course, when companies make different devices, they often find that their technology is maxing out in one direction or another. But it's worth stressing there's no inherent aspect to the way these things are traded off in these technologies. So what have we done with this? So, of course, one thing is we've created reference genomes. This is one of my favorite representations of the reference genome, the human genome, which is a bookshelf, which you can see at the Wellcome Museum in Euston Road next to the Wellcome Trust, where they have printed out in not that small text, all the entire reference genome as books. And it's quite a big bookshelf. Every cell in our body has that bookshelf— asterisk, not red blood cells. So every nucleated cell in our body has that bookshelf inside it. But of course, there has been a tremendous benefit of when we've done molecular biology. If we can make that molecular biology have a nucleic acid, in particular DNA readout, then we can scale that technology massively. And so that's about measuring RNA abundance, RNA-seq, measuring epigenetics, measuring chromatin transcription factor binding or histone modifications, that's ChIP-seq. And there are these increasingly complicated da-da-da-da-da-seq methods. And this is this trick about you do something at scale, or rather in solution or in the cell, you get the readout to be DNA, and suddenly you can scale your molecular biology into a new space. And then, of course, it's been very exciting about the way we've integrated this already into clinical practice. I'll come back to this in one or two areas. But it really works in rare disease. It works really well in rare disease, which is not actually in aggregate that rare. It works well in cancer. And I think there's huge amounts of confidence that it will continue to really inform cancer clinical practice. And of course, in COVID-19, we discovered that indeed doing tracking these nasty infectious agents using their— in the case of COVID's RNA sequence— has been tremendously informative about infectious biology. And again, you can see that just being a routine part of how we think about infectious biology. So I'm going to switch gears and talk about a different technology, which is imaging. And imaging has had this beautiful explosion of different scales, different time domains, different fields of view, so different sizes of view, and then finally, different fundamental measurements, which I'll come on to. It's not just— scattering and reflection and these other things. And there's a host of different technologies that underpin that. At the atomic scale, it's when we use electrons rather than photons as our probe. And so these are electron microscopes. And that, due to the incredible advances in detection devices, have allowed us to get to near atomic scale resolution from direct imaging. It's kind of just amazing, really amazing. There is a big space of aspects around super-resolution microscopy. This is where you trade information, resolution that we have in time to gain resolution in space. And so you can go below the diffraction limit of light because you know that the molecules that you're looking at are often sparse or have certain particular properties. And that way, you can take multiple photons that you are pretty sure come from the same molecule, and you can now isolate better than the diffraction limit where that molecule comes from. Rather amazingly, we can now track little microtubule motors live, not quite in a cell, in an in vitro solution using super-resolution microscopy, and watch these little motors drag little things around. Really, it's remarkable. And then we have a series of other higher-resolution schemes. And that includes a lot of very clever use of photons, in particular, two-photon, three-photon technology. It uses these things such as the way scattering works. And this is optical coherence tomography. And it uses other ways of doing interrogations of living matter, for example, magnetic resonance imaging. Now, one thing that really spans all of these things is in none of these cases do we use our eye anymore as the image collection device. You cannot put your eye— you do not look at these things with your eyes anymore. Of course, the result you visualize on the screen, but the capture is happening via some detector, and then it's being processed inside of a computer. And this is most obvious in super-resolution technologies, Super-resolution technologies, the thing you capture are just blinks, are just tiny little blinks of photons. And so you really cannot— I mean, there's absolutely no transformation from the video that you receive into the 3-dimensional or 2-dimensional space. It must go through a computer to be visualized, basically. Now, there are too many things to talk about here. And I just want to pull out some of the cases that come from AMBUL, Embel Heidelberg is particularly strong in this. The left-hand side here is from Julia Mohammed from Embel Heidelberg. She did in situ structural biology, so EM tomography on host cells. And so in this case, there were bacterial cells. And she was able to get near atomic resolution of the ribosomes. And these weren't just collections of in vitro ribosomes. They were the ribosomes actually in this bacterial cell. And so she could then also do things like introduce an antibiotic to a different sample, do the same procedure, and see exactly what the antibiotic changed structurally of these ribosomes in vivo. This is being redone for many other different cellular aspects. And so we're getting cellular or subcellular resolution of how these molecular machines work using this EM tomography. It really is quite remarkable. Second one is, again, from colleagues, including Cornelius Grace in Montefratondo, using X-ray, high-resolution X-ray imaging. What's quite exciting here, we've got quite a lot of space, which is kind of— we want more resolution than light microscopy, but we don't need the resolution of EM, and we want a wider field. And a great example is tracing axons in brains. Axons are small things. We want to know how the connections work, but we want to do a lot of them at the same time. And X-ray scanning, high-resolution X-ray imaging, is a very good solution to this. And the final one is this absolutely amazing thing, Brillouin.— it's a French name, I've murdered it, I'm sorry to French colleagues— microscopy. This is where you measure the stiffness of a material in vivo by imaging the material and passing a sound wave through it at the same time. And you look at the deflections of the wavelength of the light as the sound wave moves through the material. This is a non-invasive way of getting a property now out from your living material, which is quite an interesting property. How stiff is this material, and does it change? So here's a biophysical property that you can extract out of this particular device for this. And we call it imaging, but it's kind of stiffness. It's no longer a scattering, a photon scattering component. So what is the third technology? So I'm sure you're all aware, obviously, that in those two technologies, they've been sort of welded with computers for a very long time. No one by hand aligns Illumina reads. No one puts their eye to an EM microscope and works out what's going on. You no longer use us as the data collection device and the data interpretation device. You have to have computers, and you have to have systems to build and interpret it. But I think there's something different which has happened over the last 5 years, and that has been the development of artificial intelligence technologies, deep learning technologies. And I'm going to spend a bit more on this. And take you through it. So what is this sort of artificial intelligence? So this is the way I think about this. There are two different key things which are kind of inputs to it. One is good, very good data engineering and very good collections of data. This is incredibly easy to put up in PowerPoint slides. It is really annoying to do. It's like the grungy work of data science. Every tech industry person knows that this is some of the really hard yards where you actually just get your data straight. It sounds completely trivial. Lots of things fall over at this stage. The second breakthrough was this development of what I like to call calculus engineering. So what is deep learning at its core? It's gradient descent. So this is where you take a function, you differentiate it, and you want to try and You want to find a minimum, and so you follow in a high-dimensional space the gradient to go down. Now the thing about this is previously— I mean, this technology has been around almost since Newton. But doing it not merely in a multivariate way, but in a bonkers big multivariate way with very, very complex functions. And it's that bonkers big multivariate gradient descent which is at the core of neural networks. And to make this work, you need good hardware, which has turned out to be the same hardware that you use for computer gaming. I mean, who knew? But if you get a lot of these computer gaming chips, that is good hardware to do this. You can have specialized hardware for it, but actually, the ray tracer GPUs do really, really well here. And then I've got 3 different flavors of deep learning. So what I describe as labeling, another one which I'm going to describe as modeling, and the third one is I'm going to describe as alternative uses of this gradient descent. Now, if you want— this does feel very black boxy. And for me, this was an annoying black box until only 2 or 3 years ago. And now some of you are ahead of me, but I suspect most people are with me or behind me in terms of the frustration about this kind of weird, like, magic happens. I'd really recommend this video short called AlphaGo. It's on YouTube. And it's about the case where Demis Hassabis's team from DeepMind created a deep learning program which is about playing Go, which was considered to be a game that humans could use that intuition and other aspects and beat many computers. And in fact, DeepMind created a deep learning architecture that could do this. It's a lovely story. There's just a lovely aspect of kind of Korean culture and Go about it. It's really sort of beautiful in many ways. But if you watch it, wait for the moment where they realized that AlphaGo has a new way of thinking about Go. So humans made a program that came up with a new way of playing Go. And that set of strategies has become an important part of Go strategies in the future. So I just want to step you through some of this. So as I mentioned, you've got this data engineering aspects of pulling data together and getting it ready. So there's kind of two flavors of this. One is when you're doing it because you've generated the data. But now you want to generate lots and lots of data. And it's incredibly easy to get lost and do silly things when you have these large datasets. And it's a bit embarrassing. Nobody talks about the fact that, oh, I lost track of my data and don't know really where it all is. It's a surprisingly common failure mode in data science. It's a very important thing to do well. And that goes up to, at the biggest level, I've got a picture here of the UK Biobank cohort. That is a big cohort. In some sense, that is a massive data organization project, longitudinal over time, keeping all these things in sync in many, many different places. Oops, sorry. Another source of data is the data deposited by many different scientists over many different years, and here, colleagues at NCBI here and ourselves at EBI represent that collective memory for the scientific community and allow these datasets that have been generated by many different people over many different years to be reused in these methods. And then I won't talk too much about the calculus engineering. The calculus engineering, you can write it, you can try to write it out like this. My colleague Moritz Gersting says, you know, we can't even write these out using mathematical notation. I mean, the only sensible way to do this is to show someone the PyTorch code. Although it's maths, it's sort of impossible to digest as maths with notation. Really, the succinct way of doing it is you show it through these code kits, such as PyTorch or JAX. So let me go through the first application. And this is where we take a high-dimensional data set and we label it. And what do I mean by that? So here's a picture of the human heart from an MRI scan. And the labeling here is to label the left ventricle and the right ventricle and the blood pool and all of these other things. This has become almost routine using deep learning. You set this up with some training data. There's some tricks of the trade about how you do this. And you can pretty consistently come up with a good model that will label future hearts. And it's completely transformed the way we think about images. Because I can now look at a big set of images and say to myself, well, that's fine. I'll make some kind of sensible image-derived phenotype from that big set. There's something which is kind of, again, labeling, but you're letting the network do a bit more. This is a very good example from Platanarius about looking at cells that grow in this little organism where they had to label cells but also cluster them. The interesting thing is that we didn't tell— well, my colleague Anna Kruszczak did not tell the computer what the right answer was. It was more like, please find a space for the clustering of these cells, but segment them at the same time. And then just to go over to the nucleic acid technology, this is Oxford Nanopore. Oxford Nanopore signal is determined by base calls by a deep neural network. So it's using a deep neural network to go from the signal to the base calls. Which in some sense is a labeling problem. I've got signal. I want to label it with the base pairs. Sometimes they're just 4, A, T, G, and C. Sometimes it's 5, A, T, G, C, and methyl C. Sometimes it's 6, A, T, G, C, methyl C, hydroxy methyl C. And so you can start to build up this labeling problem. A real step forwards has been this thing which has been closer to AlphaGo. Where we make the— get the computer to make models of the world. And sometimes, just like AlphaGo, the computer makes a model that we ourselves don't truly understand. And the breakthrough here— and it was jaw-dropping when I first saw it— was AlphaFold. This is the work, again, from DeepMind, Demis Hassabis, and John Jumper, where they took multiple alignments, not very deep multiple alignments, just about 10 sequences, 20 sequences. And they trained a neural network that they had designed themselves to predict the 3-dimensional structure. And that neural network did 2 things. Firstly, it very often got the right result. And it met John Maltz's criteria for a near-equivalent method to experimental results through the very rigorous CASP competition, which is great. The second thing it did is it has an internal view about whether it's got the right answer or not. So AlphaFold itself has a calibrated view of whether it's right or wrong. And so there's times when it says, no, no, no, I'm wrong. Or it says, no, no, no, I'm definitely right.. And that calibration is accurate. So AlphaFold itself knows whether it's right or wrong. Now, a lot of very, very clever people— I've met some of them, and they're definitely cleverer than me— have tried to solve this problem over many, many different years. We've had about 30 to 40 years of biophysicists, bioinformaticians, computational people attack this problem. And it's really interesting that it's fallen at this point. So we have similar things in the prediction here of genomic tracks. This is using data from ENCODE often or from the other kind of chromatin aspects, where you try and work out what is it about this DNA sequence in this particular cell type that means that you get this particular ChIP-seq or this particular open chromatin. And that, again, has been a very robust process. You can train models that do this and work well. You can also do this for splice sites. You can make models that predict where splice sites are in transcription. Now, what's interesting— let's do the splice site ones because it's a case where we understand the molecular machine that does it, the spliceosome. What's interesting about that deep learning model that does very well is it has some of the classic things that we know. The 5' splice site and the 3' splice site looks like they're in that model somehow. Model has learned about those things. There's also a lot of kind of weird and fuzzy stuff. There was a point where I first looked at these things, and I was very frustrated about these models. Because why is it that we're generating— why isn't it not clean? And then you have to realize that molecular biology is allowed to use quite a lot of fuzzy stuff itself. So these models, which are accurate, generalizable predictors of molecular biology, I think are tremendously useful. In lots of different areas. And then we can do, for example, other things. And this is more of a data fusion thing where we're bringing images and genomics together, trained on— to classify cancer tissues and to classify the kind of mutation type that's happening in cancer. And so there's a class of scenarios where you're doing this for a kind of data fusion. Task. Just diving into AlphaFold a little bit better. So it's been really interesting. So I should mention— I don't know, do I talk about this? No. I should mention that we had a— AlphaFold themselves use the databases from the EBI, which are done in collaboration with colleagues here at RCSB. In Rutgers and other places, the Worldwide PDB. It's been running since the 1970s to do this. So this database was absolutely critical in the ability for AlphaFold for them to learn to train. We were very, very lucky that DeepMind partnered with us for another task, which was to partner with us to distribute the results. So DeepMind decided to make the method the code and the data from AlphaFold all completely open. And they partnered with us to make all the data open. And Samir, who did this— we did this for 29 key species, including unsurprisingly human. And then we did it for 100,000 proteins. And then Samir said, sod it, let's just do it for everything, every single known protein. And it's worth saying that— Computers scale in a way that experiments struggle to scale. And so when we can solve things in computers— and we can't always solve these problems in computers— but when we can solve them in computers, we get this remarkable piece of scaling for this. And it's been interesting to ask why this has been achievable. And there's some very technical aspects about the way the neural network has been put together. I do not understand all of it. One important thing is that this whole process is one differentiable function. So they can do end-to-end training. They don't have steps. Although this is written out as a pipeline of steps, it is presented as one function to the training system. And that one function is differentiated and trained by gradient descent. The other aspect of this is that John Jumper stresses that the diversity of data in the PDB, the diversity of structures in the PDB, was absolutely critical. It wasn't simply about the number of structures. It was that the structures represented such a diversity of different scenarios where proteins fold that the neural network had to kind of learn the physics, presumably, the evolution and the physics behind it. And then I finally want to touch on a different thing. And just to stress, although we often use these gradient descent functions for neural networks, which are these, really, these combinations of linear models. There's nothing in the maths that says that they must be used for neural networks. You are allowed to use them for many other different things. And this is from Moritz Gerstung, who you can see I love working with. And he has repositioned the Cox partial likelihood on these maths engines, this calculus engine. And this means— he can do bonkers big Cox proportional hazards, this classic 1970s epidemiological model that helps you understand risk factors that underlie a disease. But now we can do it with 10,000 different variables simultaneously. It's kind of trivial. If you're an epidemiologist, you might say, are you sure you want to do that, number one. Number two, where on earth do you get the data to train a 10,000-variable epidemiological model? And the answer there is you get it from Søren Brunak and the wonderful, amazing Danish National Patient Register, which, would you believe, has got data, pretty much complete data, of the Danish population since 1979 as they've interacted with hospitals across Denmark. So that is— there's about 5 million Danes alive at any point in time. Some of the Many of the people in the database have now died. Many of them are alive. And we have the data from 1979 to do this. So we have the scale of data. It's about 1 billion or 2 billion interactions of those people with the health care system or more, I think. And so we can both computationally scale it using this calculus. And we have the data set to derive this. And that has been really interesting. There's a preprint about individualized cancer prediction. We are rolling this out for non— for common disease. And there's lots of obvious things, and then there's some non-obvious things. And of course, it's always a bit of a head scratch about whether the non-obvious things are new or misleading or what's going on. So at that stage of working out these things, but I recommend the preprint. So what do we need? To make this future continue. And that's a future where biology is done at scale with technology using these neural networks and deep learning technologies. So one is very close to my heart. We need open, organized, fundamental biological biomolecular data. I wanted to stress AlphaFold would not have been feasible without the WWPDB, without the Worldwide PDB. When structural biology community came together in Long Island in the Protein Data Bank to start that off, they did not foresee this use case. They could not write down that this was going to happen. But they did know that what they were measuring was really important about understanding life, and they were not going to have their experiments go to waste by simply having them on their own. There weren't even hard disks at that point. They were probably floppies or some iron drive or something. So just as our colleagues from 40, 30, 20 years ago, we need to propagate our information correctly in the future. It needs to be as open as possible, as restricted as necessary, The only restriction which I think we really should totally honor— I mean, we should obviously honor all restrictions, but the one which is most understandable is patient and citizen privacy and laws. But we should try and make it open as possible. And I should stress that we've got a kind of new skill set, which I like to call data engineering. And that is about just helping flow, manipulate, structure, store, reflow to somewhere else these very big datasets. And it's, again, often something that we skate over because it feels so trivial. And yet, if we don't get it right, nothing works. And of course, this is close to my heart because of what EMBL-EBI does. And just like NCBI, we go through this loop of Having scientists that generate data and make discoveries, they deposit on publication. We archive and share the data with global collaborators and all scientists. We classify, enrich, combine, and analyze, so there's this data engineering, which is more than just storing the data, but sorting it out. And then that allows us to distribute not only the raw data or the deposited data, but also the value-added data resources., and then scientists can go and build on that in the future. And this is a little map. It's not live because I'm not that bold, but this is all the different people who use the EBI at a particular snapshot in time. And I should say, people often wonder, how is this done? And there's a feeling, I think, somehow it's somebody else's problem. Like somehow it's like the pixies of the data world will somehow look after our data. The kind of house elves, if you're into Harry Potter, sort all this out. And these people, that's sort of true. But we must do it. This is what science is about. It's propagating this information into the future for scientists to build on scientists, as that's happened. All through science. And the Global Biodata Coalition, the GBC, is an international forum to help coordinate the funding of that. This is a very deliberate pitch to the project officers and people in the NIH to enjoy the integration or enjoy this forum as a place to discover this. And credit to Eric Green for helping setting this up. But we also have another amazing opportunity in the future coming our way. And that is because our science of molecular measurement and imaging has become really useful in clinical practice. And that means we have an entirely new industry, which is not research, which is measuring things on one organism. Those organisms are humans. And they're doing it at remarkable scale. And this is a picture from— which I'll come back to a bit— about Genomics England, which is the clinical genomic sequencing in the NHS. So all the sequences done in Genomics England are for clinical benefit of the patient, but at the same time, they store the data to allow for future ongoing research. Research for when patients give consent and patients can withdraw. And the genomes, by the way, are kept in a kind of walled garden, a private cloud that researchers can get access. This is true not only for genomics, but also for imaging, and of course, for health care data, just as I showed you with the Danish data. And I like to think of hospitals as massive phenotyping centers for humans. So we can phenotype humans. We can image humans. We can measure the molecules that are present in humans, not least the DNA. And we, the researchers, do not have to do the heavy lifting. The heavy lifting is done by the health care service. And that is a tremendous opportunity for tapping into those data flows. Now, to do that, we need standards that span this research world through to this health care world. And because these are about citizens and patients, basically, one has to be a lot more compliant to national laws and the way people operate healthcare. And although if we have mouse data, we can have these data commons like the EBI, where people deposit data for the PDB or for mice or for whatever, or for human cell lines into a single place, and then somebody can download and use them. Or in a situation where they're deposited and the analyst comes and talks to them on the cloud. When it's about clinical data, the tendency— and this is, I think, is just going to become the norm— is that you must go and visit this data in their separate locations. And definitely my group does this. For UK Biobank, we can use the UK Biobank RAP, which is their name for their cloud. For Genomics England, we use the Genomics England TRE, Trusted Research Environment, which is their cloud. For the Danish data, we use the Danish Computerø, which is their cloud for accessing the data. It is quite irritating. My postdocs have about 4 different laptops sometimes, because some of the paranoia involves you must have a Danish laptop. But it is doable. It is a doable way of doing research. And just as the internet needed standards to enable good interoperability across a kind of federated peer-to-peer world, we also need standards in this world. And this is that standard-setting body— I'm very lucky to chair this— which is the Global Alliance for Genomics and Health for responsible sharing of genomic data for the benefit of human health. Health. And we have tight collaborations with HL7, which does that for, in some sense, just the healthcare data, though that is very much driven from the healthcare side, whereas GA4GH, I think, is much more evenly driven from the research and the healthcare side. And I won't bore you with the wonderful graphic about GA4GH things, but if you— please do use GA4GH standards. And as CRAM, BAM, SAM, and VCF are GA4GH standards, if you do genomics, you are already using GA4GH standards. So just be particularly proud at that moment. But we also need to evolve our workforce. And I think this is a really interesting challenge for many, many different institutions. Many, many of our discoveries are going to happen inside of the computer. Sort of by definition because our measurement devices are so wedded to computers in genomics and in imaging and in other things, but also because of this AI aspect. And it's important to realize we don't need to have just one type of skill set. There's actually quite a lot of different types of people we need there. So at one level, there are— I'll do the middle one first— the people who come from maths. Often, this This is Virginia Ullman from the EBI. She's deputy head of research. She has a maths background. She's into AI, deep learning, and machine learning, and theory. She likes to think about images in a mathematical way. It's all very clever stuff. And she really does the methods development that is appropriate. And then those methods have to often be used. Now, sometimes these people— will use them and deploy them and make discoveries. But very often they're motivated by the methods. And that is great. We should celebrate that. We should not ding them for being methods developers. It's a real mistake to say these people, oh, you're not quite a scientist in biology because you're doing methods only. You know, you're a computational person. That is shrinking our space and shrinking our ecosystem. We need to be very generous about that. Well, we definitely need a lot of these people. This is Evangelia. I'm afraid I've taken lots of people from EMBL. She is a data— she leads a data science group. So she's a faculty member who leads this group of data scientists at EMBL-EBI. She's obsessed with pathways and cellular switching and cellular decisions, proteomics and phosphorylation and that sort of stuff. And she, of course, has to have a good interface for people like Virginie. But she wants to make biological discoveries. She wants to discover, understand bits of biology by using a computer. She's totally happy if she uses a chi-square test or an AI machine learning test. Focus is, do I understand this bit of biology better? But as I said, this gets underpinned by this class of people. And here I'm showing the Associate Director for Data Resources, Joe McIntyre. She leads basically the 400-strong set of predominantly data engineers at UBI to deliver the data resources behind this. So the ability to flow, shape, deliver, and integrate data at scale. And it's engineering, but it's not software engineering. It's data engineering. How do you put these big bricks of data together? Involves an awful lot of software, but the end result is a dataset and databases and data sources. And then finally, this is Jan Korbel, who's at EMBL Heidelberg, who's some— probably many of you know many of these people, Joe and Jan and Ivan Galyan-Vicini. Just to stress that we need these people at all levels, at the PhD level, at the staff level, at the staff scientist level, at the faculty level, and in our leadership. And actually, I think there is a gap here because we, by definition, our leadership tends to be older. And so as we go through this change, it is difficult to bring people in who can understand how— what does a good data engineer look like? What does a good methods developer look like? And so it's very important. We have to accelerate the track of these people up to leadership levels. So I'm about to finish. Some of you might be skeptical and say, well, does it really matter? I find the skepticism, you know, oh, that was a really easy problem. There was no way the protein folding problem was an easy problem. Absolutely no way that was an easy problem, and it fell to a computational method using these kind of techniques. But I just want to make it really explicit, and I'm going to give you two examples. Example one is an example from basic research. This is work from Jan Kaczynski in EMBL Hamburg, where he developed a very accurate and good model of the nuclear pore Now, this is one of these kind of crazy big Death Star-like complexes in our cells. It's got multiple components, it's got multiple proteins. And although we had EM maps of these things, we really didn't understand how the different proteins fitted in there because we didn't have atomic-scale resolution of the components. And with AlphaFold and with experimental work, and with EM models and with chemical footprint, hydrogen-deuterium exchange, and single-molecule fluorescent resonance energy transfer, he was able to put together a credible model, really credible model of this big beast of a complex. And you can see that we're going to walk around a lot of the things in the cell using this kind of integrative structural biology, and then the work that I showed from Julia Muhammad about that in vivo captured biology of these complexes in action. And then I just want to give this the opposite end of this, which is these clinical operations, going back to Genomics England. And I just want to talk you through what's happening here. So for the genomics people, I— we should all be very, very proud about how we have changed clinical practice. For rare disease. So I'm just going to make this concrete for the UK case. This can map to US, times the numbers by 5. So there's about 6,000— 600,000 live births a year, and about 50,000 of those in the UK within the first 6 months are drawn to concern of a clinician. The pediatrician is concerned about the baby and thinks that the baby has a rare genetic— possibly a rare genetic disease. Very often, that diagnosis starts 24 hours after birth when the pediatrician first looks at the baby. It's kind of a remarkable process. In genome— in England at the moment, those children can be scheduled to be sequenced, have their genome sequenced. And wherever possible, the NHS tries to recruit both the mother, which is usually feasible, and the father to have their genome sequenced at the same time. So for each child, we get 3 genomes. Kind of on average, it's about 2.5. And so I'm just giving you the number of genomes per year that come through this process. By the way, at the moment, we're only seeing about what we think is 1% of the 2%— not 1%, we've seen 50% of the 2%, 1% overall coming through. So we think this number is going to double once we get full penetration. About 25% to 30% of these children are diagnosed. And this here, there's a component where we're rolling in bits of deep learning into the variant calling. But also the interpretation of the variants, for example, splice sites or protein structure, from deep learning techniques. And that's where the computational techniques will make a gain. Now, let me just tell you about the outcomes. So there's quite a good publication here in the New England Journal of Medicine. So if a child is diagnosed, the diagnosis— post-diagnosis, on average, they have 50% less visits to the hospital over the remainder of the follow-up time in that study. So it's very clear that the clinical practice for those children have improved. 25% of the time, there's an immediate change to how those children are being treated. And then 5% of the time, there are these transformative changes, really transformative changes to the child's life. And also, this has a big impact on the families. So families who have— where they have children with a suspected rare genetic disease, and then they get a diagnosis, have much better informed choices if they want to have future children. They want to have future children, they can make better choices about this. And the second thing is that the family clinicians have closure or end this diagnostic odyssey. And that is actually a non-trivial thing about removing the worry and concerns from this. And just to say that many of these people also consent for their data to be used for ongoing research. Not everybody, but many, many people. That's now more like 90%. Now, the interesting thing here is every time we roll in a new enhancement in the interpretation interpretation of the mutations, we get more diagnoses, and more children end up on the right-hand side of this picture. And just to give you some of these profound changes in outcome, so the most amazing one is this RPE65 loss of function, which is a gene in the retina, in the retinal pigment epithelia. If you have a loss of function in this gene, the child nearly always goes blind. And there's a gene therapy that will fix this. And the children don't go blind. Now, of course, that therapy only works if you have an accurate diagnosis that the gene that's causing the blindness is this gene. This will not cure any blindness. It will only cure the blindness caused caused by this particular gene. Now, that's about 30 children a year. The next gene coming through this process, which is RPGR, is about 500 children a year in the UK. And I hope that many more of these things will come through, especially with CRISPR. A second example is this deficiency in this gene. This is a type of severe immune deficiency. But actually, it wasn't severe enough for the clinicians to really recognize it at birth. And so this child was coming back and forth into hospital with infections, scheduled for genome sequencing. They said, oh my gosh, you've got a defective immune system, and so scheduled the child for a bone marrow transplant. And another example is the maturity onset diagnosis of diabetes for the young. And this looks like diabetes. Well, it is diabetes, but it's not one of the classic type 1 or type 2 diabetes, and you have a completely different treatment for that type of diabetes with a completely different set of drugs. So thank you very much for listening. I hope I've helped you think about AI, genomics, and imaging. And I'm very, very happy to take questions.

Okay. Well, thanks, Ewan. I know there's over 300 people listening by Zoom. So I suspect there might be questions coming in. There's a bunch of questions. But people should also come to a microphone. I'll kick the first one off. Ewan, some of the last data that you were talking about from Genomics England, this was all the intervention to get genome sequencing was when the patient was symptomatic? They were of concern. Yes. They were of concern. And was there any modeling is to tradeoffs in terms of getting it just outright—

[Speaker:JOSHUA_ELKINGTON] Yes, so you're forestalling the Genomics England project precisely to do a really good study on this, which is to sequence 100,000 newborns completely flat ascertainment, and just to understand sort of to what extent could we augment the very successful metabolic screening of our blood spots, if we did genomic sequencing, how many more very clear-cut cases would we get where you would be confident of doing an intervention for an apparently not sick child? And that's started?

That's in the planning.

In the planning, so not launched yet. Well, it's agreed, if you know what I mean.

Yeah. It's not recruiting. Go ahead.

It's on.

After such a forward-looking talk, I hate to ask an in-the-weeds kind of question, but when you were talking about AlphaFold and saying, well, let's just apply it to everything, I mean, there you have this interesting opportunity that the model knows kind of when it's wrong. What was the rate on that? Like, have we surveyed, like, with crystallography most of the structures, or is there a big chunk that's missing?

Does it kind of direct you where to go? There's some good papers on this by my colleague Christina Rengo and ex-EBI colleague Pedro Beltré. There's a bit of a kind of argument about whether new novel folds have come out and whether AlphaFold can see novel folds. It all hinges on this definition of the word novel fold. It's a bit boring. Two things, though, which is really clear. It can definitely predict structures where you're like, well, maybe it's an alpha beta beta or whatever they come up with, but it's very different. So there's definitely that. And it's also really quite good at doing membrane-bound— it doesn't seem to have a problem with membrane-bound structures. So these things are always alpha helical, nearly always alpha helical bundles. So in some sense, it's not a fold. But for the membrane people, it's like, this is amazing.

I'm so excited. Well, the membrane people never get a break, so— Sorry?

Membrane people never get a break, so they do that after all the years. I've met quite a few, like, this has changed my life conversations with people in that space. What is interesting about this sort of aware— this calibration error model, and there's a big debate we had, Samir, with the people in DeepMind in particular about whether DeepMind AlphaFold should show the low confidence regions. Now, if you look at an AlphaFold model, these are the bits that look like to be spaghetti, and they're red. And there was a big thing, no, no, let's trim it back to only the things that AlphaFold knows is good because that looks kind of pretty and, you know, knows it's good. It's ended up being— and I'm very glad that Samir said, no, it's really important we show the whole thing and we talk about it and stuff like that. It looks like a lot of those are intrinsically disordered regions, and so they really do have no structure, at least in isolation. And I think that's stimulating a lot more thinking about these intrinsically disordered regions. So it's quite an interesting question about whether you even call that a misprediction by AlphaFold or not. In some sense, it's not a globular fold. AlphaFold's kind of doing the right thing to draw it as spaghetti, even though it looks somewhat counterintuitive. So there's quite an interesting story behind all of that. Thank you.

We'll go behind you, and then we'll go over here.

Right here first. Thank you very much for your talk. It was very interesting. My question is about— more a broader question about the interpretability of AI methods. And I guess for things like AlphaFold, this might be more straightforward. But for some of the other diagnostic applications you described, for like— Yeah.

How important is that? Yeah. So for starters, it's not straightforward in AlphaFold. It's actually quite still a bit mysterious about why AlphaFold works, just like AlphaGo. Was making moves in Go that people were like, that's a crazy move. And then they're like, wait a second, that move was really important. How did they know that was such a good move? And so there's a similar thing with AlphaFold. We don't really understand it. In terms of the interpretability of a lot of these models, I have grown much more comfortable with it, where, say, if we take the splice site model, you build the splice site model, you train it, you convince yourself through a variety of methods that it really is generalizable. And then you study the model. So you start deleting bits of the model or changing some of the inputs or seeing how it behaves in different scenarios in the computer. That makes sense. And so you stop asking the model itself to be obviously interpretable, but you think of the model as— Almost like a synthetic data generator. Yeah, as a thing to study, an artifact, a proper artifact in itself. And I think that solves a lot of problems. I will say for— I mean, I know this is obvious. I haven't stressed it. But when we launch any type of machine learning, but in particular deep learning, on data that comes from observational humans, we get all the biases and all the complications that we have in society is represented in the data. So all the stratification. Or the race, ethnicity, or the social structuring, or the language stuff, all of that comes through. And these techniques are very good at picking up on stuff. So we have to be very, very careful about how we use these models, because we must remember that very often we will replicate the world that we live in now, not the world we want to live in. And so we've got to be super, super careful about that when we're talking about human observational human stuff.

That makes sense. Thank you. Hi, great talk. So my question sort of relates to the fact that you've talked about sort of technological advances and also the way that the computational part comes in. So my question is, will we ever get to a point where you could get a blood sample from one individual put it on one machine, and get the sort of full biological spread of DNA, RNA, proteins, everything? Or will it be that you get one thing, and the computer will be able to tell you all the biological parts with some degree of certainty?

So I'm worried that I will— if I said yes to that, I'll get sued in a Theranos style. Style way. And I definitely don't want to go down that path. Yeah, absolutely right. I mean, I don't think that's going to be— you know, it's going to be a messy middle, okay? We'll get multiple measurements over our life course. I definitely think we'll all have our genomes, or the future population will have genomes sequenced at birth, will be one thing. But we'll probably check in, have a bit of, like, a bit of metabolomics, a little bit of immune you know, genetics or whatever, some key moments in our checkups. And then there'll be some kind of massive learning machine that's kind of crunching the numbers, being very careful about not trying to replicate the world we live in now, but rather the future world that we want people to live in. So that's going to be an interesting problem. So, oh my gosh, there's a lot of moving parts to that. Let me remind you about the data engineers. I mean, you think that this problem is a statistical problem. But the problem before the statistical problem is just, is all the data straight in the right place at the right time? Am I pulling my hair out to get the data there? But still, it's a future we can imagine, I think, quite well. And it's really interesting seeing this starting to happen, baby steps starting to happen, be it in Denmark or the UK, being two places that I know well.

Ben?

Yeah, and then I'm going to go to the computer. Thanks. That was a fantastic talk. You talked a lot about how these— over many years of hard work, the data was ingested by various organizations that are using it to produce the unbelievable stuff like AlphaFold. And you've talked to some of the organizations that are producing the code that does this.

And it's incredibly—

it's amazing that it's available, whether it's TensorFlow or PyTorch. AlphaFold. So I guess, what is your advice to all of us about making sure that things flow both ways in terms of data going in and especially code coming out so that can be available to everybody out of some of these private organizations?

I mean, I think, although there's a lot of people who can kind of complain about the commercial companies, realistically, I think they've played a blinder, to use the British phrase. Be it DeepMind or Facebook or all these other places. So I think we have to realize that molecular biology and this and the biological life sciences has always had this close relationship with pharmaceuticals. We kind of understand how that interface works. Some people interchange with that. We're going to have the same relationship. We do have that relationship with the tech companies. And of course, it's different. Different career structures. They pay more. That's one headache. Massive headache. But, you know, the way we have this kind of, you know, we're in it together, but of course they're commercial with pharma, we've got to have that same kind of positioning with tech. Be very confident about that open academic side of us. But I think be really open to when these companies, be it pharma, be it tech, want to play with us because we we want to play with that.

Adam, you want to ask 2 or 3 questions from— Okay, yeah, there's a lot, 11 or so that came online, so I'll give the first askers the priority. The first one is with regard to the Cox proportional hazard models and whether those variables are being tracked across time to identify individual transitions between higher and lower outcome states.

And so that was just kind of a temporal Yeah, I mean, you're going to push my— the question is going to push my understanding of Cox proportional hazards to its limit. But, you know, I think the default way of thinking about this is just that time is one of your axes and you're looking for the shift in risk. And so that doesn't come out of the vanilla model. The vanilla model is just what is your probability of having an event on this time axis relative to each other. So the vanilla model doesn't have kind of transitions in it. I think there are unvanilla models, but we haven't deployed them.

Next question from Laura Gorell. How do we enable the scientific community to use technologies when they might not have the technological background to write code workflows?

Kind of to your grow the community. Yeah, so we've got to grow the skills. I think the first thing to I've always said this, you'll be amazed how just a little bit of Unix, a little bit of Python, and a little bit of R will get you. And I think we, for a variety of reasons, we make the computer scarier than the chemical hood or the tissue culture, when really it's quite a learnable skill. So the first thing for the colleagues who are like, oh, I can't do the computers, is just spend a little bit of time time, you don't have to go very far. It's like the one little baby step outside of Excel, and then you're like, the world is open to you in a completely new way. So do do that little conversion course. And then, yeah, we've got to train and allow people to move skills. As well as retraining people, I think one has to really focus, or at least have a similar amount of focus, on the future pipeline of people coming through.

And then I'll choose this one because it's pertinent with regard to ChatGPT, which has been in the Twittersphere a lot lately. I love my large language models. And the question, which I'll summarize, is that some of these large language models can make very confident garbage as an output. And so the summary of this question is, do you have any concern that AI generation of data such as AlphaFold models have the potential to poison the source data for future studies by having correct-looking but untrue results?

So the first thing about it is, I don't know about you, but I'm unimpressed by ChatGPT. I am impressed about some of the things you can do. You know, it can compose poems and limericks and all of that. The creative side, it does well. But when I ask it a scientific question, boy, does it get it wrong sometimes. And I think that says something important about the difference where the output is do something creative that a human can do. I mean, it's a very different task to please correctly summarize the medical records of this individual. I mean, they're very, very different tasks. And we should not kid ourselves. That they're the same task. And what the questioner just mentioned there, it is striking that an important part of AlphaFold that it came with an error model. And that error model was well calibrated. And I do think that's one of the criteria we should have for what is a good model. Not only do you make a good prediction, but do you have a well-calibrated error model of your prediction? And then the don't poison stuff It's obvious, which is just keep track of why you've recorded things. It again goes to this data engineering. So this is in the parlance of databases, evidence codes. So it's totally fine to fuse prediction and experiment side by side. But you must keep track of it. You must keep track of what is prediction and what is experiment and why and stuff like that. And that's a really good example of quite a subtle piece of data engineering. You don't get that right from the start, 5 years later you're hosed because you can't untangle it afterwards.

Carolyn, I'm going to give Carolyn the last question if that's OK.

Yep. So one of the things that I spend a lot of time thinking about is when are things sort of still in a developmental phase and when are they ready to be sort of deployed at the size and scale scale that you're talking about with a lot of these examples you're giving. And I guess, is there a way— do you think there's a way that the machine learning can help inform that decision? Or do you think that that's going to lead to bad decisions?

I trust you over machine learning. So I don't know that that's a well-formulated question. A kind of well-formulated problem. This goes to an interesting thing. One of the reasons why DeepMind chose the protein folding problem, not only was there a good data set, not only was it a well-defined problem, there's a very good competition, very well-run competition, CASP by John Mold, funded by NSF and by GMS, maybe? I don't know who funded it. And it was the presence of that competition that was— so you have a scoring— to compare it to. Yeah. So they knew when they were doing better and stuff like that. And that goes to these problems which you think, oh, surely a computer can integrate all of this. And you're like, well, can I define the problem? If I can't even define the problem, I can't get a scoring metric. No. Not going to happen. And I feel that the problem there is the score is— as much as the problem's interesting, I don't think there's a very well-defined define scoring function for it.

Thanks. Okay, I think we're going to cut off there. We have got to keep you and I on a very busy schedule. So on behalf of the dozens of people in the room, the hundreds of people that watch by Zoom, thank you, and what a great start for a seminar in 2023. Much appreciated.