Scientist Stories: Fei Wang, Computational Health

introduction. I actually don't know NJI has such a gorgeous building here. You can see Statue of Liberty, so that's nice. And when you invited me, I was, I mean, feeling like, what can I talk about? I mean, because I used to work on drugs, and then I shifted to be really focusing on biomedicine and healthcare. But recently we have some interested to shift the focus back to graph. I will, I will show you. So a little bit about myself. Yeah, a little bit about myself. So as you can see, so I, I got my PhD on machine learning and my thesis about hardcore machine learning, I mean hardcore graph learning, 20 years ago. And then I did postdoc in statistics up in, up in Ithaca, and then I started to focusing on machine learning for healthcare. But since I was, um, uh, one— hello, doesn't work. Hello? No, there's no light. I think it's a battery issue. Oh. Hello. So I began to focus on machine learning. I used to just developing machine learning always doesn't care about the application, and I started to focus on healthcare and medicine since 2010, where I was one of the founding members of the First healthcare analytics research group there and worked there until 2015. I joined Cornell since 2016 until now, and last year I founded this institute. Although I am in medical school, but because of my training, I have a broad connections and collaborations with different campuses, Ithaca and Cornell Tech, and I have quite a few students from computer science as well. So I'm thinking of maybe I can briefly talk about some of our research on healthcare and medicine. And I want to point out some of my own opinions on some of the— I don't know if pitfalls is the correct word— the current research on deep learning, especially in the healthcare world, and my own vision on what should be the path forward and why graph can play a role there. I think, as I said initially, at my time when I did my PhD study, we do linear algorithms, we do label propagation, we study the spectrum, we do normalize the cut. And now everything becomes nonlinear and of course deep and more complicated. But like, this is an early work when I was at IBM. We tried to use some of the early label propagation type algorithm to do drug recommendation. Collected a set of patients and a set of medications, um, from electronic health records in the love region. You know, we calculate patient similarity, drug similarity, uh, we tested— I mean, we also have algorithm do metric learning to optimize the metric, but it is, uh, uh, wasn't, um, going well, uh, on the clinical side, not the performance, but the meaning. So, I mean, we can learn complicated, let's say, generalized machine learning to covariance measures. How do you explain that to the doctors? So in this work, we just use very simple Jaccard index and draw— they call that tiny multiple coefficient. So you look at the SNAC not the snapshot, but the building blocks of the chemical structure. And you just look at for every drug, so appears, not appear, and you do the intersection over union. That's the JAKR index. And for the cross node pipeline, it's really you look at if the patient took the drug or no, and the label is really if that drug has demonstrated to be beneficial in this context, to look at the different you know, statin drugs like treating— hello? For treating hyperlipidemia, there are so many different type of drugs, statin drugs like simvastatin, atorvastatin. I think we now see 5 different type of statins see which one will be effective individually. We talk about precision medicine, you can see the concept later on as well. Hopefully with the different, you know, status or conditions of the patient, I think this one really has some problem. Just turned off, turned off. Hello? Hello? It works. You can hear? Yeah, I don't know if this is needed for people online, so Okay. I just use this spot. So yeah, we look at the different type of study and drugs and see which one is effective for individual patients. And we consulted our clinician. I mean, how we need the measures to see, you know, what do you mean by the drug is effective? So essentially look at their LDL level, which in the baseline period is out of control. And then start taking one type of statin drug, and within a certain time period, um, uh, with, with two consecutive measurement of LDL, if it is under control, we say, uh, the drug is effective. So we come up with the label, and this is a hybrid graph where you have, um, uh, the patient, patient side, patient drug side, drug side, and you have the label of the drug if it is effective for patient. And for the label on the drug side, it's really indicating if it is the drug itself or not. So after propagating, it really means on this cohort, on the effectiveness level, so if the drug is similar to other drugs or no. And we show by leveraging all those different— we do label propagation, and then we show leveraging all those heterogeneous similarity information, the algorithm works well. This is early, and then I forgot from which year. So there is a kind of study of graph convolutional networks. Initially, it is not called graph neural networks, right? So it is really studying how do we define the convolution operator on the graphs, and it is not easy. So people transform you know, the graph into their frequency domain and according to the convolution theory. And so the convolution in the original domain is equivalent to the product of the frequency domain. And you do the product and you do the inverse convolution back. So it's kind of like in that way. So, and then people do approximation on the convolution kernel and, you know, there's a Chebyshev approximation, and later on there is a paper talking about first-order approximation, and the graph neural network is pretty much similar to that first-order approximation. So something like that. So we are kind of like, we are thinking of using graph neural networks to do some brain network analysis. In this case, it's essentially through the MRI scans. So want to see if the patient's actually having Parkinson's disease or not. And here we don't have the luxury of being able— the sample size being able to work on pixel level. So we use some software, a free server developed by Harvard, to process the brain into 84 regions according to a particular atlas so that you can think of every region represents a particular area in the brain that have certain functionality. And I'm sorry, this is not on MRI, but on DTI, Diffuse Tensor Imaging. And DTI can tell you across different regions how the signals are communicating with each other. So you can somehow look at the connectivity and based on the different connectivity patterns, that may indicate if this patient is having some abnormalities or not. But what we did is for DTI from the raw image to something we can really process. So there is a quantification procedure and there are various ways of doing that and different quantification can give you, you can think of different brain connectivity network. In our case would be 84 by 84. So if we think of every quantification method as a view, we already call view, and now people like to call them modality. So you can think of, let's say you have 6 different methods doing the quantification, then you have different, 6 different modalities or views, although with the same brain, but can encode different kinds of information. We were trying to see if we can encode them all together to come up with something to do the classification. So we use the GCN, we use early chip shift approximated. I remember we used, I think we did some tuning, we use 13th order approximation. And from the original brain network, we use this multiview GCN. You can think of a GCN on every modality and then do a max pooling. So you've got a feature representation. And another challenge we have is the sample size. So we only have in total in this case, that's very normal in biomedicine. So it's not like in other domain you can easily get billions or trillions of samples. Here, I mean, in total it's a cohort study. We just have 500 patients. So training such A network from scratch is challenging, so we adopted Siamese architectures. Instead of doing sample-wise classification, we do pairwise comparison. If the pair of DTI images are from the same class or no. So it's in spirit, it's pretty similar to contrastive learning nowadays, just that we don't have batches. It's, you know, I think people in the past called it Siamese network. So we do comparison. So in this case, it can sort of boost the sample size because you're doing every pair, comparing every pair to a certain extent. And this is the performance. So we compared with at that time, this was which year? I remember 2018. So 6 years ago. So We look at, you know, these are on the top portion. Those different columns are different modalities, and we compare with what if you just use a raw matrix, you do PCA. People in medicine like PCA a lot. So fully connected network and fully connected network with a deeper architecture and a graph convolutional network. And then on the bottom portion, as we further show, if you combine all of them, we can further improve the performance. So that's kind of like one of the early work when GCN was out. So we were thinking of applying that in this context, and that actually, this paper was pretty well cited. And then after that, I think it came to the age of people got very enthusiastic about molecules, all kinds of molecule design algorithms. I don't know how many papers published every year. So this was like, I believe the work started in 2018. I think the paper was submitted in 2019 and published in 2020 on KDD conference. We, I think before our method, I think the state of the art I remember is the junction tree type of algorithm from MIT. So designing, of course, small molecules. And we were thinking, because I have a post working on graphs and networks, so we were thinking of doing a generative model on small molecule generation. And this is, if you now look at that, it's fairly straightforward. So you decompose the graph into the connections, and also that's the bond that will be a tensor, right? And also the different type of atoms, and you generate two flows. So one is working on the bond, the other is conditional flow, condition on the bond and how the node is generated. And then you have a latent space distributed, it's spherical Gaussian, and you use a reverse flow to reconstruct that so that you can do some generation, and from latent space linked to another layer of predictor, you can do some property prediction. So it's a pretty standard— I wouldn't say not too much of difference compared to a normalizing flow model, but I don't know if there is any work earlier than ours that applied this flow. Now there are a lot of flow type of model for designing molecules. And so we achieved a pretty good performance at that time. As you can see, junction tree is one of the best algorithms. And also there are like some other Bayesian algorithms and we compare on different indices, validity, uniqueness, novelty. If you work on this, you know what are those. And we also created an index called NUE, which is Essentially a multiplication of those three novelty, that's N, U is uniqueness, and validity. And we did some validity check as corrections after the flow, so we can get some good performance there. And we also show like I mean, the generated, how the generated molecules look like. So after we got this published and we saw this is a neat algorithm and we write a pretty good software, we wrote a pretty good software. So I was thinking then, so what will be the next step? So the next step, because we are looking at molecules as potential you know, pharmaceutical elements that could be utilized. I mean, maybe later developed as drugs. So I think the next step, the key is to find biochemistry people to help us to synthesize the molecule and do all, and then reach out to basic science people to get those molecules tested. And first in cell line, and if it works well, then grab some rats and do some animal experiments. And then if it, if we are lucky, it works well, then go to human phase 1, 2, 3. And then ultimately go to be on market and there will be phase 4 for post-market surveillance. But I mean, we are pretty resourceful, I will say, because while it's not just the Cornell Medical School, we also have Rockefeller, we have Memorial Sloan Kettering. And various hospitals, but we did not find a very good chemistry partner to help us. So that's why you— from my group, this is the only paper we generated, I mean, or wrote on molecule design, because I don't see a very clear pathway forward because this is If you really want to show the success, that's the pipeline we have to do, but we cannot get good chemistry partners. So, but something, a pleasant surprise was last year, NVIDIA in the GTC conference, they announced this new service, BioNemo. And BioNemo is a software service for whatever large, small molecule design. And they integrated or implemented different algorithms for different types of molecule design. So for small molecule design, so they incorporated MolFlow. That's our model. Then they, another model, there's only two models here. The other model is something by NVIDIA themselves. So we were, when they reached out to us, we were pretty happy because the paper has been like stay there for a while. It is well cited, but As I said, so in order to make real impact, so we should really reach out to biochemistry people and later on pharma people to really get the synthesized and see the real impact. Ideally, to make a real drug, it's not like building another organism on top of that. So, but this is that, so as you can see, so we still try to keep track of the graph learning algorithms and within different contexts. And for a lot of the cases, we are pretty early in trying out if new algorithm works out. But in recent years, there are just too many algorithms popping out, and a lot of the times we don't see why do we need graph-based algorithm. So we didn't really try a lot of them. Instead, we really look at, so what is needed in the actual problem, trying to solve the problem. So that's kind of like some of the works we have done involving graph-based learning. But now I'm trying— I'm going to introduce some of my own opinions on this. So first, I think you have all heard of this notion of precision medicine. It was proposed for— this is which year? 2015. So almost 10 years. So essentially it represents a paradigm shift on treatment development. So in the past, so whatever pharma or whoever developed the treatment is based on the hypothesis of one size fits all. So develop a drug that can cure the entire, let's say, whatever, cancer or COVID. But the challenge is that these diseases are complicated. One of the important characteristics of complication is heterogeneity. So although people infected by SARS-CoV-2, by the same virus, but different people can have very different clinical manifestations. So ultimately, precision medicine aims to develop targeted treatment for every individual, but it is pretty challenging, I would say, because you, on the data side, you really need a whole, a huge amount of data captured for each individual, you know, not just the molecular level, like genetics or genomics, not just the clinical level, but also environmental impact, impact, and also other things. So as an intermediate step, so one of the viable way could be, can we find out some subtypes? I mean, or stratify the patients who discover some of the patterns of disease and look at stratified medicine. But no matter what, So this piece, Francis Collins is the prior NIH director. He indeed pointed— I mean, this is a copy from what he said in this paper, the need for machine learning, because computation in biomedicine is not something new. I mean, statistics have been there for I don't know how many years, probably longer than machine learning as a field itself. So why do we need machine learning? So because statistics cannot handle a lot of the complexities in biomedical data in terms of the scale and the complexity. Then I'm thinking of as a talk, the title of my talk, I'm thinking of what should be the right way of thinking about machine learning in biomedicine. So I'm thinking of like previously what we have done, and I bet a lot of the labs also doing things like this. So we just look at what data do I have and what new tools comes out, and I see if that can be applied to, or I can develop new tools with my data. But I'm saying it doesn't matter in what context. It can be a disease context, it can be this drug discovery and development context. If you talk to, let's say, talk about drug discovery and development, if you talk to people working in pharma, So that they never tell you like if you design a good molecule is a success or you, you did, you design a good clinical trial is a success. I mean, there is a pipeline starting from hypothesis generation. I mean, this is knowledge graph we built can serve as a tool, but essentially for pharma, they typically have the disease they want to target or they have a set of molecules, I mean, store in their fridge. So which molecule can target which disease? That's the hypothesis generation. And then to molecule design and synthesis. So this is the work we have done. I mean, the MoldFlow work is kind of in this way. And then to clinical experiments. After you have the molecule design and you synthesize the molecule, you have to show all kinds of wet lab and the animal experiments to show its effectiveness. So this is another work we have done, analyze animal data on pathology slides. And then it goes to human clinical trial. This is perhaps takes the most of the longest time and also the most of the money. So we have done some work on trial emulation using real-world clinical data to inform human clinical trial design and then even to Post-market surveillance. So after your drug is on the market, you don't want anything really bad to happen, like a severe side effect which didn't happen in your human clinical trial phase but happened in the real world. This is because the population discrepancy. So the participants in the trial is always different than the real-world patient. So any of the step or component fail, this process fail, so your drug fail. So it is never like comes as individual step. So like really like thinking in a holistic way from let's say whatever the clinical evidence that can trigger some design principles in molecule design. I don't think I see too many works on that. Of course, data access is the challenge, but the translation gap is a huge huge problem. I mean, you design a good molecule, even test it, it works well for animals. Most of the time it wouldn't work for human. It happens a lot. So how to, you know, think of the data and design algorithm to crosswalk at different steps is important to consider. And I show two other examples on concrete diseases like for COVID. We have done a lot of research for COVID, like from epidemiology. Usually people think about using— I mean, it was from early 2019. People talk about in the summer when the temperature and humidity increase, the pandemic will automatically stop. So we just show— we collected lots of information from the internet and show that based on the evidence from China and the United States, I mean, just purely based on the temperature and humidity, that is not enough to reduce R-naught value to something below 1. So that's what you need to stop a pandemic. So purely based on climate change is not enough. And to mechanism, so we have done some study to pick out a particular gene IL-6, which is expressed consistently high in patients with cancer infected with COVID So that potentially explains why cancer patients are more likely to get severe acute outcome for COVID and could be a potential drug target. But of course, the mechanism COVID is still under study. There are lots of hypotheses, but a lot of times if we can, using computational technique to generate high quality hypothesis that can also guide the basic science folks to do experiments and validate in a better direction and also treatment. So we write this perspective piece on how to integrate different information at different scales to come up with effective repurposing hypothesis for COVID. Currently, I will say almost all of the FDA-approved COVID is based on repurposing. It's not original indication like remdesivir. Original indication is Ebola, like, what is that, paracetamol. So that's an interesting story because there is a company called Benevolent AI. They said that Varicitinib could be effective for COVID is discovered from their knowledge graph. Varicitinib is an autoimmune drug developed by Eli Lilly, and based on their hypothesis, Eli Lilly conducted a clinical trial and later on got good results, and FDA authorized an emergency use for that. So these are good examples. There are not too many of these kind of examples, computationally derived hypothesis that works in the real world and got FDA's approval. But we need more, and I bet we can see more. But you really need all those broad knowledge. And also clinical care, we wrote some papers on like at early stage how to do early prediction and also tease out the heterogeneity of COVID in the acute phase. All those are The demand is coming from our clinical people working in the frontline taking care of COVID patients. But also like Alzheimer's is also another important disease that although there are very few treatment just approved in recent 1 or 2 years, but it is very controversial. So if you look at Twitter, there are lots of debates and discussions about the safety and the effectiveness of those drugs. But so one of the reasons is certainly because Alzheimer's disease and related dementia is too complicated. And we have a series of work studying the genetics from the clinical side, how to design the trials, and what is the impact of the environmental factors, and even in terms of diagnosis because now we rely on imaging scans, neuroimage, or proteins from spinal fluid. Both procedures are pretty invasive to the patient. So, you know, we are taking a trial from Oregon Health and Science University which collected patient-doctor conversations. And you can think of this as The notion is for patients who have cognitive impairment, they're going to talk differently than normal people. If you look at the scripts, actually a lot of the conversation, you really don't know what the patient is talking about. So is there any way we can train, let's say, a chatbot? Now we have ChatGPT, but when we wrote this paper, there's no such high-quality chatbot yet. So through a conversation with the patients with this agent, so is there any way we can discover some early signs of cognitive impairment serving as a screening tool? So that's very realistic. So as you can see, there are a whole bunch of that. In total, there are 6 R01s and ongoing R01s. We study different aspects of Alzheimer's disease. But more than that, so this is kind of like more holistic. View of the portfolio of my institute. So, we touch really on— I try to— on different aspects of the data. This is a data type with respect to different diseases because different data are going to have different information and encode different aspects of the disease. So, if you really want to understand in a comprehensive way about the disease, you really need as much as possible the data from different sources and different types. And of course, the challenge will be it's going to trigger a lot of innovations in algorithms as well, because this is a complex dataset that might be unique for biomedicine in lots of ways. Like, you need to protect the privacy, which you don't need to consider in a lot of other domains. You need to consider fairness. You need to consider the variety. I mean, you can see the data is really really heterogeneous and explanation, you know, it's not like we always hypothesize medical people must require us that the model must be explainable. But if you talk to actual doctors, they are much more open-minded than we thought. So some of them said, I don't know, I don't really need your algorithm to be explainable as long as you can show me, you know, in a convincing way that your model works universally well in a lot of different scenarios. So that comes to the problem of evaluation. And the example is always like in drugs. So a lot of the drugs we take prescribed nowadays, like aspirin, we still don't know the mechanism of action. So you can think of the drug itself is actually a black box. But over the years, so people have accumulated lots of evidence and understand how drug works. So they said that it's okay. So, but there are lots of debates and it is hard to, of course, performance is the first. So it's hard to say which one is the most important. We should ignore the other aspects and really consider all of them. So that's why we gradually seeing in machine learning conferences, you want to develop not just a accurate model, but also fair. And at the same time, you want it to be secure, protect the privacy. And of course, lots of times this has also to be explainable. So you integrate all of those different characteristics into one model design. So I believe that's the future. So we're going to see more and more and better and better models satisfy all these needs. Certainly one important aspect is how can you integrate all of these different domains together? So as I showed in this portfolio figure, so in the middle is the knowledge graph. So we need things like that to connect the data from all different aspects. But the difference typically between biomedicine people, I mean, Knowledge base or knowledge graph or ontology is a pretty traditional topic in biomedicine. But at the same time, we also have lots of folks working in computer science and constructing large-scale common sense knowledge graph. What's the difference? Computer science people care more about scale automation, but not too much about the quality. I mean, when I say not too much, so compared to biomedicine folks, they almost check every pair in a manual way. Okay, so I mean, the reason is understandable because in biomedicine they don't want to make any mistakes, and because that is— could be like lead to a drug design and later on put patient life at risk. So they don't want any risk on that. But at the same time, they also want to scale. So I think one interesting topic is how can we leverage all those large-scale and automatic technologies to really focusing on creating high-quality knowledge graph, specialized knowledge graph. Common sense is important, but how can we define the quality? That's a few riots and biomedical proposals or papers. That's the part that got criticized a lot by domain experts. But no matter what, I think this kind of resource is very important for linking up the information from different places. Another concept I think is important— so that's where knowledge comes from, right? Another important concept I'm thinking of is, so nowadays, So lots of people talking about federated learning. So federated learning is what? So you have the same, well, not the same type of feature and across different sites, and you don't want to share the data. You want them to work collectively. So federated learning is a strategy that can help you do that. And people also talk a lot about multi-model learning. So this within the same dataset, you have these information from different modalities and you learn how they align with each other. But in reality, especially in biomedicine, so it's typically the case will be like this. So although with respect to the same disease, let's say Alzheimer's, you have one cohort just containing the clinical data, you have another cohort maybe with a little bit clinical data but very rich imaging information. And the third dataset still very limited imaging information, but very full sequencing data, let's say. So different cohorts with respect to the same disease capture different information. So it really leads to this mosaic type of setup. And you can think of like, so what is the challenge here if you want to learn from these data? So it's not just the cross-modality alignment. I also need handle the distribution shift across the different size or dataset. You may argue, look, if I build a foundation model, this wouldn't be an issue, but I can tell you it will be a long, long, long way to go. In biomedicine, it is very difficult to build what is so-called a foundation model because of the variability of individuals. It is not like cells. Cells, I think it is possible. There are some promising single-cell type of models that show some success, but still it's just based on single-cell RNA-seq. There are also atoxic, there are also proteomics. I mean, if you want to cover all the modalities, you have to have the same scale on all these modalities. Ideally, they compare with each other. So, but the cell is much more or much less variable compared to human. Okay, so it's, I don't know how many data is needed, and also because we don't know the complexity of the problem space yet. So I think in at least 5 years from now, most likely we're going to still deal with this situation. But still, I'm thinking nowadays people are used to thinking of e-learning this way. Like a row, multimodal learning, or a list like a column of federated learning. But this is the reality. We have created lots of good datasets, but how to integrate them and do joint learning needs new methodologies, and that can lead to new discovery. And I think graph-based approaches can play an important role in this case. We are starting to do something like this. Think about in the future, it will be like if you have a model like this, it'll be like if you have the clinical data, you can guess the genetic profiles and you have the clinical data and genetic profiles, you can almost guess what is the neuroimaging profile. Okay, so that's a pretty, I would say, transformative. So that's, I think that's all I want to talk about and some acknowledgement. And so as I said, so I assume there are lots of students in the audience. So if you are interested in working on biomedicine, so this is my institute and we work pretty diversely with a lot of disease area. And for every disease area, we have a clinician. So we work with them day by day, and certainly we work with computer science people and statistics people. I have computer science students and statistics students as well working on theories. And the last thing is if people are interested in working on machine learning and healthcare, so we are organizing a conference, CHEL, so Conference on Health Inference and Learning. So at Cornell Tech campus on Roosevelt Island at the end of June, and the doctoral symposium is due on March 15th. And me and Irene Chen from Berkeley are the program chairs, and Marzia and Matt— Marzia is from MIT and Matt is from Harvard— are the general chairs. And Monica I forgot her last name from Duke Computer Sciences, chair of the doctoral symposium. So there is still time to submit, and I don't know how long for the submission, doctoral symposium, but you are more than welcome to submit and join us. That's all. I have time to take one or two questions, so any questions, please. Thanks for going to the mall. Well, certainly, several words about nuclear design. You said becoming soft with, you know, nuclear design for a while, actually, it's all kind of, uh, from the media. Nowadays, people use more the one-third approach that you see. I mean, probably, probably the scenario is to say we need all faster speeds from I think I am not the right person to make comments on methodologies on that, but I'm thinking that the way of working, like, uh, recently I believe, uh, Google— I mean, because they now combine Google Research and DeepMind, so they— I think some sort of deal with Novartis and other— that's the right way to do something. So it cannot be just IT company, machine learning company developing the model. So there will be some downstream pipeline to get the chemical, whatever protein, large, small molecules synthesized, and downstream there will be experiments can be conducted with the real whole. Making that a real job. I think that's the way to go. I mean, algorithm-wise, I mean, if you look at paper, because that's all on benchmark data, so performance you can always optimize, but we need to see in the real world. So what's the effect in this? I mean, not too much yet, but I know there are some companies like, you know, Silicon Medicine. So that Russian guy is the CEO. So they seem to have something in the trial according to what he said. On Twitter. So let's see. I will say very soon, probably in 1 or 2 years, hopefully we can see some successful drug. Successful means past phase 3 clinical trial. That's the most difficult part. A lot of the AI-designed drugs in phase 2 but fail in phase 3. So past the phase 3, so that's— but designed by AI. I'm pretty optimistic. So But you need to work with pharma. I have a personal question. I think you talked about for those multimodality learning or patchwork learning, we have those missing modalities and information and stuff like that. And foundation model would not work in that situation. But I'm just wondering, what's the performance? What's the capability of foundation model? No, I'm not saying foundation model cannot work. I'm saying it is too challenging. Almost impossible to train a foundation model. If we have a foundation model, that is not a problem. We just— everything got solved, right? So you can think of— you essentially can predict everything. What I'm saying is biomedicine is too complicated. It's as far as the— I mean, current status, I see huge challenge to build a general foundation model for human health. But you ask about like the performance, I think it will be. I mean, of course, I mean, from the matrix or the patchwork, it's almost like an imputation problem. But imputation is not the ultimate goal. Ultimately, it will be like disease prediction or treatment development will be the downstream task. But hopefully we can like impute better information that leads to more efficient and effective upper box downstream. Okay, let's thank you again for this wonderful talk, and we will have 20 minutes public break, and then we'll go to the next talk. Thank you.