Exploring the COVID-19 Open Research Dataset

The scientific engine has really spun up to handle this current situation. As far as I know, there's been more than 4,000 papers released since January on COVID 19.

Wow.

And that the number of papers continues to grow, but more importantly, the number of papers released every day continues to grow. So we're up to maybe more than like several hundreds of new papers a day.

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog do not underestimate the power of the independent open cloud for developers. Yes, I'm talking about Linode. Linode is our cloud of choice and it's the home of changelog.com what we love most about Linode is their independence and their commitment to open cloud. Open cloud means being unencumbered by outside maximizing value for the community, not shareholders. And that's exactly what Linode represents. No vendor lock in open at every layer. If you want to learn more, head to linode.com open again linode.com open. Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changeaw.com community and follow us on Twitter. We're at PracticalAI FM. Okay, take it away, guys.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International, and I'm joined, as always by my co host, Chris Benson, who is a principal AI strategist at Lockheed Martin. How are things down in Atlanta, Chris?

Doing very well down in Atlanta. Got a bit of a cold, so may cough my way through the episode, but other than that, doing great. Spring is sprung. It's beautiful.

Hopefully just a cold.

Hopefully, yeah. We're just crossing our fingers. Took my daughter a couple of weeks. I'm hoping it was her cold. Took her to a pediatrician. The couple of weeks ago we actually had to go in because it kept going and it was a frightening thing to say. Well, it could be strep, could be this or it could be COVID 19. We can't exclude that. And as a parent, that was like, whoa. So I. I've. But I've gotten through that. Just a cold. We're good, Good, good. Rolling forward.

Well, we're surviving here in in lockdown in Indiana. It's actually pretty nice outside. It's mushroom season here. So there's these like wild mushrooms that come out in Indiana just around this time. They're called morel mushrooms and we go every year hunting. So my family has some property that's all forested. No one else is there. So we've found some good times just going out there and walking through the forest and getting outside. So that's. That's been nice.

Yeah, that sounds nice. Whether you find any mushrooms or not.

Exactly. Well, I guess a related topic because it's really the only topic these days affecting us in all of our lives in, in a big way. We've had another episode a couple of weeks ago about the COVID QA system, which is a question answer system related to COVID 19. And they were also using this data set called Cord 19. And today we've got Lucy Lu Wong from the Allen Institute for AI. She's a research scientist there. And we're going to be talking all about the Cord 19 data set, the ins and outs and the story behind it. So welcome. Lucy.

Hi. Thank you, Daniel and Chris for having me on the show.

Yeah, it's great to have you here. Appreciate you joining us. This is of course a pleasure. Big topic. Everyone on Twitter and all around is talking about this data set and how it's being used. So we're really excited to talk about it a little bit more here. But before we do that, I'd love to hear a little bit about your background, how you got into AI related things and ended up at the Allen Institute.

Sure. Yeah. So I guess my background's maybe a little less traditional. I started out more in kind of biomedical engineering and physics and worked in like a host of biomedical medical startup companies creating medical devices. And over time I was kind of doing more simulations and incorporating more data science and machine learning techniques into my work and found that that was what was very motivating for me. So I decided to pursue a PhD in Biomedical Informatics where I focus primarily on biomedical applications of natural language processing techniques and creating models to try to connect these automated methods with the type improvements in clinical care and biomedical text mining that we so desperately need these days.

Yeah, so when you're talking about NLP for like biomedical applications, would are we talking mostly here about like medical records and doctor's notes or like whatever that is and trying to, to extract relevant information from those and patterns and mine those for useful things? Is that the main sort of drive there?

That's definitely one aspect of things. I am also very interested in looking into the scientific literature and trying to extract entities and relationships and useful information out of that body of work. And I think that's what really I'm working on at the Allen Institute for AI or AI 2. I'm part of a team called Semantic Scholar, which I think a couple weeks back, you had an episode about Semantic Scholar. And it's a literature search engine project. And for Semantic scholar, we've indexed 180 million papers. There's a really rich corpus of text to work with, and as part of the research team there, I've created a number of tools and work on a number of projects to kind of understand more about the content of that text. And that's kind of what brought US to the cord 19 dataset, is we have this kind of underlying infrastructure for processing scientific text, and we were asked to contribute some of that expertise to creating the data set.

Awesome. Yeah. And I'm curious. So I'll definitely reference the Semantic Scholar episode in our show Notes so people can listen to that, because I think that provides a really good baseline for, like, the sort of data that you came into this recent work having, which is a really great way to find and discover scientific information and related data across scientific literature, which is amazing. I was wondering if you could comment before we jump into cord 19 specifically. I know we're kind of in a really interesting time where a lot of people are publishing a lot of things about COVID 19 very rapidly. What does that situation currently look like? Is it like, you know, are we talking thousands of papers over how much time? How rapidly are they coming out?

Yeah, I think the scientific engine has really spun up to handle this current situation. As far as I know, there's been more than 4,000 papers released since January on COVID 19.

Wow.

And the number of papers continues to grow, but more importantly, the number of papers released every day continues to grow. So we're up to maybe more than like, several hundreds of new papers a day.

Wow.

And it's kind of intimidating to look at this source of information and kind of see what people are discovering.

So the fact that you have so many new coming in every day, are you refreshing the data set? Is the data set static at some point in time, or is it something that you're constantly updating and refreshing?

Yeah, so I guess maybe folks already know what the Cord 19 dataset is, but it's a kind of a collection of papers about COVID 19 research, including historic coronavirus research. So we have a collection of historic Research. And then we also have all of the new research that is being released daily. We update the data set currently at a weekly cadence, but we are rapidly moving to a daily cadence since there's just so many new papers released every day.

Yeah. So since we kind of went that direction, I was wondering if you could maybe tell a little bit of the story of how this data set came about. I mean, obviously you had data about scientific literature within Semantic Scholar, and you were already doing certain things as relating to, you know, tracking entities or topics covered in those. How did the idea for Cord 19 come about? And I know that there's others involved in this too. So there's Allen AI, but there's also, I think, Microsoft and the Chan Zuckerberg foundation and others. So how did this come about?

Yeah, so the entire project is kind of a coordinated effort by the White House Office of Science and Technology Policy. And I think sometime in early March, a group at Georgetown, the Center for Security and Emerging Technologies, Georgetown, cset, they reached out to us at Allen AI to help coordinate the release of this data set, along with a couple of different organizations you mentioned, msr, Microsoft Research, Chan Zuckerberg, Kaggle was also involved, and the National Library of Medicine, which is part of the nih. So all of these groups were going to come together to essentially create this data set to help, I guess, create text mining and information retrieval tools that could assist medical experts in understanding more of what was going on with the epidemic. And for analytics, the way that we got involved is we had recently sort of created a new pipeline to revamp our research corpus, our open research corpus. So we had a pipeline for essentially taking these paper documents, which are traditionally in kind of a PDF format, not very easy for text mining, not very accessible, and converting them into kind of like a structured full text format where you could run these natural language processing models on them more easily. So that's sort of our major contribution to the data set is the pipeline for both harmonizing the paper metadata that we've collected over the years and also producing these structured full text parses so that we can run our models over that text.

And I know one of the big things that we talked about when we talked about Semantic Scholar was the ability to kind of find relevant data that might be buried in the wealth of scientific literature that we have about a certain subject that is of interest. So when you came to chord 19, there's the extraction of the metadata and the actual content of the paper. But then how do you even go about saying, like, these are all the papers related to coronavirus. I mean, I know and I'm a little bit ignorant on this subject, so you'll have to forgive me. I know that coronavirus is kind of a family of things. It's not just this COVID 19 which is associated with coronavirus, but there's coronavirus associated with the common cold and all these things. So how do you go about saying like this is what we're scoping down our data set to and finding that and along with that deciding what you're going to exclude? I guess as well it's a great.

Question and I think it's a question with very open answers. So what we started with were a couple of trusted sources that we knew needed to be included in this data set. And those sources were a collection of papers curated by the World health organization on COVID 19 specifically. And we also performed searches over PubMed Central, which is a biomedical paper repository run by the nlm, the National Library of Medicine, as well as these preprint servers, BioRxiv and MedRxiv, which were publishing the latest research on COVID 19. And we went out and collected papers from these sources using a set of essentially like keyword searches to make sure that they were relevant to both COVID 19 or the family of coronaviruses in general. Because I think historical coronaviruses like SARS and MERS are also extremely relevant in the current case.

Yeah, I'm kind of curious as you reached out and made the dataset available and you look across some of your partner websites like Kaggle has a call to action and stuff and you're trying to get AI practitioners and data scientists to focus on kind of important questions that need answering for this purpose. How do you provide guidance in that way for people who are going to engage on the data set? Is it something, something where people just grab it and do whatever they want? Is there any kind of organization across teams? You know, there's a lot of human factors involved in this and how was that conceived?

Yeah, so there were a lot of challenges for Kaggle. When we opened the challenge initially, the Cord 19 challenge, there was a set of kind of 10 slightly open ended clinical route questions which were given to the community. And the engagement at Kaggle, the response we've received has been like absolutely incredible. There's been like millions of views on the landing pages. The data set has been downloaded 70,000 times or more. There's been lots of teams that have cropped up and self organized to work on this data set. I think there's a group called Corona Y that's like several hundred data scientists and medical experts who've like bonded together to work on the cord 19 dataset and other coronavirus data sets. And we really want to just offer support to these community members. So there's, you know, a couple of sources of information that we've created to help facilitate these things. So on Kaggle the forums have been super active, there have been a lot of people answering questions for each other, including from the organizations that have created these data sets. We've also established like a discourse to answer questions specifically about Cord 19 the data set. So that's a great place to get answers. And finally, I think for these Kaggle challenges and for these shared tasks, one of the things that we're really trying to do by host hosting these shared tasks is to connect ML experts with the medical community and kind of experts who can judge the answers that are being retrieved and extracted by these machine learning experts and see whether they have practical application in the clinic. So that's been a challenge.

What's up? This is Daniel Whitenack, one of your practical AI co hosts and I hope you're enjoying this episode and staying healthy during these crazy times. I'm working on some pretty cool AI stuff here from my home office, but I've also found that I'm having to get a bit creative and be intentional when it comes to honing my AI skills and virtually connecting with the AI community. If you're in a similar situation or you've been inspired by the practical AI we talk about on this show, I want to invite you to a live online AI training event I'm hosting this May called AI Classroom. In AI Classroom, I'm going to teach you the practical skills I've learned over the years using the latest open source AI technology. You'll learn AI theory along with practical hands on implementations in both PyTorch and TensorFlow. And after the training you'll be able to understand the latest AI models, implement your own models in code, train computer vision and NLP models, create model inference servers, and experiment with state of the art methods like reinforcement learning. AI Classroom is taking place this May. It'll be taking place live and completely online in a high quality virtual classroom so no travel is required. There'll also be two cohorts with convenient time zones for Eastern and Western hemispheres. Don't miss out. Tickets and more information are available at Datadan I.O. that's Datadan I.O. and Practical AI listeners can use the code PracticalAI10 for 10% off. See you online in AI classroom. So, Lucy, you just brought up something that I think is really interesting, which is the sort of interaction between the AI community and the medical community. And I was actually wondering while you're talking about, like, okay, this Core 19 data set exists, but I know I have some AI expertise, but I don't necessarily have a lot of medical expertise outside of knowing like, that I should wash my hands and these other things that kind of the top five that have been going around. I guess I was wondering, as you've got more experience with this kind of intersection between the AI community and the medical community, what has that interaction been like in the past? Has there been much overlap between the AI community and medical practitioners? And then secondly, as we enter into this new Cord 19 challenge, has that changed in any sort of way or been rapidly advancing in any sort of way?

I've sort of worked on the intersection of these communities for a number of years, and I think there's a lot of great collaborations going on. I think a lot of folks in the computing community are incredibly motivated by these very practical questions that need to be addressed. Ways to improve patient care, ways to help with drug development or vaccine development, and questions of this nature, as for kind of COVID 19 specific initiatives. So I can kind of give you two anecdotes for ways that we had annotators or I guess medical experts interact with computing experts. So for the Kaggle Challenge, seems that what is happening is a lot of people are developing different systems, different information retrieval, different information extraction systems, and those systems need to be reviewed by experts for usefulness. So in Kaggle, there's essentially kind of like an army of medical students and other people who are willing to provide their medical expertise and volunteer their medical expertise, who are actually going through and manually reviewing a lot of the extractions that are coming out of these KAGGLE challenges and creating these kind of living systematic review pages with the answers to some of these questions. So if you go to the Kaggle page, you can see these reviews kind of being created in real time and updated in real time as new literature is released. And another thing that I've been involved in lately is we're kind of running a TREC challenge on this data set. So TREC is the Text Retrieval Conference, and it's been a project at nist, which is the National Institute of Standards and Technology, for the last 20 years. And these folks are really good at information retrieval and judging information retrieval systems. And the way that these systems are judged is by having expert medical Annotators review all the results and provide gold rankings of what is most relevant to query and what is the kind of least relevant. So there's a lot of this, like how incorporating experts in the loop, incorporating humans in the loop to kind of bolster our machine learning systems. And that is not something that we're going to be moving away from anytime soon.

As you're talking here, I'm looking at the various questions there that are listed on Kaggle, the tasks to go answer and kind of extending this thing about this collaboration between the AI community and the medical community. The questions themselves, where did they originate from? How were they decided as the important questions that we could all give a shot at going and answering with the data set? How was that?

Yeah, I may be wrong, but I believe this set of questions originated from the White House Office of Science and Technology Policy in collaboration with Kaggle. And you have to understand. So this challenge and the data set, during the early days we literally had just a few days to turn around this data set, put it out there and kind of publish this challenge. We wanted people to start looking at this as quickly as possible. So there were, you know, a lot of the questions that you see on Kaggle right now are very open ended, be interpreted in different ways. And as time has gone on, as we've learned in this last month, there's actually some of those questions are more useful in clinic, some of those questions are less useful. Like clinicians already know the answers to some of these questions. So now as we move into the second month of this challenge, there will be a new batch of questions released to kind of motivate new work and kind of questions that have not yet been answered by the community.

Yeah, so I'm curious with that. It seems like you could have various bottlenecks in this situation and one of those I think you highlighted is this sort of useful interaction between medical practitioners and the AI people that are trying to do something with the data set. So I was wondering, do you have a sort of healthy community of medical practitioners that are very deeply involved in kind of looking at what's coming through Kaggle or these other teams that are kind of self organizing? Because I know that's one. Having worked at a non profit for a bit, one of the things I've seen is like, oh, people really get behind like the social good like challenge and they work on it like on the hackathon on the weekend and then the project kind of dies. So how, how are you kind of, what are the ways for people if I want to step into the some cord 19 related work. What are the ways that I can kind of step into that but also get connected with the right sort of medical people to make sure that what I'm doing is useful and not just like an kind of interesting weekend thing.

Yeah, of course. I think like now that COVID 19 has sort of taken over all of our lives, a lot of people are feeling very motivated to do something in this direction, contribute their skills. And I think I mentioned some groups earlier, groups like Corona Y, which is self organizing to analyze this type of data. And that's a group made up of data scientists, machine learning experts, medical practitioners and so on. And so it's like a good place to get feedback on one's work. And Kaggle forums, similarly, I think there are a couple of threads out there essentially continuing this discussion of how to connect to medical experts, how to verify results and so on. A lot of people have, you know, taken it upon themselves to build systems. There's tons of systems out there for searching COVID 19 for extracting information out of cord 19. And you know, to be perfectly honest, not all of those systems are going to be like highly usable or used by any kind of clinical audience. But we as a community need to essentially figure out which of those systems are most promising, figure out where to expend additional energy, like more development time and so on. And that's where our annotators come in.

Going back to what you said a moment ago, you have put this together so quickly and gotten it out there and the whole world has kind of dived into it, the whole world of data science at least. And you know, that's very different from how most communities form. And I'm kind of wondering with hindsight, totally recognizing that you had no choice, you had to get it out there. And you did a fantastic job with what the kind of pressure that you were under. If you could go back knowing what you know today, what are some of the things around community that you would like to have done and that maybe going forward, as you're looking at like the tasks and how to move us going into the second month, how to move us in the next set of directions that you need people to take? What are some of the ideas that you're planning to implement there to kind of evolve this process?

Let me try to unpack that. No worries.

Any way you want is fine.

Sure. I think we've learned a ton over the last month. Speaking with some of our collaborators at kaggle, like Anthony Goldblum was mentioning how they've really fallen or Kaggle has fallen into this place with this challenge where they're kind of in new territory. Like the type of challenge that the Cord 19 challenge is, is very unlike most of the challenges hosted by Kaggle. And there's this very open ended nature of it. We're like trying to discover answers, but there's no sense of like what a gold answer is. And they reacted to that in a very like, kind of wonderful way by essentially harnessing these medical students as a resource to make judgments on people's extractions and kind of putting an effort there where it seemed like the results were most useful or were going to be the most useful. So I guess that's one thing which is trying to figure out as early as possible where the most useful results are and putting in additional effort there and maybe even abandoning things that are not worth pursuing. And then I don't even remember what the second half of that question was.

No worries. Things that you might be thinking going forward, but no worries, we can come back to that.

That's right. I mean, I have lots of thoughts on that as well. So certainly we're going to be supporting Cord 19 for as long as it makes sense to do so. Certainly until the epidemic seems to wind down a bit. There have been lots of requests for additional features and additional content. So that is one of our priorities. Additional content comes in two forms. One is simply providing more faithful parses of the papers. So right now, first things might be including things like inbound and outbound citations, tables and figures, other places where the answers might be. And these have been requested by a lot of folks. Another is content in the sense of more papers. So one thing that we've been very grateful for is that a lot of publishers have made their COVID 19 articles open access. And by making them open access, they've allowed us to release a data set like Court. But the fact is, if you look kind of at the data set and at the papers that are being cited by the papers in the data set, there's actually a lot of content that are outside of this direct core set of articles on COVID 19 and coronaviruses that are also very relevant to the content of the data set. So it would be great if we could work with publishers or they could work with us essentially to provide additional content that could be useful for discovering information about COVID 19 and its treatments.

Yeah, so it's, it's like you've amassed this kind of central hub, but from that hub of papers, obviously Those papers cite other papers and those papers cite other papers and there's other related work and you can kind of go down a rabbit hole. I know we talked on the Semantic Scholar episode about this sort of graph of, of relations and that sort of thing as papers cite each other. I mean, there is a wealth of papers already in the data set. What is the kind of current size of the data set? And you know, you mentioned a few different sources could give us a sense of kind of the descriptive statistics of the data set at this point, I guess.

Sure. Probably should have started with that.

Yeah.

So the data set currently consists of more than 50,000 papers and approximately 40,000 of those papers have full text content available. And these papers, as I mentioned, come from a diversity of sources. There's a list of WHO COVID 19 papers, several hundred of those that have been curated by the WHO, and there's preprints from bioRxiv and medRxiv. That's numbers in the thousands. And actually the vast majority of papers come to us via PubMed Central. And this number will actually continue to grow because many publishers are now depositing all COVID 19 content into PubMed Central.

So, Lucy, I'm kind of curious, so recognizing that your responsibility has been putting this together and getting it out into the world, I imagine that you've probably talked with various teams or at least observed some of the efforts and work that's going on. And so I'm kind of curious, what are some of the more interesting or innovative or you know, pick your adjective of the efforts that you've heard of and kind of gone. Wow, that's a pretty cool way of approaching this. Any stories to tell us on that?

I think folks have done a really great job diving deeply into this data set. There's so many different kind of search engines that have cropped up over this data set. If you go to the dataset landing page on Semantic Scholar, there's like a list of maybe several dozen that we've heard of and are enumerating there. I'm sure there's many more that we aren't aware of. People are pursuing lots of different technologies for these kind of search engines. Some are kind of using the latest kind of state of the art transformer models, neural models for ranking. I think Covidex from Waterloo and NYU is using kind of the latest Scitee 5 model. Very cool stuff. And some of the search engines are actually using very traditional methods using Lucene or elasticsearch and focusing more on how to search and filter using entities or other paper features. And we've kind of a funny thing that we heard from Kaggle is that for many of the Questions on the Cord 19 challenge, something simpler methods, the more traditional methods have actually worked better for kind of extracting answers. And this came as a surprise to me.

So I have a follow up on that. And that is like for working better, you know, kind of putting that in quotes. What does that mean? Who out there is looking at the various results that are coming back from teams and making those evaluations and kind of. And obviously you've already said that you'll be steering in the new tasks on Kaggle, learning what we know, who's making those evaluations and the decisions associated with that to keep everything focused.

So I think this is primarily work that has been taken on by organizers at Kaggle and kind of of medical students that they've had come in and evaluate some of these answers. So really like they put in a ton of effort in curating these results. And I think as for like the metrics that we use to judge these results, I think currently like they're mostly kind of information retrieval based metrics of success.

Yeah. To give people an idea, I'm on the Kaggle website and just looking at some of the tasks to maybe for those listeners out there that don't have a good idea of the kind of scope of these, there's information or some of the tasks that are listed are what do we know about COVID 19 risk factors? What do we know about vaccines and therapeutics? What has been published about medical care? And if I kind of dive into each of these, there's a number of submissions, but things like, you know, Cord 19 analysis with sentence embeddings, COVID 19 literature clustering, full text search of research papers. And so you can kind of already see. And here's another one, Bert Squad for semantic corpus search. So you get a sense, I think of what you were talking about, Lucy, that some people are kind of going after these sort of transformer based, maybe extractive QA sort of things. Others are maybe using full text search capabilities that have been out there for a while, like elasticsearch sort of capabilities like we talked about on the previous episode. Also if I'm looking at the data just to kind of have something in, in people's mind, I'm seeing, of course you have the categories of source like the bioarchive and that sort of thing. But if I'm looking at the individual papers, I can see the abstract of the paper and the body text and if I'm looking at, you know, the, the body text of this one, I'm reading things like to assess the effects of truncation of the poly C tract on replication, blah, blah, blah, which some of that doesn't mean a lot to me, but this is the sort of data that's in there. I was wondering as a complete noob to, you know, a lot of this medical terminology, what are maybe some good ways. I know you've got this kind of covis from the Allen Institute, which is helping kind of explore some of the looks like the genes and cells and diseases and chemicals that are connected throughout the data set. What are some good ways of kind of onboarding into the cord 19 work for people that might be sort of new to medical terminology? Are there ways to kind of of pick up some of that or explore some of it and form some of those connections in a reasonable sort of way?

The domain knowledge?

Yeah, I think domain knowledge is one of the greatest barriers for working on this data set. And thanks for mentioning the Covis project. That's a tool that was released by Allen AI for exploring this data set in a slightly more meaningful way. And for that tool, uh, we essentially ran models to perform extractions of these entities from the text, uh, entities of different classes like drugs, genes, diseases, phenotypes, um, things of that nature, and created a visualization to allow you to browse the relationships that are most prevalent between pairs of these entities. So that's a great way to kind of explore what's in the dataset. There's also just like exploring the articles. We have uh, kind of a core 19 explorer to help people that. But in general, I think unless you're willing to spend sort of a couple years of your life in medical school, it is very hard to understand what some of these terms mean. Certainly knowing what class or category of entity is being mentioned is important. So knowing something is a protein, knowing something is a receptor, knowing something belongs to a particular biological pathway. These are kind of key for gaining an initial understanding of what is being said in a text snippet. But that's also why we need medical experts to assess the act actual utility of some of these extractions.

So I'm wondering, I happen to have a daughter who is a third year medical student and I've told her very recently because we had the other episode about this and stuff, but she hadn't been aware of it. Is there any need to connect with medical schools? Has anybody kind of taken that on to try to, to, to gather those Together and stuff. Because it's been, obviously there's been an enormous effort in a very short amount of time totally recognizing the constraints of the reality that we're in today. Is that something that y' all are kind of thinking about in terms of going forward maybe for stage two or whatever you want to call it?

Yeah, absolutely. I'm sure your daughter knows that. I mean, I don't know if she's still. If medical school is continuing as usual, but I think during the third and fourth years, you're mostly in clinic. So I know of a lot of medical schools where there are these kind of more senior medical students who really want to contribute how they can, but really aren't able to be in clinic at this moment.

Yeah, they've been kicked out of the ER for her. You know, they're working with the health department locally, and I think that kind of alternative work is really common right now for advanced medical students.

Yeah, exactly. So for the TREC task that I was mentioning that we're hosting, we actually are enlisting medical students from a number of institutions from the Organ Health and Science University and University of Texas and University of Washington to kind of help with, provide annotations on some of these extractions. So I think depending on where your daughter is or where some of these medical students are, there's probably going to be other initiatives like this one that really need their help. So I would definitely encourage anyone to look out for that.

I guess I have one more follow up to that that you just mentioned. Do you think, like, recognizing that and recognizing that we're going to be past this moment at some point, this kind of very unique moment in our history. But just as, you know, the kind of. The widespread introduction of like, Open Source Software really changed, you know, industry itself from being highly proprietary to being, you know, open source, became not only a part of business models, but even underlying part of a lot of commercial software that's out there. And it fundamentally changed how that works.

Work.

Do you think this is a moment just because COVID 19 passes us, you know, and we get past this, that. That maybe there are other challenges, whether they be things that we've been dealing with a long time, like cancers or. Or new things that may come, that this may fundamentally change how we attack, you know, really hard medical challenges with AI and that integration with the communities that has happened out of necessity.

I am certainly hoping that to be the case. So we've definitely seen what do when they come together for a month or two. And it's incredible. There's so many people being engaged and building interesting tools and useful tools. I think there's a couple of maybe things that I'd love for us to be able to extend into the future. One thing is definitely publishers coming together to release more open access content on really kind of important topics such as COVID 19 and then the community coming together, especially crossing boundaries, crossing boundaries between computing, the computing community, the medical community and policymakers to really build something useful.

So I'm curious, a little bit of a follow up I guess to that, that question as well. I know there have been things as I've worked on related work with SIL, where I was thinking, oh, if I would have done this prior to this crisis, like I would be able to do something better than what I'm able to do now. In hindsight it's easy see those opportunities. I'm curious on your side with your own research and work, I mean you were kind of, I'm assuming you're working on various things related to Semantic Scholar prior to this crisis and now your, your heads down working on, on cord 19 and, and getting this in shape. What are you interested in exploring in the Future? Not necessarily Cord 19 related, but how has the, this whole process shifted what you want to work on in your own research in the future?

Yeah, so I think this brings up a slightly earlier point which is we really became involved in the creation of this data set because we had at Semantic Scholar built a bunch of infrastructure for scientific papers and a collaborator of mine, Kyle Lowe, and I had also been working for the past nearly year on a way of essentially creating a full text extraction pipeline for some of these papers that we are using COVID 19 today. So a lot of this was infrastructural work. It's not particularly glamorous, but it is really important and it really became more important in light of what happened in the last few months. I guess one thing is infrastructural improvements can be really important, even if it's not particularly sexy and then kind of going forwards. There's certainly things I care about besides kind of creating data sets of papers. And my research focuses on making scientific literature and making this content more available to biomedical researchers and more understandable. And as you mentioned before, there's so many entities, so many kind of like very domain specific words and relationships that exist in the biomedical literature. And even for someone who is a domain expert, some of those terms can be very hard to parse through and understand. So a lot of my ongoing projects are trying to create systems that understand particular types of relationships, for example those that understand drug drug interactions or can mine them out of the literature. Those that can understand medical images better. These are the types of projects that I am hoping to continue to work with on in the future.

Awesome. We're for sure going to have the links to the main dataset website in the show notes along with the Kaggle Challenge and the various other projects and groups that you've talked about. Really appreciate you coming on the show and describing a bit more about the data set, how it came about, and your own work with it. Really encouraged by the work that the Semantic Scholar team and collaborators are doing. And thank you for your hard work on this and taking time to talk to us.

Us. Yeah, thank you so much for having me. And we really hope that other folks are encouraged to kind of contribute and become involved in this project.

Yes, please do.

Thank you for listening to this Episode of Practical AI more like this@changelog.com PracticalAI there you'll find our latest as well as lists of our most popular episodes and the ones we recommend. If this show has helped you on your AI journey journey, please leave us a five star review on Apple Podcasts, Heart us on Spotify, Star us on Overcast, and tell a friend what they're missing out on. Practical AI is hosted by Daniel Whitenack and Chris Benson. It's produced by me, Jared Santo, and our music is brought to you by the Beat Freak Breakmaster Cylinder. We have awesome sponsors. Please support them. They support us. Thanks again to Fastly, Linode and Rollbar. If you and your organization could benefit from speaking directly to all the AI practitioners out there, you should sponsor the show. Podcast advertising is one of the most effective ways to spread your message in an authentic way. Plus you get the added bonus of supporting something you love. That's all for now. We'll talk to you next time.