COVID-19 Q&A and CORD-19

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by DigitalOcean. Digital Ocean's developer Cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, team accounts, worldwide availability with a 99 point uptime SLA and 247365 world class support to back that up. Digital Ocean makes it easy to deploy, scale, store, secure and monitor your cloud environments. Head to do co changelog to get started with a $100 credit again.co changelog.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence pract, practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changehaw.com community and follow us on Twitter. We're racticalaifn okay, here's Daniel and Chris.

Welcome to another episode of Practical AI. This is Daniel Whitenack, one of your co hosts. I'm a data scientist with SIL International and and I've got my co host with me, Chris Benson, who is a principal AI strategist at Lockheed Martin. How are you doing Chris?

I am doing okay. I'm safe and I'm well and so is my family so that is a good sign for me.

Yeah, that's great. I'm recording in a new location because we've got a bit of a full house at the moment with my brother in laws being back from college and living with us and so I've transition my studio out into the dining room. So it's been an interesting week in that sense as well.

We're all making adjustments these days. The world is an unusual time.

Yeah, yeah, we're all making adjustments. But yeah, I've actually it seems like I've been more busy work wise since the crisis started than even before because SIL has been making various efforts to contribute in beneficial ways related to COVID 19, including trying to translate the phrase wash your hands into as many languages as we can. Part of that is machine generated and then part of that is just crowdsourced translations and I think we're up to 454 on my last check.

I know I saw you tweet about that a few days ago and so if you're not following Daniel, you definitely should. And you can see the work that he did there.

Yeah.

And those conversations and that work also led to some other discussions with one of my contacts at intel and she pointed me to this other project called Covid qa, which some people at intel started collaborating with with this team from Deepset AI. And I was super fascinated by this project and also interested in potentially contributing because they're looking to up the language support as well. But I had a conversation with them and they've agreed to be on the podcast today. So we've got Timo from Deepset and then also Tony from Intel. Welcome guys.

Hey, welcome.

Thanks so much.

Thanks for the introduction. It's really been a great week and two weeks actually, because this is how we started this Covid QA project. But should I talk a little bit about myself first to begin with?

Yeah, please do if you want to introduce yourself and then we'll ask Tony to do the same.

Okay, yeah. So, hey, I'm Timo and I'm co founder of an NLP startup in Berlin and I would say a total NLP and natural language processing geek. I studied data science and computational neuroscience and then co founded the startup Deepset two years ago in Berlin, which is actually a really great place for a startup. A lot of talent are coming here and also a lot the open source companies are based in Berlin. For example Spacy or maybe you know, Rasa.

Yeah, we've we had Spacy on Spacy on the podcast a little bit ago and yeah, Berlin sounds like quite a place to be a developer. I definitely need to make a trip there.

I was going to say we need a practical AI road trip.

Exactly, yes, totally.

Yeah, it's totally great. Vibrant city. Of course nowadays it's a bit more empty and calm when you go. For example, we have a huge airport that got shut down after the change in government and this field is completely empty nowadays and normally it's full of people and a lot of people doing sports or celebrating there. But yeah, so at Deepset I'm responsible for innovation because we believe that there has been a lot of advancements in deep learning and also natural language processing and this has to be brought to the industry. But also what is really important to us is getting NLP technology to work on German language and for this we are very deeply rooted in open source technology. We train Birch model like these language models that got open sourced by Google and we trained this on a lot of German text data and also open source to this and this is giving us a lot of traction from the community. A lot of researchers, German researchers are using this and yeah, this is just really great time to contribute to open source projects.

Awesome. Yeah, thanks for the intro, Tony. You want to let us know a little bit about your background and how you eventually ended up where you're at now?

Yeah, absolutely. Thanks for having us. So I'm Tony Reyna. I'm a medical doctor and data scientist and I'm chief AI architect for health and life science at Intel. So my primary role is actually taking artificial intelligence algorithms and trying to make them run faster on. Well, on intel products. Obviously a lot of what I've been doing has been in the medical imaging space, so CTs, MRIs, you know, things like that. But been branching out into genomics and particularly natural language processing. So the NLP stuff, Timo and I kind of first met with some of the German work that he was doing, which I like the playfulness of nlp. They name things after Sesame street characters like Bert and Ernie and Elmo and things like that, which is kind of a fun group to be working with when you work with researchers that really love to have kind of fun things to do.

Makes the logos a lot better.

It makes the logos a lot better. Nobody forgets Bert now.

Poor Bert.

Poor Bert. I mean, but Bert's now like a world celebrity now in terms of AI researchers. But yeah, no, it's. It's really great. And you know, I mean, Bert's only been a couple years now, maybe even less that it's been in existence and it's just kind of taken the field by storm. So, yeah, this project with Timo that we kind of looked at, since we were already connected, he just popped up on my LinkedIn page and said, hey, we're doing this Covid question and answer thing. We'd love to get some help. And I said, well, let's figure out how to help out. So that's what we've been doing is obviously we've been really busy at intel, just working, you know, as everybody is around the planet, basically trying to figure out ways we can help with COVID And obviously we're a tech company, so we're not healthcare. We're not going to, you know, be able to go out there and do magic like healthcare providers are doing. But we're going to do what we can and this is one of the ways we think we can really make a difference.

Yeah, you know, this is just such an unprecedented time and as we just for. It's moving so fast that for context, for listeners who, who are tuning in. We're actually recording this on Tuesday, March 31st. And we don't normally say that when we record episodes, but given the topic and given how fast this is evolving, I thought that a point in time was worth having just to, to set the context and then kind of. I'd like to come back over to you, Tony, for a little level setting for us. I know that right now we're at a point where there's 203 countries, areas and territories that have COVID 19 cases as of today. The World Health Organization said 754,000 and change. Pardon me, I'm just to round out the numbers of cases. There's almost 37,000 cases around the world that resulted in death. In the US we're at 163 and a half thousand cases and we, we are approaching 3,000 deaths which we may hit today based on the current run rate, which would here in the US which would put us on the same as 9, 11 in terms of that. So it's a moment in history that none of us have ever experienced. There's nothing, I guess, other than maybe the Spanish flu of 1918 that's comparable in any way. And I know that has limited comparisons. I'm wondering if you can kind of level set, you know, beyond just the numbers that I called out, you know, that are on the websites everyone is following. You know, where we are today, what that looks like from your perspective as a medical doctor that's dealing with this. And then obviously we'll talk about how we're using data to start attending to these problems that we're facing.

Yeah, I mean, I think just from the medical aspect, I mean, I haven't practiced in over 20 years, basically, so I mean, I wouldn't be the best person to answer about kind of the clinical things. But this is the strangest part of this whole thing is that when people ask me how was it going, I all I know is because we're kind of locked down and stay in place and shelter in place, I can tell you how it's going in my house and everything else I get from reading and from listening to the news and things like that. And I think that's what's just kind of curious about this is I feel like I want to be out there and doing things. And even my wife's a retired psychiatrist from the Navy and she was actually even thinking about should I kind of lend a hand somewhere? You know, what can I do? And you know, and I guess that's a Great thing. But that's also kind of where we're at, is everybody wants to help, and yet it's this odd situation where, you know, the best thing for most people is just to shelter in place and make sure that we don't, you know, keep spreading things and trying to get it under control.

That's a great point right there.

Yeah, definitely. And one of the things you mentioned, which really struck a chord with me, is the idea that we're all kind of sheltered in place, at least for the most part. A lot of people are. And we're getting information from various sources. There's so much information swirling around. Some of that's recent, some of that's not recent, some of that's from trusted sources, some of it's not from trusted sources. We're hearing anecdotes from our friends and family. They're hearing things, and things are getting passed secondhand. Could you talk a little bit, either one of you, about what you've seen in terms of the spread of accurate information and the problem related to actually information spread and the virus?

Yeah, I think Timo should go on that, because he's definitely the one that started this on trying to get factual information out there.

Yeah, exactly. So, of course, I mean, social media is quite difficult to dissemble, like, really truthful information. And this is exactly how we started the COVID QA project. It was two weekends ago. There was a hackathon organization by the German government and other authorities. It was actually a huge event. 45,000 people in one Slack workspace.

All virtual.

Yeah.

And all remote and like a beehive buzzing about. And part of this hackathon, we decided to focus on getting factual information. And that's why we looked at official government pages and already saw quickly that if you look at a single government page, there's not so much information. And that the information needs is actually spread across a lot of official pages. And this is exactly the birth hour of COVID qa, where we wanted to aggregate these official sources and make them available and searchable in a meaningful way. And yeah, this was during the hackathon two weekends ago, and there were about 25 developers just jumping on to this project. We were five core developers from Deepset, worked basically the whole weekend through. And with the support of these external people, it was really fun to develop the ui, to develop the back end, develop scrapers that scrape these sites and bring all pieces together. And afterwards, after this hackathon, there's also people now interacting, but also people from external coming and wanting to collaborate, wanting to help, wanting to extend. And this is exactly how we got in contact then with Tony, us following on LinkedIn. And I think this is the most great part of this project to have really like a community that is fast, agile, that is not bound to bureaucracy, that there's no improvement processes, like long approvement processes. It's just we have a situation and we need to work on it, and this could help people actually saving their lives or the lives of their relatives.

That was the nice thing about, you know, with Timo's group, is that Deepset was already set up to kind of do NLP and do it at scale. And so it was one of these things where I knew coming into it that A, they had something already in like the first weekend that you could work with, and B, they had the engineers and they had the data scientists that could make this thing scale. So, you know, they really just needed, you know, resources to kind of come in and help them to make it scale. But they had the machinery ready to go.

So I'm curious before we. And I want to dive into that machinery here in just a second. And terms of like, the end goals of COVID QA and its functionality and kind of some of the things under the hood. But before we get into that, maybe we could just kind of set a foundation in terms of, you know, after you've kind of looked at what, what sources of data are out there, what sources of information are out there, what they're talking about and what people are asking, what. What is the sort of information that people really need to know during this time? Is it symptoms? Is it, you know, best practices for hygiene and hand washing? What are you seeing as kind of some of the main pieces of information that really need to get to as many people as possible?

It depends on. So I think there's two groups that we're getting at with this. The first group is just the layperson that's out there, and they're the ones that are going to hit the tool that's as it exists right now, which is one that basically will sift through a lot of World Health organization and the CDC kind of FAQs, and they're looking for, you know, what's the best way to disinfect, you know, my house, or, you know, what's the best way of, you know, washing my hands or, you know, can I eat this? Or, you know, will this help? That's where it is right now. So that's kind of the first group of people that would be using this and then what I thought was interesting was coming in to add the second group, which is going to be the researchers that want to look for new things. And these are the data scientists and geneticists and physicians and epidemiologists that want to come in and actually do research on Covid and on coronavirus. And so one of the things that Tima and I talked about was there was a data set that was released on KAGGLE by the Allen Institute, by the White House, nih, Georgetown, czi, msr. It was a whole group that put it out called Cord 19. It was the Coronavirus Open Research Dataset challenge. And it's something like, I want to say like 25,000 PubMed articles. So these are peer reviewed, you know, high quality articles that they basically just as a search on like coronavirus and virus and you know, and got all the articles basically. And so the idea was, well, you know, Bert, and all of these great models have, you know, things called like extraction AI where you, you get to do a question and answer system for this large body of articles. And so the question would be, you know, when the KAGGLE thing went out, it was like, here's a bunch of data. Can you find interesting things to do with it? And I thought, well, the first thing you need with a mountain of data is a way to sift through it for actionable, you know, relative data that's actionable. And you know, Tima's group had something called haystack which was like, you know, trying to sift through a haystack for a needle. And I thought, what if we take this, we annotate it using like the Stanford question and answer type of models, the squad models, and be able to actually have researchers give a free tool that researchers can go through the CORD data set and be able to type in random questions that are, you know, things that are not going to be how to wash your hands, but things that are like the beta subunit of the globulin of the such and such, you know, whatever, and it will actually give you a relevant answer and a few articles published, articles that you can actually look to and go through these 25,000 articles and get the real meat of the issue.

Yeah. Also I really like this dual use of this project. And to come back to the question, I think I looked at quite a lot of FAQs for the general public there. The most important information is really informing people how corona spreads and how to prevent the spreading. I think if this is hitting larger cities where people are crowded together, this information needs to be There in the right way and in a trustable way. I think this is really important. And then this dual use for the general public and then as Tony mentioned for the researchers, this will be incredibly useful to speed up the innovation process.

Hi there, this is Daniel Whitenack, one.

Of the co hosts of Practical AI.

And when I'm not working on practical AI I'm developing my own AI applications or I'm training teams at other companies. I've been doing this for over 10 years now and I've trained more than a thousand people. Now I'd like to invite you to my new live online training event called AI Classroom. In AI Classroom I'm going to teach the practical skills I've learned over the years using the latest open source AI technology. You will learn both AI theory along with practical hands on implementations in Both PyTorch and TensorFlow. After attending AI Classroom you'll be able to understand the latest models, implement your own models in code, train computer vision and NLP models, create model inference servers and experiment with state of the art methods like reinforcement learning. AI Classroom is taking place this May. It'll be taking place live and completely online in a high quality virtual classroom.

So no travel is required.

There will be two cohorts with convenient time zones for Eastern and Western hemispheres.

So don't miss out.

Tickets and more information is available at Datadan IO that's Datadan IO and early bird pricing lasts until April 3rd. See you online in AI classroom.

Foreign.

So I guess coming out of, you know, out of that and into kind of looking at the next layer, I'm, I'm wondering. We've kind of talked about what Covid QA is and we've kind of talked about it being based on the cord 19 data set. I'm wondering if at this point now that everyone has a kind of a sense of what you're trying to accomplish, if you could kind of dive in, into specifically what this is that you're putting out there and making available to the public. And as we get a sense of that we'll dive into kind of how it works and what's the technology underlying it.

Yeah, let's maybe also best then separated this tool, use one for the general public, explain this, go into more detail and then also the researcher use where we mainly use extractive technology.

Sounds great.

Exactly. So for the general public it is basically matching user searches, user questions to the questions we crawled from the official FAQ pages. And the technology is based on open source technology, Pytorch hugging face transformers and also our other framework, Haystack, that can basically do question answering at scale. And we started off with the question matching in a very simple way. So basically we just indexed the questions in elasticsearch and incoming queries were then matched with this elasticsearch index, which is basically just a rule based matching. And we thought this is like a good baseline for people to continue working and developing because it's easily extendable to other languages. Elasticsearch doesn't really care about the language so much that has been inputted. And it is also super fast. And during the hackathon and the last days we experimented a lot with BERT based embedd. And if you just. So BERT is like a language model where you stick in text and you get then a vector representation of basically like virtual vec was before for words. It now works on sentence or document level to get like a document representation of it. And these models, they really don't work so well when you just take the embedding and you compute similarities, like for example, with a cosine similarity metric, they don't work so well out of the bo. You need to find ways to adjust these language models to suit your needs. And there we used really nice library sentence transformers. It's from a German NLP laboratory, ukplab Nielse Reimas is also a main contributor there. And this basically takes a BERT model and creates a clone out of it like a Siamese network. So the weights are totally the same. You stick in the query that the user types in and on the other side you stick in the questions that you have already crawled and then you get representations for both and then you can compute similarity metric. And this whole network is trained end to end with exactly these user questions and the questions you have. And this works really, really great. Like the more data you feed into this network, the better it can match questions. And we've then also seen over the course of the hackath that this is the way to go. And we need to extend this also to other languages because the questions from official FAQ pages are phrased in a very official phone. And people who want to ask questions more, right in a colloquial manner or also there are spelling mistakes. And these models cover this by part quite well. And this is why we are actually trying to push in this direction more, more and more.

So I'm pretty curious about that. And Chris could probably guess that I'm very curious about that because of my interest in languages, which we've talked about a lot. So you started talking about the sort of Elk Stack or elasticsearch matching with the index and then talked about bert. So I'm curious. There's of course a lot of marginalized language communities out there that also need this sort of information and are only becoming even more marginalized because. Because they don't have access to proper health information. And I'm curious. So with that sort of flow that you talked about, you have a language model, let's say like BERT or a transformers model. So you could train this sort of model assuming you had data in the language. And then there's this sentence transformers and matching piece that you talked about in terms of transitioning that piece and the training of that piece you mentioned. You have kind of your set of scraped questions and then the set of user questions. Do you need annotation in terms of, you know, matching known user questions to the properly matched faq, you know, scrape data? Is that what you need for that certain piece?

Exactly. This is exactly right. And we created manually, so the core team distributed a set like 30 questions and we manually created rephrasings of these questions to basically evaluate the models. But now we all also implemented a feedback mechanism into the ui. And also we have a telegram bot and we can maybe talk about the telegram bot as well later. And there people can actually give positive feedback or negative feedback saying that maybe the content is irrelevant, it doesn't match the question, or the content is outdated. For example, to inform us that we have to adjust our scrapers in a way and this we hope will scale to other languages. And all the data that is coming from this will be open sourced in this Covid QA repository. And we'll make this also available to other researchers that they can improve a question matching for Covid related questions.

This is super cool. I love what you're doing on this. I guess one of the things I wanted to ask is maybe as people focused on these technologies and we're doing this day to day in our live, and you know, the efforts that we're engaged in in our own projects are built on these data sets that we have. And yet as we look at this crisis and we're looking at the fact that the data set may or may not have everything you need, which kind of alluded to there, you know, in terms of how applicable it is and getting that feedback, what kind of strategies are you thinking of in terms of being able to provide the outputs that maybe some users are needing if they're not in the cord 19 data set inherently? Is that just a limit? Is that A hard limit of what you can tackle, or have you thought about how to extend beyond the limitations of the data set you're working with?

So exactly like this is then the second stage where we have more an extractive QA that takes some unstructured text database, like the Coord 19 data set, for example, and then extracts question. We think that this will be related to researchers, but we could also envision that more text that the general public would be interested could be searchable with this system. The only problem there is that this extractive QA mechanisms are incredibly hard to scale to a huge amount of users. So we would possibly do this for the general public in more like an offline way where we collect questions. And if a lot of questions come up that cannot be answered, we might need to use these extractive QA models to answer them from different data sources.

Yeah, so just to kind of follow up on that, I think that it's worth noting here that I think it's really cool how you've approached this because there is existing sort of question and answer data out there that's from a trusted source. So let's say FAQ pages from the World Health Organization or something like that. So in the first case where you're talking about doing this matching with the transformer models, you're actually matching a user query to a trust source answer for that question because it was posted on an FAQ site. But then in the second piece, what you're just talking about in terms of extractive qa, really now what we're talking about is saying, okay, well, and correct me if I'm wrong, but I think the goal here would be to say, well, the user isn't asking exactly what's on the FAQ site, or maybe the user isn't sort of general public user, but they're a research user. And like you said, Tony, they want to know a very specific question that's not on like a FAQ site, but it is in some research article or it is on some trusted source page. So this is like totally unstructured data. It's just like an article and you want to ask some random question about that article. And that's where you know, this extractive QA comes in. Tony, could you maybe comment a little bit about about that and how this sort of extractive QA model is maybe different from the sort of embedding matching that we're talking about in the other case?

Yeah, sure. So, I mean, the way these models typically work, they talk about a language model, I Usually think of it, it's learning the statistics of a language. So it's effectively like learning I before E except after C or learning that if. The way I kind of go over it is if, you know, if the Dewey Decimal System is this random alphanumeric number system, but you know, you give it to any librarian and they're able to take very different books and be able to basically place them correctly in a library in an ordered kind of structure. And so a lot of these models are doing the same thing. They're basically just looking at patterns of word co occurrences and you know, the statistics of how words, words occur. And what we're trying to do here is for the models that have already been trained, they're usually trained on things like Wikipedia or you know, English language, German language text, which is great. But for these sorts of things you really want to get into the domain specific terms. So the medical terms, the genomics terms, the more difficult and more infrequent terms that be showing up in Wikipedia and trying to learn the statistics of that data set. So they have existing things like Cybert and Biobert, which are BERT models that are built using things like the BioASQ data set. And so Timo and I kind of had talked and we said, what if we took the cord 19 data set which is supposedly, we've looked at it now. There are mostly coronavirus articles, but there are lots of other articles as well in there. The first rule of data science, the data is always going to be dirty coming in. So what we did is we said I'm an MD and intel has a lot of contacts. And so I contacted people from the American Medical association and people that I knew and just basically put out a call and said, hey, could we get some domain Experts, physicians, nurses, PhDs in biosciences, people that are probably in some cases sitting at home. I heard on the radio that that third and fourth year medical students are being told to kind of stay home. And I was a third year medical student. I mean, I know how difficult it is. You want to be there, you want to be doing something. You're incredibly intelligent and you have all of these skills that you've spent the last two years doing. So I just put out a call and we set up a Slack channel and Timo's group had deepset, had this annotation server. So we put up up this Cord data set and it's essentially the Slack channel allows. I think we've got, we've got like 24 in the Slack channel right now. We just, we just started yesterday on the annotation and we've, right now we got over 100 question answers off the data set just in the, in the first day. And so these are things like, you know, I'll read you some of the ones that I'm looking at now from, from the website. You know, how many amino acids are in the SARS COV protein and the answer is 76amino acids. And so this is something where, you know, Wikipedia is not going to be able to get you there. These are directly from the articles. It has a link back to the article that you're talking about. Things like what does the SARS CoV protein activate? Nlrp3 inflammasome. Again, very, very detailed kind of question and answer things that are either specific to viruses or specific to epidemiology or specific to SARS itself or Covid or MERS or any of these kind of similar pandemics that we, that we've seen. And the nice thing for the domain experts is they just log into a website. All they need is a web browser and an Internet connection and as long as they can highlight some text they're good to go. And so we just throw them at it, give them, you know, some walkthrough videos and let them go away and annotate.

I'm kind of curious as you're talking about the specifics of amino acids and such as the. I know that China had done a complete genome of this virus fairly early on and published it. Have you been using that? Has that been helpful and has that informed any of the work that you guys have been doing in the project here?

Not for us. So what we've got so far are just the published articles that were on like pubmed that were pushed into the core data set. But it's certainly something that we could add into things. So kind of the mechanics of how these question and answer systems work is that it's kind of the annotator kind of goes backwards from the model. The annotator reads through the article. So this is a published, you know, peer reviewed article and the annotator comes across a certain fact and they say that's interesting, that's something, you know, that's specific, that might be interesting. So they highlight it and they click question and they make up a question based off of the highlighted text in the article. So if they had the genomic sequence or something like that, they could certainly, you know, if that were in the, in the text article, that's something that they could definitely highlight. And make up a question to. And then the great thing about this is, you know, on Timo's side, when he, you know, creates an extractive AI model for can actually extrapolate and then say, okay, I, I understand the context and the statistics. And so if you threw me a new question that wasn't something that the annotators at every come up with, it should be able to do a pretty decent job at kind of figuring out what it's looking for in that article. So if I put up a brand new article, if I put up a genomics article to this website and ask some questions, it should know where to look for in the text. And that's what's coming back, is the model is not just making up words, it's identifying, it's highlighting the text and saying, here's the highlighted answer in the text. Does this seem right to you?

The Changelog is deep discussions in and around the world of software and it's been going for over a decade. We interview hackers like Chris Anderson from 3D Robotics.

At the time, drones were like predators and global hawks and military industrial and they were classified and super, you know, ten billion dollar things. And we had just built a drone with Lego pieces around the dining room table programmed by a nine year old and it's like, okay, that should not be possible. When a nine year old can do something that is classified, that literally export control as munition with Lego with toy pieces. Something important in this world has changed.

Leaders like Devin Zugel from GitHub in.

The 10 to 15 year range or 20 year range. What I would really like is for if you have like three 12 year olds hanging out and one of them is like, I want to be a firefighter. Another one's like I want to be a lawyer. I want one of them to say that I want to be an open.

Source developer and innovators like Amel Hussein.

I've yet to kind of see applications at scale that don't use multiple languages, that don't have just arcane stories behind why this weirdo thing exists. You know, like, all right, when you open this file, you're going to have have to turn around three times and tap your nose once. It's just the most hilarious stories, you know. But applications are living, breathing, they have cruft, it's normal. So I want to normalize weirdness because that's just how applications evolve over time.

Welcome to the Changelog. Please listen to an episode from our catalog that interests you and subscribe. Today we'd Love to have you with us.

So we've kind of talked about the QA annotation that Tony has kind of helped spin up and really utilizing that expert input from doctors, from medical students, from medical professionals on the cord 19 data set. I was curious to kind of push that back to you, Timo, and see what your thoughts are in terms of. Let's say that Tony was able to get all this annotation in place and sounds like there's a great start on that. How do you see that being integrated into the COVID QA system itself? And maybe how do you see the two sides of the COVID QA system developing?

Yeah, exactly. Good question. So I would say it starts with a scale of data that is needed. So for this question matching we can either use already pre existing external data sets and then this matched questions, maybe one or 2,000 per language. This is enough to get a really good matching system already going. But for this extractive question answering, there's much, much more help needed. Those scales are there like this common data sets squat from Stanford is in high 150,000 question answer pairs are there. And that's why we think it's really great to have external help because this will be the largest level getting this data in a format that we can then use in the frameworks. This framework we actually use, Haystack, which is basically enabling question answering on a larger scale. So normally if you just use a language model like Bert, you basically take paragraph, a small paragraph, like maybe two 3,000 characters and you ask a question and you compute this. But for a large document base, this would be very infeasible. And then you need a two stage process. In the first stage you preselect documents that could be relevant with a very cheap and very fast solution. And then you apply more powerful models like bert. And this is definitely not a real, it's definitely not a new invention. For example, there's a framework out from Facebook, it's called Dr. QA. This is exactly doing this retriever and then reader architecture. But Haystack is doing it in a bit more modular way and modern way with a BERT based extractive question answering system. And we think there's a huge gain in performance. And so we can take these labels that Tony and the collaborators produce and stick them into frameworks to train end to end systems that answer questions on this large COD19 data set.

So I'm curious, you mentioned kind of the scale of the data and as we've sil has been working to get translations in place over the past days, definitely. Translations in the thousands seem within reach. Annotations in the hundreds of thousands is definitely a tough thing, especially when you're relying on experts. But I was wondering if you could could speak to, you know, I know some of these sort of domain adapted models so Cyber or other ones, do I have it right that those are transfer learned from another model? So like if you have a model trained on the squad data set for question answer, which is totally general domain, is it possible to then transfer learn a domain adapted question answer model with the data that Tony's working?

So it's a little bit different. You have to separate a base language model that can just transform text to vectors and then you have to take this language model and adjust it to suit your task at hand. For example, document classification or extraction of named entities like persons and cities. And also question answering, you have to attach a prediction head so another small neural network on top of this language model and then train this whole joint network on this target task. And models like cybert or Biobird, they are just pre trained on a large biomedical corpus. What we are also doing right now is we took a BERT on English data and adjusted it because this process of adjusting this language model to a domain is not that computationally expensive as for example training the whole network from scratch. So we took this network, adjusted it to the coordinating data set and if we take this adjusted chord bert, let's call it like that, and stick in the data, the labels, the question answering labels, it will hopefully perform better than just a plain BERT trained on English.

So you know, I guess how would it be useful, would it be useful to get more annotators involved in this? And if so, what types of skills do you need with annotators to make it useful for them to apply annotations to the data set is who can do that and how many more people would be helpful?

As many more as possible. It was a simple answer. What we're primarily looking for are people that are kind of master's level or above in the biomedical sciences. So I mean you don't have to have a PhD to do this. But I would like to see someone who is comfortable in reading an academic paper and being able to explain it to someone else or be able to, to point out the salient points. Some of the other things that are, that are useful even if you're not one of those expert annotators or you're not sure if you're an expert annotator are just proofreaders. So we've Got on the Slack channel, a sub channel called Second Opinion, just like the medical jargon. And so Second Opinion is where we have somebody that just is looking through the current answers and the current question and answers and going, I wonder if that seems quite, quite right because that doesn't seem to make sense to me. And so they'll put it up and say, hey, question and answer 2, 3, 4 is kind of weird. Can I talk to the person who annotated that or can I talk to somebody else who might be able to give us a yay or nay, whether that's a good annotation or not? So things like that are always useful and I'm getting good response so far but I'd love to always get more. As we talked in the beginning, just people that are kind of at home trying to figure out things to do again. You know, we've got geneticists who might not be doing anything right now. We've got people with, you know, biochemistry degrees that maybe, maybe they're not doing anything right now, you know, or maybe they're, they're grad students or, you know, perfectly, you know, would love to talk to them and try to onboard them for this. If you have an Internet connection and know how to use a web browser, you're set.

I have a daughter who's a third year med student so I'm definitely.

Oh perfect. She contact me? Absolutely. We'll get her going today.

Hey, I'm definitely going to bring it to her attention.

Yeah, we made it 25 at least on this.

Excellent. We made it to 25.

We're good.

And what's the best way for those sorts of people to contact the effort and get onboarded? Is it the best way through the GitHub repo or is there another way to do that?

Yeah, so for the Programmer it's the GitHub repo and then for the domain expert it's probably going to be the Slack channel and we can put the Slack channel up on the side and we're kind of keeping those two communities kind of separated so one doesn't get freaked out by the other. So we'll keep the coders on one side and the biologics on the other.

Side and we can also add that into the show notes, which I think is what Daniel was about to say there. That way it makes it easy for them to slide through to this.

Yeah.

To click on it. Yeah.

Yeah, we'll definitely get the Slack team added to the show notes. So if you're listening and you want to be involved in the annotation. Take a look at those show notes and make sure you reach out. I was curious on the kind of circling back. So that's a great way to contribute from the research user side of things. In terms of scaling up the general public use case of COVID qa, it sounds like there's definitely still some needs there around language support and maybe that has to do with. So maybe Timo, you could mention what would be best to add there in terms, maybe it's scraping more information, more FAQs or adding them. And then also on the development side, where are your biggest needs right now in terms of making Covid QA really useful for the general public? What are some needs that you have? Whether that's front end development, maybe not even AI related or maybe it is AI related.

I'm really great that we are separating labeling process and developing process because it will get super complicated. And I also wanted here to thank Tony that he's taking also a lot of initiative for supervising and pushing the label process because I think I have never heard of open source labeling process and I think this will be a mess at some point and it will be very complicated to interact and also to supervise the quality of the labels. But I think this is the great part of this challenge. We just have to try and have to make it work. So this is really great on the labeling part, on the development part. For computer scientists of any sorts, there's a lot of help needed.

So.

So this is just a hackathon project. It's not like a full fledged professional industry solution. So we need a lot of help. I'm in contact actually with a data scientist also from intel, through contact from Tony and she's working on an intelligent scraper. So right now we have very manual scraping processes for each page that adjusts to the HTML structure. And we would need more intelligent scraper that you can just point to a FAQ page and it automatically extracts questions and the answers. Of course there will be errors, but I mean this is a little bit unavoidable. Maybe we can do a review process there afterwards. So this intelligence scraping will be extremely helpful. And then you also mentioned then that bringing this question matching to other languages, this is something that is personally very important to me because I think this will create the biggest societal impact. And there's a lot to do because for now we have the question matching algorithm with sentence transformers and Bert just implemented for English, but making this work for other languages with multilingual language models, for example, with this cross Lingual language model open sourced by Facebook, for example. This would improve the experience a lot. This on the modeling side and then also a huge help we need on getting this actually to people. And after the hackathon we got contacted by a person, I don't even know his or her personal name. It's the Apache 64 and this person just programmed a Telegram integration. So this service has an API where it can match questions and you can call this API and integrated this into Telegram and this bot is just. Yeah, it's working. Also we integrated the feedback mechanism to feedback the user information back into our system. So this help is really appreciated. But what I think could be important, there would be maybe a WhatsApp integration and maybe even if we extend this really to low resource languages where people might not have access to mobile phones with Internet, maybe have like a text message interaction, but this would be a little bit further away, I would say.

So I guess as we get toward the end here, I want to ask a question and I'd like each of you to give me your perspective since you're coming from two different places on it. As we look, we are in this global crisis which is unique and has stressed all of us and forced us to think creatively in ways that we have just never done before. It's sort of like living in a science fiction novel to some degree. And so as you guys are looking at the role of artificial intelligence at the world within the world and we're looking at suddenly we have this crisis upon us. How do you see artificial intelligence technologies and data technologies impacting our way through this crisis at large? Not just the project that you're in, but its kind of role in the larger world. How has your perspective possibly changed over the last few weeks with regard to that? And what opportunities do you see as the most exciting, you know, in terms of the path forward now that you are involved in this and seeing the results that you are. Timo, you want to go first?

Yes, totally. So I would say this will like largen the way corporates contribute to like solution that is helpful for Everybody. For example, DeepMind has announced quite early solution where they basically analyze the molecular structure with reinforcement learning. And if basically through interdisciplinary collaboration that is now made much, much more possible, less bureaucratic and very fast and agile, I think a lot of great solutions can emerge. And I also think that a lot of corporations give their employees actually some dedicated time to work on these solutions. So like a collective effort of everybody around the world to work on on something that is not directly related to making profits, but to solving this crisis. And I think this is something unique, as you said, it's like a unique situation. And, yeah, it will hopefully make people collaborate a little bit closer on things that is relevant for society.

Yeah. So I love what Timo said about trying to do things that are relevant to society. And I'm not an official intel speaker, but I can tell you that, you know, we're very creative people and we're allowed to do a lot of, you know, things kind of what interests us in addition to our usual, you know, what pays the bills. This has actually been kind of interesting that now the things that are interesting, the whole company and in fact the whole, you know, world, basically now all the extra stuff is now going toward what can I actually do in terms of the AI stuff? I mean, I kind of go back to, you know, we're sheltering in place and we're trying to get through, you know, kind of of the scale without being connected. And somehow AI is kind of helping us to get through the mountain of data that's coming in and trying to maybe focus us a little bit better. I mean, it's designed to be a tool, you know, it's not designed to replace anything. It's just. It's designed to be a really, really nice way of sharpening the edge to figure out exactly what we want to do and what. What's possible. So that's where I see AI coming in.

Awesome. Well, we appreciate you both taking time to join us today. I know that especially during this time, there's a lot to work on. So thank you for taking time and definitely, if you're listening out there and you are wanting to contribute in a positive way, using your development skills, using your AI skills, using your health knowledge and your medical expertise, please check out this project. The links are in the show. Notes, notes. And reach out as well if you're having trouble figuring out how to get connected. There's our Slack team as well, which you can find@changelog.com community and we're happy to get you connected to Timo and or Tony and their team. So just make sure you get connected and contribute. And thank you both Timo and Tony for joining us. Really appreciate it.

Thanks, Dan. Thanks, Chris.

Be safe. Stay healthy.

Thanks for inviting us.

Yeah.

Thank you for listening to this Episode of Practical AI more like this@changelog.com PracticalAI there you'll find our latest as well as lists of our most popular episodes and the ones we recommend. If this show has helped you on your AI journey. Please leave us a five star review on Apple Podcasts. Heart us on Spotify, star us on over here, Cast and tell a friend what they're missing out on. Practical AI is hosted by Daniel Whiteneck and Chris Benson. It's produced by me, Jared Santo. And our music is brought to you by the Beat Freak Breakmaster Cylinder. We have awesome sponsors. Please support them. They support us. Thanks again to Fastly Linode and Rollbar. That's all for now. We'll talk to you next time.

Sam.