The AI doc will see you now

When people think about massive amounts of data, they just say, oh, I just want as many images as possible. There's a saying by my CTO that I love. He says, guys, what I want is not to have the most images. I want to actually train most effectively on the least amount of images if I can do that. And how do you do that? By selecting the right cases for optimization. So in medicine in general, annotation costs are sky high because you need physicians for that. So. So you really want to pick your battles and the case selection methodology even before saying about the biggest, which is something that I'm proud of. But he's saying like you shouldn't be proud of that. I would ideally want to have the smallest annotated imaging database, but the most effective one. So the case selection how do you pick the right cases to optimize the learning performance as much as possible?

Bandwidth for changelog is provided by Fastly. Learn more at fastly.com Our feature flags are powered by LaunchDarkly. Check them out at launchdarkly.com and we're hosted on Lino Cloud servers. $100 in hosting credit@lino.com changelog hey friends, this episode of Practical AI is brought to you by Kodish, a podcast from the team at Heroku that explores code, technology tools, tips and developer life. There's tons of great conversations on the codish podcast, so I would encourage you to check it out and subscribe. But in particular I wanted to bring to your attention two episodes, episode 98 and 99 or Julian Duquet explores the ethical and technical sides of deep fakes. The rise of manipulated pictures and video and other forms of computer generated media are able to cause uncertainty and doubt in what we see and hear online. So how are we able to use these tools for good if at all? Here's a sneak peek.

Let's say we want to do a deep fake of my voice and we train the model and we have enough data and everything. This will be also able to imitate my accent, for example, like how I pronounce English and the strong pieces of my accent or is not not there yet.

It really depends. If there would be a person with similar accent on the input then it would be fine, but it's kind of cheating. You, you could, you can think it's cheating because we're reusing accent of a different person that's similar to your accent. But if it would be like in like an American native speaker or a British. A person with a British accent or like whatever diff whatever, other, other accent, then it will kind of be a mixture on the output. So we're not there yet in terms of converting accents. It's a little bit more difficult than we initially anticipated because, like, when we started the company, we thought it would be, you know, we'll kind of solve it in a year or something. But then it turned out that, oh no, we're here for much longer.

Check these episodes out. Links are in the show notes to both episodes or head to Heroku.com podcasts to listen and subscribe. Again. Check the show notes for links links or go to heroku.com podcasts.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com, and follow us on Twitter. We're at Practical AIFM.

Welcome to another edition of Practical AI. We are the podcast that makes AI practical, productive and accessible to everyone. My name is Chris Benson. I am a principal Emerging Technology strategist at Lockheed Martin. I can't say my own title and unfortunately Daniel Whitenack, my illustrious co host, is not able to join today. We'll miss him, but I am pretty excited because we have an interesting show ahead. With me today I have Elad Wallach, who is CEO of adoc, joining me today and looking forward to learning all about ADOC and that industry. Welcome to the show, Elad.

Yeah, thank you, Chris, for having me.

So I guess to get us started, if you would kind of let us know about yourself, who you are, how you got to the point where you're at, and a couple of moments, we know you a little bit better. Uh, I'll ask about your company, but looking forward to finding out kind of how you arrived, where you're at right now, before we actually get to the topic at hand.

Yeah, so again, thanks for having me. And I'm really excited about this Practical AI podcast because I come from the AI world, so my background is I'm an Israeli, so everyone in Israel, we also serve in the military. I've had the pleasure of being in what's considered the most elite technology unit at the Ministry of Defense, a unit called Talpiot. And as part of that, I headed the AI at the Israeli Air Force. Okay, so no healthcare, no knowledge about CT imaging or X ray, and to be honest, I didn't even know what's the difference when we started back in the day, but what we were really passionate about is utilizing the skills we learned, which is basically applying artificial intelligence to real operational problems, solving real world issues to the healthcare space.

So before you even dive on, you started with something so cool that I don't want to let that get by. So when you talk about that unit with the Israeli Defense Force, could you tell us whatever you can a little bit about that? Because I've just never had someone say that before. It intrigued me. And then I want to keep going. With your own history, but you can't start with something like that without talking a little bit about what it does.

Yeah. So I'll say what I can.

Okay, fair enough. Say what you can. And I get that there's a point where you can't. I work for a defense contractor, so I understand that.

So I'll start with the grand vision of the Topio program. So in Israel there was a big war called the Yom Kippur War. And Israel was technologically surprised. And that was the key. So Israel decided strategically we need to always have the technology on top. And their idea was, let's take 18 year olds right when people are in their prime time and let's make them like the innovators of this really means defense. So let's take the brightest 30, 40 people every year, screen them very rigorously and give them a lot of benefits, but allow them, I would say, to serve their country. So when I was 18 year old, I was screened for that program and as part of that I had to sign up for a decade at the Ministry of Defense. That's a service you have to perform, but the benefit is that they basically teach you everything there is to know about the military. So we kind of visit all units and they give you this very rigorous training you in both technology and leadership, and then they catapult you to leadership positions across the Ministry of Defense. And I was very fortunate to be at the Air Force in a time where AI was just starting to pick up in terms of applications, computer vision, NLP and what can we do? And it's just so much fun to look at the academy and all the innovation and trying to apply them for real projects that help save lives.

Sounds good. Okay, thank you for at least kind of getting us going on that. That sounds like a pretty cool start, pretty cool way of getting into this. So pick up now that I've rudely interrupted you. Go ahead and pick up and tell us about yourself from that point.

Yeah. So when I finished the service after 10 years, I've met my two co founders which were both part of that program as well. One headed the department for the special forces. One had been in kind of the Israeli nsa. And we were just very passionate about the healthcare space. We saw that there was this massive pain point with increasing amounts of data. We can talk more about that. But we needed to know how we can use the tools that we got through our service to build a company. And we're passionate about healthcare space basically for two big reasons. The first is that it's really a deep tech area. So we didn't want to compete in three guys in a garage. Right. We wanted to build something that is higher barriers, very defensible. And the other reason is more an emotional one. We just wanted to feel that direct connection. I think there are a lot of companies bring value, but to be honest, I love the fact that once a day we get a text message from a physician that tells us, hey guys, you just helped save somebody's life. And I think that's what we were looking for when we started the journey all those years back.

Okay, I guess if you can tell us a little bit about what it is in terms of the company that you've started, what the vision is for that. Where are you going at a high level? Tell me what you're doing.

Yeah. So we are utilizing artificial intelligence for radiology. Radiology, for those who don't know, is the science and art of interpreting medical images. CTs, X Ray, MRIs. And in recent years we've seen a massive overflow of information. You have cheaper scanner, you have higher resolution, you have less radiation. So you do this more and more. And the problem is that each and one of those images has to be interpreted by an expert physician called a radiologist. Unfortunately, they are stagnant and have been stagnant for years. In terms of number of radiologists, the situation today is almost absurd. Mayo Clinic did research showing that today they have about two seconds to read an image. Think about a human has to look at an image, massive image, we're talking big images, two seconds to detect if something is wrong with you.

Right.

Obviously it's an error prone process. Obviously it's very difficult and obviously we're starting to get delays. And because radiology is so critical to all of healthcare, you know, now you do, you know, even for stubbedo, you get radiology. Right. So every delay, every error has massive downstream implications and we see it impacts, you know, stay and errors and patient lives at the end of the day. So yeah, pause here if, see if that makes sense.

It does make Sense. It makes perfect sense. And it's interesting without specifics. I have heard about AI being used to do radiology in terms of the interpretation, you said. The one thing, though, that I'm just curious about. It's a totally. A side note, but I can't help but ask. You mentioned that the radiologist has like a couple of seconds to make that, and is it just too costly, given that they're doing rounds and they have many patients or something to spend the time, but, you know, they're just expected to obviously see it quickly. But I'm just kind of curious what's driving the metric about how little time they have to make that, given the fact that that human in the loop right there, as they're looking at that image, has a huge downstream, you know, cost to getting it wrong. Is it just too few radiologists for too many patients in general?

Exactly. It's kind of supply of demand of images. And another big factor that comes in is more data per image. So if images were small, so like the CT exam in the past, they had, you know, 50 images, something like that. They were talking about thousands for a single patient, for a single exam. So that obviously factors in as well. Right.

I don't think I realized that. I just learned something. I was thinking, you know, when I'm in and they throw two or three up, you know, and all that, but they have a whole lot more than that to deal with today, is what you're telling me.

Yeah. The analogy I heard from one of our customers, I love that. It's like phone cameras. Right? So first of all, you have much higher resolution. Right. And you have all the different filters now, and you need to look at all that. And the other thing, you know, everybody's doing imaging today. You know, you have selfies all around. So CP images are like, you know, are like phone cameras now. Everybody's doing selfies. And now you have a lot more resolution.

Gotcha. It sounds like a good problem to scale with automation. That's what I have to say about that.

Yes, yes. Or augmentation.

Fair enough. So why don't you tell us a little bit about that augmentation. What is the business model that you're addressing here? How do you fit into that process?

So basically our AI is a safety net to radiologists. So the AI constantly runs in the background, nobody needs to activate it or anything like that, and searches for certain type of findings that require prioritization. So the big problem we try to solve, or at least the area we started with, is critical Care because you have so many patients. And the big thing if you have more data and less people, is that those patients can get treated much later. Our AI runs in the background searches for those critical findings. Stroke, bleed, pulmonary embolism, spine fracture. If we find something, we basically alert the whole care team, the radiologist, to the existence, to the presence of a critical finding. And this can shorten the time to treatment significantly. We're talking about, you know, even in ED settings where time is really fast, we're talking about an hour or two. But for elective care, it could be sometimes, you know, days of reduction in terms of time to treatment.

So what, what does that look like? What, what is the relationship? So you're doing the interpretation at this point with the AI, what does that look like?

We are basically flagging cases for the radiologists still. The human is doing their interpretation. We're just helping him screen through the data, see where areas the radiologist should focus on first. Let me give an example of what it means without us or with us. Let's say you have one example is the emergency department. Typical emergency department, especially at night, very busy, you have dozens of patients coming in and you really don't know which of them are really critical or not. We have a patient, let's say, fell down a few stairs, coming to the ed. They're even doing CT imaging for that patient, trying to see if that patient has a brain bleed. But unfortunately, today, just because of the overload, that patient could wait hours before anybody interprets the exam. That's the key bottleneck, looking at the exam and seeing if there's a bleed. So today you could wait a couple of hours. And we're talking about very time sensitive pathologies. Right? Time is brain. We all heard that. So we can basically reduce that time because we say, we look at the image, we say, hey, there is a brain bleed here. Let's alert the radiologist. You should definitely take a look at that patient. Even though that patient is 15 year list, that should be number one. Take a look right now and get the patient the treatment they need.

It's interesting just to throw in as you're describing this, I have a very specific context that I'm applying it. I have a stepdaughter who is a fourth year med school student and she is right now working in an emergency room at a hospital near us. And that, that's what she does. And so she relays a lot of her experience. You know, because I have no background in that, she likes to tell me these things. So it's very interesting. I'm envisioning how this fits into her world as she's been describing to me over time. And she has talked about the fact that getting the image results can be a big delay in the ER when they're trying to move quickly, especially on a, like a Saturday night or something when things are coming in at a very fast pace. So I guess you. So you still have the radiologist directly involved in the process. Is it essentially just speeding up their process to draw their attention directly to what they need to focus on? Speed that up.

Exactly.

Okay, but you're anticipating, at least at this point in time, the radiologist is always the key to the picture still, along with the technology that you're providing. The two are partnered essentially.

Exactly, exactly. And I think that's key, especially this day and age. And right now I think radiologists have to be involved. It's a. Because it's really necessary. Because at the end of the day I wouldn't want to have AI interpreting my full images. And you know, I'll add one more thought to that. The AI is really good. I trained him to be the best brain bleed detector I can. But we have thousands of diseases that we can have. And where radiologists, where humans are really good at is that comprehensive picture. They know how to aggregate information, they know how to look to this. Something here looks wrong. That's where humans are very good at. We're not replacing them in any way, shape or form. We're just providing this augmentation so they could focus. If there is a critical finding, let's focus on that right away. Let's make sure that we treat the patient very fast for that downstream impact.

We deserve a better Internet and the Brave team has the recipe for bringing it to us. Start with Google Chrome. Keep the extensions, the dev tools and the rendering engine that make Chrome great. Rip out the Google bits, we don't need them. Mix in add and tracker blocking by default, quick access to the Tor network for true private browsing and an opt in reward system so you can get paid to view privacy respecting ads. Then turn around and use those rewards to support your favorite web creators like us. Download Brave today using the link in the show notes and give tipping a try on changelog.com.

Okay, so that's really fast. Fascinating. We've kind of talked about how your technology fits in to that. You know, in this case the ER with the radiologist having to interpret that. Before we dive into the, the technology side, could you dive us kind of specifically through the process, you know, maybe describe a little bit about the radiologist has, you know, we've talked about the radiologist trying to interpret and having to do that very, very quickly. How is the radiologist actually interfacing with your technology? What does that feel like to the radiologist and how does that change their process? If you could describe that and then after that I'd love to dive into the specifics of the AI, whatever you're willing to share to understand some of the cool underlying models that you're using to make all this work.

Yeah. So I'll start with an anecdote. When we started a company, we had a conversation with radiologists, very known neuroradiologist and we talked with him about the workflow and he told us like, guys, look, radiologists are so bombarded with information. You have so little screen real estate, you know, you really can't make us press buttons and, you know, click things. It has to be seamless so, you know, reflect back to Italy, you know, I get it. So great insight. So no more than one button click for the use of the product. He told me, no, Yeah, I don't think you got it. No button clicks for the use of the product. It has to be fully seamless, automatically appear whenever I want it. And that was kind of the challenge we faced when we started the company because there really are, you know, imagine that environment of the person that has 50 patients waiting for critical, like getting phone calls, having to interpret a lot of diseases. You really can't disrupt anything. So what we want, we wanted to create is a very seamless workflow for them. So it would appear automatically in their own systems. They're using a work list. So it would just appear as a flag in the work list, like a small pop up things that are really easy for them. The challenge was, however, we thought, well, great idea. But the problem was that in healthcare there are so many legacy systems. So imagine a hospital using systems, sometimes 10 years old systems.

I noticed that every time I do have to go to doctor's office or hospital or something, it feels like a step back in time if you're into technology.

Oh, exactly, exactly. And trust me, it's really no fun to integrate into all of those legacy systems.

I could imagine.

So that's basically the challenge we had. And a big focus for the company is actually we're an AI company, but half of my engineering is on creating a very seamless workflow. And what we did, we basically created this very cool integration engine that allows to integrate directly into those different systems. So whenever radiologist gets up in the morning, he starts seeing our results popping up within their own workflow, the same system they're using. So if they were using a specific work list in a certain viewer, we'll just start popping up within their existing systems to kind of highlight the findings. So what they do is they just do whatever they were used to doing to the state. So opening a work list and then clicking guess, we're just small flag appearing there saying, hey, you know, this is a critical case. Small, small, like priority flag. Look at that first.

Very, very seamless to accomplish that. So it sounds like you're integrating not only at a software level, but also at a, at a hardware level to some degree. Or am I misunderstanding that? Because you have this hot new AI that you're trying to integrate with this 10, 15 year old system that the doctors are used to using. How does that look, how does that merge? You're talking about kind of moving into the workflow, but how do you affect that?

Exactly. So the key is it is it's still in the software level, but it's in a lower level. You know, you have to really understand the different communication protocols and how they communicate between the systems. And there are some patterns that emerge once you kind of get understanding that. But if you do it naively and just try to integrate, you know, vendor by vendor, you really, you know, you can't. It's just impossible. It's finding those ways that are very scalable and that work very well across health systems.

So you're basically kind of using networking stacks and software to try to integrate as best you can with these older systems so that you get the most seamless workflow you can. Even though you're talking about really old stuff blending with very, very new, modern stuff to affect a user experience, that's adding value to that. How does that change for the radiologist? They now have, presumably, you know, they have the image that they would have looked at anyway. What are you doing in that workflow that changes it for the radiologist in terms of how they're, I'm using the word use case, I guess, but how their workflow seems from their perspective, what's.

Different for them from their perspective? It's almost unnoticeable, I would say. And that, I think that's the beauty of it, right? Don't want to disrupt too much. You want to just create value in a seamless way. So we actually had a customer that they didn't even show the alerts. They configured that in a way that automatically updates their prioritization stack. So they just use this as a feedback and they decided just to change the prioritization. But typically for a customer, they get up in the morning, they come to the regular work list where they have prioritizations, they just see flags of our results and if they hover on a flag, so they just, in their normal workplace, they see this orange ADOC icons and if they hover on that, we show them a preview image of what the AI detected and then they can basically choose that case. If they see the, you know, look at a preview image, looks like a prioritized case, then they open the case and then diagnose it.

Gotcha. So to turn it a little bit away from the practitioner using it now and talking a little bit about the underlying technology, I suspect we have a lot of people in the audience that are just craving to hear about what kind of, you know, which models use. Are there CNNs, is there NLP? There's probably all of this, I'm guessing, if you could take us through a little bit of how kind of high level architecture about what you're using and why, and maybe some of the choices that you had to make along the way. I realized that you have trade secrets and stuff, but I'm hoping that you can tell us as much as you can without giving us giving the secret sauce completely away. Maybe a tiny taste of secret sauce.

Yeah. So the guiding principle for us was that, you know, theoretically, AI, you could build any AI, best engineering, or just any fully connected layer to detect whatever you want, right?

Sure.

But that doesn't really work that way. You know, if you want to create effective system, you have to create a, you know, you have to steer the learning process in a way that is most efficient to learning that specific domain. So the question we ask ourselves is how to train AI most effectively to solve the specific problems of medical imaging. And what are those? And let's just give a few examples and then I can talk about what elements are we using. So the first issue that you get when you go to medical imaging is data scarcity. And it's not one of the. So you think about imagenet, you think millions of images annotated, very easy. You do mechanical torque or something, you can outsource people flagging. It's not the same for ideology. We have what we believe is the biggest annotated database in the world right now. And we have hundreds of thousands of images. That's basically we're talking about roughly order of magnitude of hundreds of thousands Million of studies, each contains hundreds of images, right? But so it's tens of millions of slices of images. But a lot of the imaging is 3D stacks. So you have one patient with 500 images. So that's one big challenge that we had to tackle. The second is class imbalance. You have look at the small patient image is like a million pixels, something like that. And a disease can be on a three by three by three. We're talking eight pixels out of a million. And that's in positive examples, right? If I'm talking Air Force lingo, I would say so if I'm creating an algorithm to detect bunkers, it's like I would give you map of the world and tell you hey, there's a bunker in here, right? Somewhere. Go and have fun. So it really trying to handle this really class imbalance where you have all these normal areas, that's like a big utter challenge. And the third challenge for us was that humans have a lot more information oftentimes that the system is accessible to. And I'm talking about the clinical context. So radiologists, they cheat in the sense that they look at historical exam to look at patient data and it helps them kind of focus on what they really want to do. And that's another thing that we really wanted to focus on. And those are just three examples of how we basically built the underlying architecture to support those pain points. So basically what we're using, it's not totally different, it's convolutional neural network on the medical imaging. But what we really innovated is on the training process to handle the class imbalance on some of the specific layers in the network to try and replicate as much as possible how a physician would look at the image. And we added a lot of clinical information from other imaging from other sources under clinical sources for the neural network to have as much information as possible when doing. And do I get. So I, before I dive any deeper, I want to pause here and see if you have any questions on that.

No, that's very interesting. I'm just curious when you're looking at the world of CNN architectures out there, what are some of the architectures that you like in general? And I'm not asking how you're using them in specific ways, but what is interesting because. And I ask with a reason. Daniel and I often talk about the fact that a couple of years ago, you know, we were seeing all the evolution of, of CNNs but we've heard more recently more about NLP and you know, transformers over the last year or two. And if you're looking at it from our perspective as doing interviews, we're hearing more about this other side of AI. I love the fact that you're taking us back into the CNNs and we haven't really heard how they've evolved as much from guests lately. I'd love to find out, like, what do you think is fantastic out there in the CNN world in terms of useful stuff? Are there things that you think of are old and you wouldn't, you wouldn't mess with them at this point, having used them a lot in the last couple years, just wherever your perspective is.

Yeah, it's a great question. And I would say that it's funny when, you know, when we submit to the fda, even so every one of our products has to be cleared by the fda.

Yep.

They asked us what architectures we're using as well. So it's important to them. And I always found it kind of. I wouldn't say funny, but when we look at our domain, we typically don't just download architecture from the. I'm not downloading a Googlenet or resnet or something and implementing that as is. We're taking elements and we're experimenting with them. But it's relatively like a new. I can't say it's even similar to anything specific because it's just so adapted to our domain. 3D data. We have the clinical information. We do love attention. That I think is a big one for us, again because of the class imbalance and all that. So that's a big thing. And the other thing we really like is tile transfer networks for data augmentation. So anything around data augmentation for data, scarce elements, I love those as well. So if I can pick two elements that I love and we use often, I would say those. But it's really hard for me to say this is an architecture I would, I would just use because it's so, so different. It's like they pick a few elements and you really experiment a lot.

Gotcha. And that was a great explanation for kind of how you're seeing that. I like that. It really sounds like you're finding the competitive advantage there in terms of customizing a number of different things in a way that's unique to your business and which you mentioned barriers to entry earlier. And I would imagine that creates substantial barriers to entry. You also talked about the world's largest annotated data set for this. Are you able to tell us kind of how you develop that, where that came from? You Know, at what point did you see the need for that and how did you approach that? And you could take it from the perspective of there are other people in other industries and other businesses out there that are trying to figure out how to solve it for their business. They know that they need the data, they're not sure how to do it. I'd love to hear how you got to that point where you could affect the, the solution that you were striving for.

Yeah, absolutely. I want to add one more point regarding the last point we discussed in terms of customizing the networks. There is a cost to it, by the way. I just want to make sure that I'm sharing that transparently and that if you are experimenting, you need to have a very flexible experimentation infrastructure. And we've actually invested a lot in developing our cloud experimentation. But even like the small challenges of how do you run a thousand experiments a day and track those and you do it on spot instances and recover experiments if they fail, like, that was a massive undertaking. Right now I believe we're in a much better spot than we were and we can run thousands of experiments every day, which is fun as hell. You know, just have an idea and go crazy with it. Even me as a CEO, I can, you know, just ask them to implement like some, hey, you know, we didn't do abc, let's try. But it's massive. It's one of those challenges we don't often hear about is the engineering of doing very efficient scalable experimentation regarding the annotation. So as you can imagine, a lot of it is secret sauce. They'll try and get the high level elements. So I want to share two things. The first is that when people think about massive amounts of data, they just say, oh, I just want as many images as possible. There's a saying by my CTO that I love. He says, like, guys, what I want is not to have the most images. I want to actually train most effectively on the least amount of images if I can do that. So. And how do you do that? By selecting the RICE cases for optimization. So in medicine in general, annotation costs are sky high because you need physicians for that. So you really want to pick your balance and the case selection methodology even before saying about the biggest, which is something that I'm proud of, but he's saying like, you shouldn't be proud of that. I would ideally want to have the smallest annotated imaging database, but the most effective one. So the case selection, how do you pick the right cases, optimize the learning performance as much as possible. And again, let's take brain bleed, I think is a great, great example. Am I taking 1000 pictures of patients with massive brain bleeds which are obvious, or am I taking maybe the only 50? But those 50 are, those reflect subtle brain bleeds that you can really train and improve the system with. So I would tackle that challenge, if more, then I would tackle the challenge of annotating very efficiently. Regarding efficient annotation, I would just say that the key we found is having everything very tight. So outsource as much as possible. The connectivity between the data teams and the AI teams and the annotators and the fact that we control all the bits and pieces, the annotation selection, everything. Everyone knows their parts and they can communicate. That was very efficient. I heard about Google doing something for, I believe it was for dermatology, for eye images and they had like a thousand page manual for their annotators. Okay, that for me is ineffective. Right. And the problem was annotators were outsourced. So just like going through all the information and helping them inefficient. What we believe is key is having like very high interactivity, a lot of feedback loops, a lot of, you know, AI people. I want this and data people, hey, this is tough for me to annotate. Do you really need that? And all of those small bits and pieces we find really help accelerate the process. And I won't talk too much about how exactly we solve it, but I would say that's, that's a big element.

Have you heard about Knowable? It is an awesome new platform for learning from the world's best minds anytime, anywhere, at your own pace, through audio. Learn about the performance benefits of a plant based lifestyle from NBA all star Chris Paul. Or how to launch a startup from Reddit co founder Alexis Ohanian. There's even a 10 lesson course from astronaut Scott Kelly. Here's a sneak peek.

We learned a lot up there. But what can you learn from a life in space? The answers might surprise you. In this knowable course, I want to share some of the things I've learned that you might not expect. Lessons about leadership on a dark night on an aircraft carrier in the middle of a churning sea. Lessons about the fear you feel with 7 million pounds of thrust exploding underneath you. And most of all, there's an idea out there that astronauts are always perfect. Failure is not an option. Right. That's why I want to take you through some of my life experiences to show you how that's just not true. I Believe everyday regular human failure, if we handle it right, can be one of our greatest opportunities to learn, grow and succeed.

Knowable is accessible on your phone and on the web, and each audio course is broken out into individual lessons, usually around 15 minutes long. As a Changelog listener, you can get an annual membership to Noble for 20% off. Get unlimited access to to every Knowl audio course. Right now, just download the Know app or visit knowable, FYI and use code changelog. For that 20% discount, we put a link in your show notes for easy click ins, check out Knowable today and start learning from hundreds of top experts from around the world. Once again, that's Knowable, FYI. Code Changelog.

So you mentioned something a moment ago that I'd like to go back to and that's you talked about kind of the ability to do your experiments. You referenced your own workflow internally, the infrastructure that you had, and, and I'd really love to hear kind of how you've approached that because, you know, people tend to talk about the AI. It's, you know, it's the sexy thing to talk about and what you're doing, but they miss the fact that it has to fit into a larger software process that can take you all the way from experimentation and figuring out where you're going and then integration into your other software that it has to, you know, because it's a model has to sit inside a larger, a larger set of components that are software and eventually has to be deployed out to be usable in the real world. And so how have you approached that? You know, there are everything from cloud to buying your own equipment, to being, you know, which workflow, which vendor, which open source you're choosing. How do you approach these challenges, these considerations that you have to make, considering that each one has a fairly substantial impact if you, if you don't get it right.

Yeah. So I think the first big element that is important to note is team structure. I think that was actually the first consideration to take into account. And our head of AI Idan, did an amazing job. He actually interviewed a few companies, how they build it, how Google does it, how other people do it. And we figured out a structure where the AI algorithm engineers are actually a small part of the team. I'll go through the evolution we had. We started with having algorithm engineers just focused on algorithms, but they weren't really focused on deploying and anything like that. We then figured out that was a mistake. One people that can take the product end to end and take full ownership and not like do cross because things get left and it's really inefficient. So we brought people in that are both very sharp algorithmically but know how to project, manage, know how to deploy, know the software side, are not afraid, you know, to have their hands dirty with, you know, implementation. Then we kind of started figuring out way but that's not also not very scalable. We need people that are even better at software and infrastructure. So we created like a whole AI infrastructure team with a ratio of about 2 to 1. So two infrastructure folks for every one algorithm engineer. And then they started working on the infrastructure and it's exactly the thing you mentioned. There are so many of those challenges. Do scale, how do you scale? This is the cloud and experimentation infrastructure and deployment infrastructure and monitoring infrastructure. Also very important. How do you monitor accuracy in production. And then we, and I would say that's kind of how the team is built today and we're scaling it the same way. We were always big fans of the cloud. We started on prem, which was, I would say, I don't know if it was a mistake. Back then we didn't have as many experiments, but for now for sure we can't do it on PREM because it's just so much experiment. So if you really want to scale fast, you really have to solve the cloud. And it's even like stupid quote unquote challenges. Like one of our challenges is that we ran out of spot instances for GPUs. That's a challenge apparently. And you know, solving that and how do you like make sure that you work on multiple regions and how can you potentially transfer experiments between so all that other small things in making it tight. I think it's just like respecting the engineering side. That was a big, big thing that I learned. Just make sure that, you know, that's probably a bigger part of making a work in practice versus like science.

Gotcha. There's so many different ecosystems. Each of the cloud providers has their own ecosystem. Nvidia with GPUs has their own ecosystem and workflow. There are a number of tools that are cross ecosystem in terms of TensorFlow will work in all these different places as well as Pytorch and others. How do you buy in or how do you evaluate a particular ecosystem or a combination of them to make that bet to go down that road. Any thoughts, any guidance that you can provide us on that?

For sure, first of all, we took the path of AWS and we're very happy with it.

Sure.

And the main reason we chose AWS was That it is in medicine we say standard of care. So that's, I would say the leader probably at this point in time still. And that makes it very easy to hire, you know, DevOps people that know how to do it. Like all the new tools go on that platform. So that was for us a big consideration and the other is just pure cost. So when I, you know, when we pick, those are the two biggest one like innovation speed and cost and can we create the architecture? The other big element in terms of consideration is who is available and can help you work through those challenges. So again, I think there are several companies that were really accessible, but we really enjoyed the interaction with aws. We had all the technical researchers to help us think the challenges. And I think what, what I learned as, you know, someone that was technical but not technical on the cloud side, I learned there are so many different elements, so many ways to create your architecture that you really want to have a partner to help you think through those challenges.

So that was super cool about how you're trying to evaluate the software. I love hearing how people make those hard choices. And clearly you have a kind of an approach to how you're evaluating the possibilities out there. That has worked very well for your organization. So that really, as we look forward, you know, there are so many things I want to ask you and kind of wherever your thoughts are taking you on this, we have the growing world of AI with all the tools on that. You're in an industry that we know is in the process of being revolutionized and probably in the very early stages of this technology. And there are so many choices that you have to make and both as a leader of your organization in terms of where you're taking the group and how to evaluate a really complex landscape of what AI is in the world that we're at and all the tooling. Where do you see the world going at this point? You know, if we're looking over the next few years in the relative short term and maybe even farther when we're talking about where we might arrive and you know, obviously, obviously nobody can read the future accurately, but I'd love to hear kind of where you think the world might be going down the road.

Yeah, it's a fascinating topic and we have to, because we're in such a fast moving industry, we really have to think about it all the time. And I'll start with taking us back into the evolution of AI this past two years just to understand what are we even talking about and the speed AI in healthcare and in medicine has been in this frozen period for a few years. So if you would go three years back, not talking a decade, three years back, really almost no day clearances went to deployment two years back. We're talking about 0.01 market share of AI adoption. We're talking two years ago and to date only us, we have hundreds of hospitals and deleting names that you would know are already utilizing AI. So we're talking about massive adoption in two years. And thinking ahead for the next two years I think we're going to see really probably everywhere you're going to see one AI or another, which I find fascinating. I love being an industry that is so fast moving, kind of like we broke that barrier and then it's starting to evolve. But I think that right now AI is in that really first age of healthcare, AI and AI today is doing, I would call it workflow augmentation. So AI right now is only helping to the humans what they either way would do but better. So in my case it helps radiologists detect pulmonary embolism faster. So that's the assistance. But if you ask me, three, four years from now, air is going to start going into the predictive space. And what do I mean by that? I mean look, seeing things that humans cannot see with their naked eye, right? So if today you can detect stroke at point A, you know, if AI could detect it in A minus 2, Alzheimer, early detection, early detection of cancer. And I think that's where really the promise of AI becomes really exciting, where you're creating new diagnostic capabilities. It's like initially you need to do a CT to even look at the bodies and understand see state. Right now we're talking about the next phase of that seeing patterns that no one of us could predict. And I think that's the exciting next evolution for AI. I think it's very challenging for that. You need very high levels of trust. You need to have mountains of data. We're talking about, we're just magnet more than there is today. But I think once you get that that's the holy grail we're all aiming for. And I think it's not that far away. We're talking three to four years where I going to start seeing things and really impacting in a way that is really impossible by humans today.

That's really interesting to me. I'm 50, I don't mind telling people that I'm kind of getting to that point where I'm thinking as a guy who doesn't care much about medicine in terms of my own body so much. I'm just doing my thing, getting out there, exercising. And I'm starting to think now about I'm at an age where this is starting to matter. And so it's really good to hear that as I move farther into my 50s, that those kind of new diagnostics are coming out, because I am. I'm counting on you to keep me healthy, buddy. So anyway, thank you very, very much for coming onto the show. It's been a fascinating conversation and like I said, for my own selfish purpose in terms of keeping myself healthy and also in the perspective of hearing these ER stories and hearing how you're changing that whole picture and changing how fast people can get care. I appreciate your work and what your organization does, and thanks for coming on the show.

Thank you so much.

Thank you for listening to Practical AI. If this is your first time, make sure you subscribe so you don't miss a thing. Head to PracticalAI FM to subscribe or find us in Apple Podcasts, Spotify, or wherever you listen to podcasts. And if you get value from the show, please do share it with a friend or a colleague. We appreciate you spreading the word. Practical AI is hosted by Daniel Whiteneck and Chris Benson, it's produced by Jared Santo and our music is provided by Breakmaster Cylinder. We are brought to you by some awesome sponsors. Shout out to Fastly Linode and launchdarkly. That is our show. We hope you enjoyed it and we'll talk to you again next week.