#124 Using AI to Improve Data Quality in Healthcare (with Nate Fox and Sunna Jo)

You're listening to Data Framed, a podcast by Data Camp. In this show, you'll hear all the latest trends and insights in data science. Whether you're just getting started in your data career or you're a data leader looking to scale data driven decisions in your organization, join us for in depth discussions with data and analytics leaders at the forefront of the data revolution. Let's dive right in.

Welcome to Data Framed. This is Ritchie and today we're talking about data quality. Before you groan and skip to the next episode, let me rephrase that. We're talking about how today's guests built a company that solves a real healthcare issue for millions of Americans just by fixing data quality problems. I know that improving data quality isn't the fun part of working with data for many people, but there's a tremendous amount of value to unlock. But both for businesses and their customers, just by paying attention to this issue, making money and helping society at the same time is the dream scenario. So it's worth taking a minute to see how they managed it. Joining me today are two guests from Ribbon Health. Nate Fox is the co founder and cto. He's got a really interesting background mixing engineering and product management and marketing and analytics. Alongside Nate is Sona Joe, a senior data scientist. Sonnet's background is perhaps even more interesting. Having switched careers from being a pediatrician to working in data science, I'm keen to hear what they've been doing with data quality and to get some ideas for how to think about data quality in other organizations. Hi there Nate and Sunnah. Welcome to Dataframe. Thank you for joining us. To begin with, I'd just like to give everyone a bit of context about what you do and what Ribbon Health does. So perhaps Nate, can you start by giving us a quick overview of Ribbon Health?

Yeah, absolutely. So at Ribbon Health, we're a healthcare data platform and we focus on helping our partners transform how healthcare decisions are made for patients. We're a B2B SaaS company, so we enable our clients to use accurate and actionable provider information to then empower people to make the most informed decisions about care. So you can get the process of a patient trying to make a healthcare decision. There's a lot of fragmented data out there, like where can I find an accurate location that has reliably a doctor that accepts new patients that but also just my insurance that treats patients like me. And so we're a data platform that focuses on bringing together this very fragmented data out there on providers to make it actionable for patients. More broadly.

Nice. And you're Chief Technology officer, so what does that mean? What do you do?

So in my role as Chief Technology officers or as a cto, I lead a team of engineers, data scientists and product managers to build robust and forward looking data infrastructure for the healthcare industry. So what this means on a high level is I'd say most of my time is spent helping build a team. So meeting with many new engineers and data scientists and getting them excited about the the really cool and fun things we're building here at Ribbon Health. And then of course working closely with the leaders across technology organization to scale the organization, thinking about the new products that we're going to build and processes we need to implement and help us scale.

And Suna, so you're a senior data scientist, so what do you do at Ribbon?

So I am a technical lead for one of our product offerings which is called provider performance, which I can talk about later. And I work with a cross functional team to source different healthcare data sets, scope and build products as well as improve our existing products. I help define and calculate metrics and I also present and bring key findings before stakeholders and the broader team for engagement and buy in.

That's great. And you used to be a medical doctor, but you transitioned to working with data, so can you tell us a little bit about what motivated that?

Yes, absolutely. So I've always been interested in this idea of data driven decision making, particularly in healthcare. So one of my research projects involved analyzing providers clinical practices in treating pneumonia, which is a bacterial infection of the lungs, which is fairly common in children, which is the population that I treated, to understand how and why their practices may deviate from the guidelines that are published and in an effort to improve the standard of care at our practice. So through this I recognized the potential to mine valuable insights from healthcare data and transform them into high impact solutions to some of the problems in our healthcare system. And I felt really compelled to work on these solutions with more of my time. So I transitioned to data science and deciding to leave clinical medicine was difficult. But I did recognize this special opportunity that I had with my particular background and skills. And I'm really grateful to be able to impact the healthcare system in a different way through my work at Ribbon.

That sounds pretty cool. So rather than just helping people one at a time by using data, you get to help lots of people.

Exactly, yes. I really love the ability to have a wide reaching impact.

That's brilliant.

One thing I think is pretty amazing is how you get to bring your clinical expertise to the products that you're building. I think the audience would love to hear about how you do that at Ribbon.

Yeah, absolutely. So I leverage my clinical experience daily, which is really amazing and super motivating. And this comes in the form of providing an additional lens on the data from the perspective of a healthcare provider. Just giving the team more context for the data that we're working with, or even using my clinical knowledge to interpret and translate the data in a way that makes sense. So for example, we for work on one of our provider performance products, we work really closely with medical codes. So these are designated codes that define certain diagnoses and procedures and cleaning and building a model on these same codes that I used to bill for my own visits as a provider and being able to recognize and understand the insights that we can get from these codes have just been a great reminder of the value of my experience.

And so you mentioned a product provider platform, so perhaps can you just explain what that is and who the providers are, what you're doing for them?

Yeah, absolutely. So our provider performance product segment as a whole, it aims to ultimately help patients find high quality, affordable and accessible healthcare through identifying the right providers for their specific needs. So some of the sort of like lines within this product segment are cost and quality product which helps identify high value providers for various use cases, including targeted care search. We also have a price transparency product that helps patients find cost effective care options in their areas and it helps them anticipate the cost of a specific route of care based on their insurance. And we also have a focus areas product which helps identify the right providers and healthcare locations for specific healthcare needs, including certain conditions and certain treatment options.

Okay, so a lot of this sounds like you're trying to match people up with what their best sort of healthcare option is based on their insurance, what the conditions are, things like that. Is that about right?

We really saw the need to bridge this gap in the healthcare space as it is today. Care search is not so straightforward. It's often not easily accessible and not interpretable. So we really wanted to create a product offering that helps patients get to the kind of care that they need and that they want.

So really this is about helping people make data driven decisions, which is nice with their healthcare. Can you maybe just elaborate on the sort of different kinds of decisions people might want to make?

Yeah, absolutely. So I think when someone is trying to find care, oftentimes they're thinking about a number of different metrics. Right. These are things like, where can I actually find a doctor that's accepting new Patients. So this is like information like actually getting the address right, which surprisingly actually half the addresses out there in directories in healthcare are actually completely inaccurate. As if you were to go to your typical health plan directory and just like say, show me 10 cardiologists in New York City that accept my insurance and you recall half of them, the data would just be wrong. Something would be incorrect. The phone number's not right. Doctors not accepting new patients, they actually don't accept that insurance even though the insurance company claims that they do. And so just actually getting that core foundational, almost like yellow page information correct, it's kind of like the foundation of powering an effective search. And then on top of that we layer many other things, things like confidence scores, right? The level of accuracy of data that people can search on. So you can say only show me data points that are 90% accurate or higher. I don't care. I care less about coverage, I care more about accuracy. And then you can layer on, show me doctors that kept my insurance. And then you can learn all the things that San just mentioned around price transparency, focused areas and other sort of like qualities that help people route to efficient care. And what's pretty amazing is actually this, this information that can then be accessed through our API has really enabled some pretty amazing like care navigation outcomes for our B2B partners. In fact actually like 3xing the speed at which a care navigator or a referrer can actually help a patient find access to care 3 x ing their ability both in terms of speed and output to actually get more patients access to timely care in a timely manner.

So it sounds like a lot of this is around data cleaning and taking just horrible data that's messy and probably wrong and turning into something that's clean and therefore useful. So I'd love to get into talking a bit about like what does this data look like? So what's the sort of data that you ingest that's a mess to begin with?

So again, on very high level, right. The bread and butter of ribbon health, if you will, is all kinds of provider data. We consider providers to be three major node types. The first is like individual providers, right? These are like doctors and they actually have a unique key which is really important. We'll talk about later why unique key is really important in the context of data science and data engineering. And so that's like one major provider type. This is like Dr. John Smith or Jane Doe is like an example of that kind of data type. And then we have facilities or locations, right? This is like where you can actually get access to care. So think of your local urgent care pharmacy, the hospital that you can may access care, and then finally organization, you think about a healthcare claim. When you file a claim, it doesn't get billed to like the provider that you saw, but actually to the organization they're a part of that's actually rendering the transaction with the insurance company. And so Ribbon Help sees all kind of data from many different sources. Think data we scrape directly from the web, data we get from publicly available data sources, data we get from claims that we have a license for, and other data feeds that we get from actually our partners as well as customers give us the customer contributed data as well. A lot of our clients are actually editing information or providing their own data feeds that they want us to ingest. And so all of this data is very, very, very messy. And so what we do is kind of coming back to those three nodes is that we take thousands of different data schemas, normalize that data to a standard data schema, and then we normalize the data structures themselves. Think of all the text strings and things that need to be geocoded and sort of standardized. And then we actually have the data cleaned up and ready to go. And then we resolve that data to recall our knowledge graph, to say, hey, we saw this record with this facility name, that's a text string with this provider at this place. Let's resolve this record to the nodes and then figure out the links between those nodes. And so this allows us to create the foundation of the Ribbon data platform, upon which we then build other data products that are key for the search that we talked about earlier. So this is where I think we get to some really interesting aspects of provider performance. And so once we have the providers that we're talking about on top of those providers, we can actually have the analytical insights that Sana is talking about. Once we actually resolve that to those nodes, maybe Sunnah, you can elaborate a little bit more on how we can go into that messy data.

Yeah, absolutely. For our provider performance product offerings, we use claims data, which is actually real world data that provides information about providers, clinical practices. So it tells us which diagnoses they treat, which procedures they perform, their billing practices, what insurances they take, and a lot more. And we receive the data in a certain format that we query, and it can be really messy as it is sourced from manually entered fields. So it's particularly prone to human variance and error. So it's difficult to interpret input values. Some of the structure is not even Intuitive. Some of the fields you really have to dig deep and look cross reference to try to make sense of. So in terms of cleaning, we obviously have to remove spelling and other obvious input errors, remove duplicates, think about how to deal with missing values. And we definitely need to standardize to make the data more comprehensible and to be able to process it more downstream. We also have to think about which parts of the data that we want to keep, which we want to put aside for later, which we need to discard. And we have to think about different ways to transform the data so that we can really translate it into actionable insights that we can apply to different use cases. And there are a lot of challenges with normalization as well. Sometimes we have to do it by segment. We even have to do it based on which field it's in. It can differ across different portions of the data. So there are a lot of unique challenges to this particular data set. And it's definitely fun to work on, I would say.

I know like everyone who works with data is like, oh, I wish data cleaning was easier. So do you have any tips for people who have their own sort of data cleaning problems? Maybe this is one for you.

So I think a lot of processes in healthcare are both an art and a science. So I think this opens the door to a lot of variants, as mentioned before, a lot of error and less direct interpretability. So I think it's really important to understand the context of the data and to really have insight into why this data was produced in the first place, who sits behind it, what the intentions were. And I think that helps. That can change the entire process, even in the way that we clean the data, in the way that we analyze it, in the way that we think about anomalies and edge cases. And I think having this context and establishing these fundamental principles, understanding that the data is not standardized in this sense, and adjusting our processes accordingly can really help us make the most use of this data, produce the most actionable and high value insights. And it can also help us avoid the mistakes of over indexing on certain use cases and not investing in the areas of the process that really will yield the most value.

And so what do you find are the trickiest parts of data to deal with? What are the big problems?

Yeah, I think particularly with claims data, again, I think a lot of the structure and some of the input is not interpretable or easily interpretable. So I think it needs a lot of additional insight, whether it be through clinical knowledge or experience or kind of cross Referencing So even just understanding it, I think really takes a lot of investment and effort, but definitely pays high dividends. And I think also just thinking about how to work with the data at different levels. So again, we have real world data about providers practices, but this intersects with medical codes that they have for each visit that they bill for at multiple locations at different points in time for multiple patients in a day. So there are a lot of levels at which this data can roll up. So even thinking critically about which levels of the data we want to look at, how we want to aggregate the data in a way that makes sense and that powers our different use cases, and even just thinking about how to adjust our cleaning and processing methods based on certain assumptions about the data or based on what kinds of outputs we want definite poses an interesting problem.

Well, you mentioned that there's a lot of clinical knowledge required. So does that mean that you have to have data analysts or data scientists working alongside doctors, maybe? Nate, do you want to talk about how the teams are structured and who interacts with who?

Yeah, absolutely. So at Ribbon Health, we have actually I'll talk about the overall technology group and how we assemble. So we have like three major groups within the technology group or organization more broadly. So the first, what we call our platform team, this is the team that gets our data adjusted all the way from the raw data that we consume. So think of all the data we collect from our numerous collectors claims, et cetera. And then we have what we call the data infrastructure team that actually processes all that data in the ways we've talked about, like resolving that data to record nodes and the links between those nodes. And then we have the API platform component, which is the team actually takes that data once it's been processed and cleaned and scored, and then delivers that to the people that are consuming that data. On top of this foundational platform, we have two major product use case verticals or groups. The first is what we call Care Navigation. That's the group that is focused on thinking about how do people actually use this data to find care? Much of what we already talked about on this call and the other major product use case that we have is what we call data management. So this is the aspect of a lot of clients actually letting us ingest and manage their data within Ribbon's platform. Because I'm sure, surprising no one on this call. Healthcare data is really complicated. So oftentimes clients want actually us to be their source of truth, but have us integrate their data within our platform. So kind of think like you're buying Salesforce and you want to upload your leads into Salesforce. People want to have their provided data sit within ribbon more broadly. And then within these groups, right, these are like the three major groups and they cover about 45 different folks. You have pods inside these groups and these different pods actually are cross mix of folks, a combination of engineering, data science and product as well. And they are designed to have their own autonomy to own the product variable that they're in. So, you know, as Sana is on the amazing provider performance team, but it has a mixture of different folks and they can have the autonomy to own their own product and very much work closely together so that they can then coordinate with the rest of the different pods and groups at Riven.

Okay, so it's one pod per product, and then you have different skills or different roles within each pod, is that right?

And the big thing that we try to think about is how do we make sure these pods can run on their own and be autonomous? Because I think we all like to be in those small, nimble, agile teams that aren't encumbered by process. And so we try to think of how can we design teams the right set of skills so they can be autonomous in that way.

Okay, that sounds very good for kind of productivity within each product. I guess the next question is how interlinked are your products and how do different pods communicate with each other?

Oh, they're really interlinked. So talk about that kind of like initial foundation and everything built on top of those nodes and edges. Right. So the products that Sana's talking about that her team works on, that is built on top of our graph of those different nodes and edges. And so there's a lot of coordination upon these different teams working together. Actually, Sana, I think that price transparency and how you have to resolve that data to the location directory that we have is a really great example of the kind of collaboration and coordination that some pods sometimes have to do. Would you like to elaborate a little bit on that?

Yeah, absolutely. And that really is a great example of how our different product segments intersect. So just as an overview of price transparency, it ultimately allows the patient to search for cost effective care in their area, in their geographic area also based on their insurance, and helps them anticipate the target total cost of a certain route of care that they would want or need. So this poses a lot of interesting data questions leading up to the production of the output. So one of the major problems that we faced was how do we resolve the provider rates that are reported by different payers like we're in the US So payers like Aetna or Cigna. So how do we resolve that they provide us regarding the kinds of services and the rates that providers report? With the power of our directory in surfacing accurate information about where providers practice and what types of insurances they accept. So we actually integrated our claims data into this problem as well. So we use certain identifiers in our claims data that we can then link to our directory. And we have a high level of confidence in the accuracy of the providers and the locations in our directory. So working off of that confidence, we can then create a mapping between the providers and the locations in our directory to what's reported at a national level to be able to report a connection from provider to location, to a certain treatment, to a reported rate that is really unique in terms of the scope of the information that the patient has access to. And I think we can really only do this leveraging all of our unique data sets as well as the value that our directory provides.

And so how do you decide, like, what sort of value a particular dataset has or how do you know what the value is of any given product or area of analysis?

So we ingest data from over a thousand different sources. And so part of our understanding of the value of a different data set is ingesting that data and then seeing how it actually compares against our broader data set. And so oftentimes we're thinking about, okay, this data set that we're seeing have a provider data, and we see a lot of provider data. And so for the 1000 addresses and phone numbers and other metadata that we see from this new data set, how much of it is data that we already have or don't have. And the other thing that we actually assess is we actually can then train models on assessing the quality of a source. And the way that we actually do this is that we always make sure we have this really strong, true set of information. And so when we want to understand how good a data source is or isn't, we're actually sample data from that source in a stratified way and then send it through actually, our call center. Our call center actually takes this data and calls real providers and says, hey, on behalf of Ribbon Health, we want to confirm that this doctor's privacy in this location. Is that correct? Are they the specialty source, so forth and so on. And so we then can then calibrate our models to actually predict, based on which sources are agreeing and disagreeing, the probability that A given data point is correct or not. So that's just like one example of how we think about net new data sources.

That's very cool. What sort of techniques do you say use in order to get the data in the right form? Because you talked about having cross functional teams with different skills in them. What are the sort of skills people need in order to do their job?

That's a great question. I think data engineering is a huge part of making this data usable. And I think it requires a lot of creativity to think about how can you scalably ingest thousands of schemas. We actually built this really fun what we call the parser tool. And so you can imagine that we have literally thousands of different schemas where you have like imagine address, right? Everyone structures address in a different way. Some people have a full address string, some people have address line one, address line two, people have street. You can think of all the different ways people structure address. And yet we here at Ribbon Health, we want to have address be standardized across all the different schemas that we see across different data sources. And so we actually have this nifty tool where whenever we onboard a new data source, we can actually map the different fields that we have to what we call our standard fields and our core schema. And so by having that parser tool, we used to like build these parser files where it would take us 20, 30 minutes in Python to code up a new data source. But when you have hundreds of sources, it becomes quite a mountain of work to actually get through and not very scalable. The parser tool actually we have a simple UI that allows you to click, click, click and it even starts to guess some initial mappings for you. And so now things that used to take 30 minutes can only take 10 seconds or 15 seconds to map that data, which makes a lot of our operations or data adjustment processes a lot smoother.

It seems like a lot of your data is text data. So do you use any natural language processing techniques?

Yeah, absolutely. So I'll write some context on a really key problem that we have in aggregating all the data and then resolving it to the nodes that we've talked about before. Providers, locations and organizations. So we're really lucky that for providers actually there's an industry accepted unique identifier called the NPI number. Every person providing care in the US think of every chiropractor, nurse, doctor, has one of these numbers. And you can actually download the full NPI data set directly from the cms. So if there's Data scientists that want to play with the 6 million row data set, go ahead and download that. It's really fun to want them to play with. But there's actually no unique key for locations or organizations, which creates a really gnarly problem for us here at Ribbon Health, because what that means is when we see a lot of data with text string names for a facility to give an address, a lot of them actually don't have a way of naturally joining to each other with a unique key. So we at Ribbon actually have to take those different text strings and try to cluster them together. You can imagine that we have at a given address for a famous hospital, There could be 20 different names for that hospital. It might be the LLC name, it might be an abbreviation of that hospital, it might be like a certain department within a hospital, like ie, obgyn. We had to figure out how do we actually cluster those names together and extract the right information, which is where like natural language and text processing is really, really important and key part of what we do.

And we actually borrow some of the principles of natural language processing for our focus areas product as well. So we identified that information about healthcare providers that is available today is not really intuitive and doesn't really allow for effective navigation. So we wanted to help bridge this gap through identifying the right providers and healthcare locations for certain needs. And we actually use topic modeling for this approach. So as I mentioned, we work really closely with medical codes in the claims data. So we apply topic modeling and other NLP tools to try to surface insights from this mess of not easily interpretable data to create patient friendly terms that patients can ultimately use to find the right kinds of doctors for that.

Okay, so I guess if the topic of a particular document is something around, I don't know, problems with like a broken bone, then you want to make sure that the organization is somewhere that's going to deal with the broken bones rather than a completely different organization. Is that the idea behind the topic modeling?

Exactly, yeah. So for a given provider, if they have, let's say what we call a mass or a document of certain codes and one topic, or if they treat a lot of, let's say wrist pain, then wrist pain will show up a lot as a topic. So we gain that insight and we're able to translate that into an area of clinical specialty for the provider.

That's great stuff. And moving from the techniques to the tools, I'm interested to hear a bit about your tech stack. So what sort of tools do you use? I guess to start with the Analytics.

Just from a starting point, we have access to a lot of different tools. And as Nate was mentioning, I think the autonomy that we have as a team is really great and being able to explore different options for approaching problems in different ways. In terms of programming languages, we primarily use Python and SQL. We also leverage Spark for some of our really big data sets. We leverage again, serverless query engines. We obviously look to a lot of different storage sources depending on our specific needs. We use different repositories for our code, like GitHub, we use Databricks. So yeah, we really have a lot of free reign in this respect. But I think those are the tools that, that we definitely make the most use of day to day.

Great. And Nate, do you want to talk about the other parts of the tech stack?

Beyond that, the only ones I'd add there are on a high level, we use AWS to sort of like for all these different services. And so we go really deep on that. And for the API, actually we have two sort of main applications that are external facing. The first is our API and that's a Django application which does millions of API calls per month. So we have to figure out how do we scale that function and make sure it's really reliable, because people expect 99.99998% uptime for our API and to be critical infrastructure. And then we also have what we call our FindCare application, which is a UI where it's for care navigators to actually tap into the data platform directly, not having to build their own sort of search tool. And so that's also another application that.

We have and we talked quite a bit about the data cleaning process and the techniques and things. I'd like to talk about the output. So what format do you provide the end data? The stuff that you say is good and clean and useful to people.

So the API is a fairly simple REST API and the output of the data is in standard JSON. And so we try to make sure that the data is cleaned up and is easily interpretable by any other person consuming that data on the other side. And so we make it fairly simple and straightforward. And I think the key thing that we do with this data is we actually apply what we call our confidence score to the various data points. And so a big part of the ML models that we have here at Ribbon Help is actually detecting what we call correctness. So when we do this truth set validation, where we build a truth set and validate information, we train models to predict the probability that when you call that phone number for this doctor at this location and see if they have that specialty or focus area, what's the probability that's actually true? And so every provide location, for instance in that JSON that I mentioned before, will actually have a field called confidence. And we try to simplify it to like a five star zero through five, where five means that we'd actually verify this to be true. Four is 90% accurate all the way down to one, which is we're very confident it's not correct. Which is useful for actually for a lot of enterprises, knowing what data is wrong is very useful for like health plans and whatnot because they can clean that data out of their System and then 0 is verified to be incorrect. And so that's just like some of the ways in which we present our data to our end clients.

So with an API sort of output that means that your target audience is always going to be developers or maybe technically focused analysts. And that sort of seems very useful for the B2B customers that you have. I'm curious because like the very end stage, like the value add is helping people with medical problems. So how do you go from the API output that you provide to I don't know, a dashboard or something that. Or a website that. That consumers can use?

Yeah, absolutely. So there are some examples of websites that, you know, right now when you search for a doctor on that website, it's being powered by the Ribbon API. So you can imagine you make a request to the Ribbon API where you say ribbon. Show me cardiologists in New York City that accept Aetna and then the Ribbon API instantly will return 10 responses. In New York City you rank by geo and then distance and whatnot. And then people will take that JSON and then in their application will show those results in a human readable format. So you can kind of imagine when you're doing like a search on Kayak, we get those results. You may get those results for like finding care. That's one way. Another way, actually it's kind of interesting is that. So for the data management use case that I mentioned before, some clients actually will have us ingest their data and then we'll actually return them their data in like a file that they ingest that shows up in their platform in other ways as well. So there's a lot of ways that our data can, you know, come up and be presented to the end consumer.

And for any data minded people or technically focused people listening, because we have a lot of them if they want to play around with your API. How can they do that?

We do have some. I will say this, the tech team is thinking of some open source data sets we want to release to the public. We're not quite there yet though, but stay tuned in 2023 later this year. But I think for at least for now you can go to our website@ribbonhealth.com and you can put a try now form and you click that button and we can give you API access for our demo token.

And do you have any success stories of cases where people have used the data and they've had something great happen to them?

Yeah, absolutely. So there's some partnerships that we talk about that are public. So our partnership with the payer provider Firefly Health, they've actually increased their average speed of a referral for patients by 3x. Firefly was effectively using a mixture of Google to look up with specialists that were close to a patient. You can probably imagine we also like that's at least I found care before a technology like Ribbon and then calling the office confirmed the doctor was there trying to figure out do they take their insurance or not. But it was very manual time to intensive process and caused a lot of operational headaches. But then with implementing Ribbon they were able to immediately tap into our network and directory of provider information via the Ribbon API instead of a mishmash of spreadsheets. And that's an example of one company using that solution. We actually we worked with a lot of health plans and a lot of health plans have actually transformed their core directory to be significantly more accurate. A lot of health plans directories are as low as 50% accuracy and it's just really bad for consumers and making it difficult for those folks to find care. And in using Ribbon actually they've been able to dramatically improve the accuracy both their locations and provider directories.

That's very cool. And I do love the idea that just simple changes in data quality. I say simple. I'm sure from a technical point of view it's a lot trickier. But simple changes in data quality can have real effects on people's lives and improve their healthcare. That's pretty great. I would like to get go to the flip side though. Are there any things that you've done wrong? Have you got any lessons learned that you can share with the audience?

Yeah, I think one thing that's really, really important from a product oriented perspective is what is your truth set trying to predict? Because the outputs of your model will only be good as the outputs that you tell it to predict or orient itself to. And so if you think about what is correctness in the world of healthcare, actually it has a lot of different definitions For a health insurance company it might mean that this address is correct if I can bill an insurance claim and my claim does not get denied. But there's not the same level of correctness for a patient where if they call that phone number at that address and they get a dead phone number, they're told the doctor's not actually there. That's not correct. And so I think when we think about our truth set, we think about what is it that we want to optimize for. And in case we're urban health, we think a lot about the patient and optimizing around care Navigation and so our sense of truth is dictated by if I call this phone number and I say can I book an appointment with this provider? Will I get to a positive outcome? And so we design our call center and our emotions for true set collection to sort of be oriented around that. And it just took a lot of time iteration to get to. And so I think in terms of things to watch out for is think really, really, really critically, especially from a data science perspective, is what is the outcome you're trying to predict that's actually useful for the end user and make sure that you've really broken that down, you understand it in a crystal clear way. So you can then of course design your data engineering data science processes oriented towards that output.

And so nut do you have any lessons learned?

So one of the main challenges that I think we face regularly is trade offs. So how do we balance the risks and benefits of whatever we are building? So one example is for focus areas for a certain specialty. For example, we want to make sure to cover a comprehensive range of conditions and treatments, but we also want to make sure that we are accurately mapping those conditions and treatments to providers who specialize in those areas. So I think in working with the data and building the mod and in evaluating the outputs, there's always a trade off between how can we not over index on being comprehensive at the risk of losing accuracy and how do we not over index on accuracy at the individual provider level at the risk of losing coverage? So I think that's something that we've been navigating as we build out this product. And we definitely got really helpful feedback from our partners and our clients. And I think that's a big area of improvement that we've been able to make.

That's really interesting that you have to balance the trade off between accuracy and coverage and just making sure that the data's right. But it's right in all cases.

Exactly. Yep.

So in some sense, it seems like your business is a clever hack around the US healthcare system where all the data is terrible and you turned it into something good. But it'd be interesting. Or maybe the ideal solution is that the data gets better in the first place. So I'm worried. I'm wondering if there is any feedback mechanism where you get to tell the data providers, please go and fix your data.

The first thing that comes to mind is that something that we have built from very early on in Ruben's days is we actually make it so that in the API, it's not just read only, but you can edit the data or information. And so as our end users are consuming and using our data, if they see the location is incorrect or they want to append the information, they can actually make a request for your API within our standard protocol and actually correct that information. Our platform is very much designed with this idea that the data is constantly churning. We did some pretty interesting analysis where in a given year There was a 30% churn in data quality, and so you constantly have to update and change it. And so the dynamicness is something that we have built into our platform more broadly. And so we actually had that feedback loop with our clients using our platform and that flows into the rest of our data. The second place, actually, you talk about, like you mentioned, Richie, this kind of idea of how do we enable and make this raw data more accessible more broadly? And something we think about at Ribbon is actually we have a strong orientation towards partnership and being connected towards the main data sources that exist. And so we actually have a lot of partnerships that we've had built with many other organizations where they want their data. Think of like a scheduling partner, right, that has online booking links for certain providers, but they want more. They want Ribbon customers that are consuming Ribbons data to actually access that data. And so we're actually designing a platform where we can connect many data feeds that then pipe into other customers, creating this very connective, broader data ecosystem where we break down the silos of its very fragmented raw data world that we give in. So the data is raw is one thing. Yes, that's definitely a problem, but the fragmentation and lack of connectivity of that data is something. Is a major problem that we try to help address.

We also, I think we care a lot about this problem, about the quality of healthcare data that comes through our pipeline, and about providing this feedback so that we can improve the healthcare data space as a whole. So, as Nate mentioned, we work really closely with our partners, particularly our partners who provide us with. With raw data. So we implement a lot of our own QA processes to evaluate the quality of the data when we get it, and then we also provide feedback that will be helpful for our partners. And we collaborate to try to improve the quality of the data that even comes through to us in the first place before we even start analyzing or processing it. And we've definitely learned a lot from that process. And we think that over time and at scale, this will have an impact on how we ingest data as a whole.

That's so well said, Sana. And I think I'll kind of use an example. The building off of Sana is where she took a question. Many clients will give us a data set, right? Maybe it has 100,000 providers in it. Ribbon will ingest that data, and we'll score it with our models and we'll say, hey, of this 100,000 records, these 30,000 we're really confident are true. And this middle bucket of 30,000, we're pretty confident. We're like 50, 50 on that data quality. It's questionable data. And this bottom 40%, we're really confident that it's wrong. Then our clients will take that data and then deploy it into their production environments, where then they'll filter out the bad data, making their new data set. They otherwise would have had all the bad data. They filter out the data that's incorrect, which is like, one way that we help our clients in the data management use case.

Okay. So just being able to identify. We don't think this is a great record. Go back and take a look at it. That sounds like a pretty cool thing, actually.

It was funny. When we first went to market, we were very focused on our ability for the model to detect data that was correct. It was very confident when data was wrong, but we didn't realize how valuable that was. But then a lot of clients started saying, actually, wait a minute. If you tell me what data that I have that's bad, I would love to know that, because then I can filter it out and not have it drag down my accuracy.

That's great. Yeah, because often you're so focused on, like, the good results that you don't notice the value in the bad results as well. So beyond the healthcare industry, it seems like almost every business has to deal with bad data sets. And so do you have any advice for dealing with dirty data?

Yes. So we need to think about healthcare Data quality in the context of all the use cases that we want to power, and not only about one segment or one particular use case. So when we build products that power care, navigation, for example, there are some metrics that will be important to assess for clients. So for example, accuracy and coverage. And then there are other outputs that we should assess to ensure that what we are delivering is actionable for the ultimate end user, which is the person seeking care. So I think expanding our scope and line of vision will help us create effective solutions to the problems that are preventing more people from being able to access the care that they need and want.

Okay, nice, Nate, have you got something to add there?

The first is this idea of having a really strong North Star on what is good data? Like, how would you actually define that? And getting really crystal clear and make sure that incorporates the context. And like Sonna talked a lot about the context in healthcare, but for whatever domain that might be, it incorporates the context and product thinking of what good data actually is. And once you have that North Star, then the second sort of advice that I would have is that don't be afraid of getting your hands dirty and that messy data, it's not until you actually struggle with the data, really play around with it, understand the ins and outs and why is it bad and then connecting that back to the North Star of what good quality data actually is, that's when you can use all the really exciting, interesting tools and data science and machine learning and AI to bridge the gap and figure out how to make sense sense of that really incorrect and messy data.

Before we finish, I'd just like to talk a little bit about hiring and what sort of skills and roles you think are important within Ribbon.

Oh, absolutely. Yeah. So it goes without saying we're hiring here at Ribbon Health and for what skills we look for, we actually hire sort of two major dimensions. The first is what we call a functional alignment. So this the idea that if you're interviewing for a data science role or a data engineering role or an analyst role, do you have the rough skills and the ability to learn and grow in that role role? We don't expect you to have the exact right skills for the role, but we take a bet on people and their ability to level up and learn on the job. And the second major domain is what we call values alignment. We have six sort of core values at Ribbon. You can go on our website and you'll see what those values are, but we try to be really forward and open about what our values are at Ribbon. Help to make sure that people have joined the organization or there's strong align between those folks to create a special culture within Ribbon. And that's how we think about it. And in terms, of course, a big part of joining any kind of company enterprise is being really excited about the mission. And so here at Ribbon Health, the way I like to paint it sometimes is for like, travel, right? We think of Kayat.com, you can say, I want a flight from point A to point B to these airlines that's nonstop. With this, the price point deploying at this time, there's so much information, there's so much data infrastructure that's making that search and decision possible. And it's just not the case in healthcare. And here at Ribbon Health, we are building a future world where that is possible. We build that really critical data infrastructure that makes those kinds of experiences possible for end patients. And if you're excited about that, we'd love to talk and meet you.

You all right? Super. I'm sure you'll see a few job applications coming your way then from our listeners. Thank you very much, Nate and Sun. It's been a pleasure to speak to both of you. I hope you've enjoyed coming on the podcast.

It's been great. Richie, thank you so much. Been absolute blast for having us. We really appreciate taking the time and asking so many thoughtful questions.

Yeah, and thank you so much, Richie. Really loved talking about the kind of work that we do. And yeah, just thinking about all these problems with with you.

You've been listening to Data Framed, a podcast by Data Camp. Keep connected with us by subscribing to the show in your favorite podcast player. Please give us a rating, leave a comment and share episodes you love that helps us keep delivering insights into all things data. Thanks for listening. Until next.