#15 AI for Good Specialization [Course 1, Week 2, Lesson 1]

With a successful exploration of your project, you've identified and communicated with the stakeholders, you've got all the information that you needed to clearly define the problem that you aim to work on, and you've determined that AI can most likely add value in this particular scenario. So now you're ready to design your solution. In the design phase, the steps you'll take are first to prototype your solution, which involves looking more closely at your data and testing out some models. At the same time, you'll also be thinking about how you're going to address data privacy and data security issues. As I mentioned in the first week of this course, whenever you're dealing with data that includes information about people or property, you have to be very deliberate about how you're going to handle that data at all stages of your project in order to ensure that it remains secure and private. And you should not use any personal data without the explicit permission for your specific use case. So whatever project you're working on, whether it's in healthcare for at risk populations or an app for a wealthier population, you should be defaulting to private data practices. In all of these cases, you should be ensuring both the privacy, the security and the dignity of the people whose data you're storing. Finally, you'll design the user experience for your project. And this can mean very different things for different projects. So let's walk through each of these steps of the design phase for the maternal infant health project that we started with in the Explore phase. In the previous phase, we explore the data and determine that look like an AI solution might be able to add value in this project. Now, in the design phase, we dug in more deeply into the data to verify ways in which our data might need to be cleaned or prepared. For this particular case, the data we had contained text messages and keywords or categories that staff had assigned to those messages. At this stage, we set out to determine what the minimum number of labels or categories should be in our training data, and then consider which categories would be hard to process automatically, which might be easier. In this case, identifying the language of a message was relatively easy. In fact, language identification is among the easier natural language processing tasks. More broadly, on the other hand, categorizing the content of the message could be very hard. For example, trying to classify whether it was related to maternal health or some other problem. Even that simple binary decision was not particularly accurate, and the accuracy varied across the languages, mostly related to how much data we had in a given language. Once we had investigated the data in more detail, we designed a Model annotation strategy, we would have the clinic staff annotate some more of the incoming messages, in particular in particular topics and some languages where we had less data initially. So this would enable us to use those additional annotations as new training data so that we could update our models and automate more of the processing much more quickly. As part of this effort, we investigated what kind of performance trade offs there might have been, for example, between a state of the art machine learning model or a much simpler model. So with a state of the art model, obviously the predictions would tend to be more accurate, but it could take longer for those predictions to occur on the incoming messages. And in particular it could take much longer for a model to retrain on additional data. And so in this case we found that somewhere in the middle, but tending towards a simpler model, where was the right solution for us, because it was more reliable, we understand the edge cases that might occur more deeply, it was more interpretable, and in terms of user experience, that model could retrain in the order of minutes compared to a state of the art model that might take hours or days. So this meant that the people working in the clinic could experience in near real time the feedback that they were giving and how that improved the automation of their task. And what we found was very much in line with what we'd found in industry, in anybody as well. A very large number, although not all of our customers, would actually prefer a model that could be updated with new data quickly over a model that was absolutely state of the art. And this is because in many real world situations, a few extra human data labels is more important than hours and hours of extra training for a more highly converged model. And so while that wasn't the case in all cases in industry, some cases did in state of the art. We were not surprised to find that a simpler but more well known and reliable model was the right choice for this particular use case in healthcare. Here we really set out to confirm that AI could indeed provide value as part of the solution. So the baseline for a performance test in our AI was the purely manual solution that already existed, and in particular Tukey performing indices. So in terms of volume of response and response time, so how many messages could be responded to in a given period of time by a healthcare worker, and how much time passed between a message coming in and the message being responded to. And then we also ensured that the accuracy of responses remained constant. We weren't looking to make the people in the clinic more accurate in this case, or that can be a Goal. In this case, by having multiple people in the clinic look at the same messages and evaluating them, we're able to determine that with very light suggestions and filtering by AI, we did not have a lower overall quality in response while we were trying to move up the processing volume and bring down the processing response time as much as possible. Now that we designed a system where we saw that AI could likely provide an improved experience, we're able to move into the implementation phase for data privacy. In this case, the data set contains personal information from individuals, including particularly sensitive information about healthcare. So we were careful to ensure that our data storage and handling was secure enough for the task. This meant, for example, the messages were never exposed to people who couldn't already see them as part of their regular work. And so we at Itybahn didn't have access intentionally to in any way see or download the data. And this is another good example, I think, of where what we learned in industry helped us a lot. In this particular healthcare use case for a lot of our industry customers, we also couldn't see the data that they were working on for sensitivity reasons and in many cases, regulatory reasons. And so, having already built these sort of protections and tested them and had them validated by third parties, we could then be more confident that we were doing our best to protect the security and privacy of the healthcare communications, as we added this new step of also putting them through an AI system in addition to their existing manual processing system. As for the user experience, it in this case, the end users were the healthcare workers in the clinic. And so we aim to make the work more efficient with an automatic message categorization tool. And so, in successful implementation of this project, those healthcare providers would see a benefit in terms of having to do less work to process incoming text messages while providing better service to their patients in reduced response times, and in turn, helping the community with more messages being processed. Once you've completed the design phase, there is another set of questions you should answer with your team before moving on to the next phase of development. First, regarding the data, how will you address the issues of imbalances, biases, privacy or other concerns with your data? Secondly, what kind of model will you implement and how will you measure its performance? Then, how will your design successfully address the problem that you defined in your Explore phase? And finally, how will end users interact with your system? But it's possible that in addressing these questions, after you're working the design phase, you'll recognize that you've run into issues like unanticipated challenges with the data or how you might build a model using that data to address the particular problem that you defined. Or in designing the user experience. In this case, you may discover that you need to return to the Explore phase and get more information from stakeholders, investigate other data resources, or even re examine the problem that you're trying to address. For this case study, our data considerations involved securely storing and handling personal information. And so for this we use systems we're already designed for industry clients, such that us at Edibon or anyone new at the clinic or at UNICEF who didn't already have access, had no way to access this data or to inadvertently download or share it with somebody else. We designed a model that was a version of what we'd already deployed for industry clients. So again, this is taking knowledge that we had developed and refined in large scale industry situations and be fortunate enough to apply them for free into this healthcare situation. And so for this application, we also adopted the strategy of having the existing clinic staff perform additional annotation of incoming text messages to build up a database that would allow the model to become more robust in categorizing the variation of languages and categories that we care about. This system would address the problem that we were working on by allowing healthcare providers at the clinic to process a larger volume of text messages and provide faster response time to their patients. Then users of the system, the stuff at the clinic would receive automatic updates on the categorization and prioritization of new messages. They would review the sign categories and priorities for accuracy and manually reassign categories of priorities to aid further improvement of the model. Now, of course, I'm glossing over a lot of the details here. For this particular use case, we spent many months working on the design phase, and there have been cases where I've spent years in the design phase of systems that ultimately weren't that much more sophisticated, which needed to be built very slowly and deliberately and carefully to ensure that we weren't violating the do no harm principle. So in your own project, you might spend any amount of time really verifying some of your design decisions with respect to the AI model that you choose and the end user experience. So once you do have what you think is a well designed solution, then you're ready to move on to the implementation phase. So please join me in the next video to look at how we move from design into implementation. For this use case.