This YouTube lecture explains the implementation phase of AI projects — “productionizing” a developed model, preparing it for scalable deployment, monitoring performance, and ensuring usability. It uses a case study of a maternal-health messaging system in Nigeria: a simple single-layer model (originally developed at Edibon) was retrained for local languages and data; annotation was performed by clinic staff for accuracy and privacy; automated language-detection handled easy tasks with manual backoff and human review focused on uncertain predictions. The lecturer highlights two gating questions for implementation: Is model performance acceptable? Can end users successfully use the system? If not, teams must return to the design phase. The video ends by transitioning to the evaluation phase.
Chapters
Once you've looked at your data and designed your model strategy and your annotation strategy, figured out how you're going to do with data privacy and security, and plan your user experience, then you're ready to begin implementing your system. In this phase, you'll get ready for production by running any final training and testing of your model and moving it into a scalable production environment. We will be concerned with monitoring your model performance and understanding potential failure modes, among other things Here. I'm just calling this productionize your model, which really just means making the model that you developed in a sort of tested environment in the design phase, more available and reliable and robust as part of your production system and available for your end user application. And then as you're ready to put your model into the production environment and iterate and integrate your model there, you you'll be at a stage where you're able to do end to end testing. So look at everything in your system from one end to another in terms of data throughput, model monitoring and reliability, systems updates and user experience. For the project that I've been describing about maternal health in Nigeria, the model we used was a very simple single layer model that we'd originally developed at Edibon for industry applications retrained for this specific use case and the languages, metadata, instruction, unstructured data that we had access to as part of your report. In the system that we designed for this use case, one of the requirements was that the annotation had to be done by the staff in the clinic. This was for a couple of reasons. One was that they had the expertise, they'd already been working with these messages and a third party might not be so accurate. And just as importantly, it was for privacy reasons. We didn't want to employ additional people who'd be able to see the personal health information of the people being supported by your report simply for the purpose of updating a machine learning model, conscious of the time it would take these healthcare workers. The way this worked was that for a very small number of tasks like language detection, they were performed in a very almost automated way by the model because they're highly accurate with a backoff where someone got something that was not in fact in language they spoke, they could quickly manually reroute that. In other cases, humans were reviewing model predictions, but with a focus on predictions where the model was uncertain. So we would raise more particular use cases where a model could not make an accurate prediction, not say what the model predictions might be, and then ensure that the manual annotations still took place there, so that we weren't in any way biasing the output by having too many incorrect model predictions. For any project that you're working in to complete the implementation phase, you need to be able to answer yes to two simple questions. Is your model performance acceptable? And are the end users able to successfully use your system? It's possible. You realize that there are problems with your model performance or usability of your system during the implementation phase, and you will need to return to the design phase and work out those problems. In our case, at this stage now, after months of work, we had developed our models, integrated them with user experience, and tested the whole system with the staff at the clinic as end users. The model performance was acceptable and we could show we were continuously improving the model performance and in turn continually improving the overall volume and response speed of the clinic staff, even though they still had to manually review and categorize some portion of new incoming messages. And so with that, we're ready to go live and deploy the system in reality. The project I'm talking about here is just one example of the many possible designs and implementation phases that you might have for any project that utilizes AI. Your particular project is likely to involve a whole lot of technical challenges related to things like system uptime or low latency predictions or retraining time, and a lot of other factors about, I guess, practical real world AI problems that typically aren't part of more theoretical or academic focused AI courses or the kind of labs that you're probably used to. And so we won't be addressing these kinds of problems here in detail, but I do want to highlight that they exist. And if you're interested in being part of a group of people that deploys real AI systems, and then I do strongly recommend that you also get an understanding and an experience of the kinds of problems you face in deploying any kind of technology at scale. The important thing for you to focus on in this program is the big picture about going from the problem definition that you identified in the exploration phase to the evaluation of the project on through the final phase. You should at the end, Stu will be able to evaluate your project it in terms of the problem definition that you initially set out to do. And so this is the evaluation of your results and the user experience of the people who are using and getting a benefit from the system that you've created. So please join me in the next video where we'll wrap up this lesson with the evaluate phase of this project.
