AI competitions & cloud resources

Hi everyone, this is Daniel coming to you with a slightly different episode of Practical AI this week. Recently, Purdue, Microsoft Informs and a few others put on a case competition which included student teams from across the nation. Around 170 something teams all working on a shared task related to image captioning. This is a task where an image is input to a model and then the job of the model is to output a text caption corresponding to that image. And I had the privilege of getting to be one of the judges for this competition and I took the opportunity to interview some of the sponsors and the participants in the challenge Also. It was really fun because the competition used some of our data, the data from SIL International in the Bloom captioning data set which includes image captioning data for a lot of languages. But specifically this competition focused on image captioning and Thai Hausen Kyrgyz. So hope you enjoy the discussion of this challenge. And here we go.

Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive and accessible to everyone. Subscribe now if you haven't already. Head to Practical AI FM for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly IO. We deploy our app servers close to our users and you can too. Learn more@fly IO.

Welcome to a very special episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and this is a very special episode because I'm here at Practice Purdue University judging a really interesting case competition, Data analytics for Good, that's sponsored by Purdue University, Microsoft, SIL International, my organization and informs, and I'm here with Matthew Lanham, who is the Academic Director of the Ms. Business analytics and Information Management Program, or BAME Program, which we've had the privilege of getting to know each other over the past few years and it's really cool to collaborate and judge this competition. Matthew, could you tell us a little bit about it and how it came about?

Sure, yeah. So as the academic director for the Business Analytics Program, my job is basically trying to make sure our students are involved in analytics and data science competitions. And over the last seven years since we've had this program, our students have won or placed in many of these national competitions. So we've got a really well established brand and name out there and we thought, hey, why don't we create our own national data analytics competition and let's do something that's for good. Not Necessarily just focus on trying to make money.

Yeah, yeah. So tell us a little bit about the actual problem that the students are working on and maybe a little bit about, like, the mix of who is involved in the competition from across the nation.

Sure.

So the actual problem is sponsored by your company, SIL International. And basically what they're trying to do is use natural language processing to do image captioning, which is not a trivial task by any means. So when we put this problem out there to the students, they're like, oh, my gosh, what is this? And the great thing is, it's not something that you would see in a traditional NLP course, this kind of problem. And there's been a lot of great learning involved. Overall, we had 172 teams across the nation apply and register for the competition. And there was 36 universities that were represented. Two of those were outside the United States.

Wow, that's. Yeah, yeah, that's really great. Yeah. The competition. So this image captioning, it's been cool to see because recently SIL put out this data set around image captioning, and it was convenient timing because about a week or so later, you reached out and said, hey, we're running this cool case competition. Do you have any cool data sets to work on? And that worked out really good. I think I've heard students all learning a lot about natural language processing, but also the world's languages. So when you try to do image captioning in Thai, for example, there's no spaces, and you can't tokenize words with just, like, spaces. So just even realizing things like that has been quite interesting to see for students. And I know I work sort of halfway through the day of judging at this point that we're recording, and I've already been surprised and encouraged by a lot of the solutions. One of the other sponsors of the competition is informs, which I'd love, you know, people to know a little bit more about what INFORMS is generally, because it is, you know, a vibrant and large community. Could you tell us a little bit about what that is?

Absolutely. So the INFORM stands for the Institute for Operations Research and Management Science. It is my favorite professional organization. So it's been around for many years. And, you know, we used to call it Operations Research and Management Science, but now we refer to a lot of the stuff that we've done for years as analytics and Data Science. Within INFORMS, there's also a certification program called the Inform Certified Analytics Professional, or CAP. So I'm a CAP. I was one of the first CAPs many years ago. And basically the whole idea with the cap is you don't have to be a technical person to get this cap, or you don't have to, or you don't have to be just like the business person. It's really for everybody. And the whole idea is, how it came about was the people at INFORMS worked with business professionals from all different areas of analytics and data science, operations research, to identify what are the key kinds of tasks that you would do as a professional. And what they did is they came up with basically seven domains. Business problem framing is the first one. Then analytical problem framing, knowing your data, methodology or approach selection, model building, deployment and lifecycle management. And if you guys, if the audience hears those kind of things, they're probably thinking that sounds a lot like crisp DM that we've all heard about, you know, in school or at some point following a process. And that was the thing is, a lot of times when we're working on these problems, you got to follow a process. And they kind of extended that crisp DM framework and there's just a lot of tasks within there that we hope people are aware of and think about when they try to develop solutions in practice.

Yeah. And we've been utilizing that framework in the judging here at the competition, which has been really, really useful, I would say, to consider these different elements of the process. And maybe how would you say after working with teams in this process, you know, for years now, actually like thinking about these different elements, how do you think that kind of rounds out someone's view of like an actual, solving an actual business problem with like AI or analytics or data science in ways that maybe is sometimes neglected in a lot of just sort of process. When you step into a problem, what are the main areas that you think kind of stretch students? But then as they go into the professional workspace, how do you think that sets them up for solving real world problems?

Great question. And this is why I just love informscap and why we make our students follow the INFORMS CAP when they do projects with companies is because you'll see a team that maybe they're the data science team, the real technical team, and they love to get into the nitty gritty details and there's absolutely nothing wrong with that. But at the same time, you need to know your audience. And I think just following those seven domains, those seven domains for INFORMS cap is important because before you even get into the nitty gritty details of your problem, you need to be able to say, well, how does, what is the business problem Here. And then how do we frame it? What are the possibilities of framing it into an analytics problem? And then. So that's the front end of the thing, right? And then you'll get in the middle part, which a lot of the stuff that we would talk about as data scientists, the data, the methodology, the model building, all the stuff that we really like to do. But then the last part of it, the deployment and life cycle management, that's so key. Right. So that's when you get into architecting and developing the pipelines, all the stuff that I know you're an expert in, it's just so key. So basically that's what the imports cap's doing is to say, hey, let's lay all this out to make sure that when we architect a solution, we design a solution, we try to create a solution to our problem, that we've thought about all these things and we haven't missed anything along the way.

Yeah. Well, thank you so much again for helping organize this, Matthew. It's been a pleasure. And yeah, really just appreciate your work on this and also your work with informs.

Thank you. Dan.

I want to share exactly how the funding happened. It was just about Azure and the socks. I mean, what is this from Microsoft?

Oh, oh, yes. So from Microsoft. So that's actually a really important point. So one of the interesting things about this competition that's different than all other competitions that my students have participated in is we designed it where there's three phases and phase two is where they actually work on the problem provided by Dan's company. But the phase one is we wanted to actually provide some training on cloud services, which a lot of people in industry, they know you got to be familiar with these services if you're going to architect a solution and put it into practice. So Microsoft offers free training on their Azure AI. Basically everything's free if you're a student. And they also offer students free practice exams and certification vouchers, which is amazing. So we told them, they said, you know, we would, we would like we could get a whole bunch of students to participate in your training events if we can kind of piggyback off of you. And they said, absolutely, we love this. So that's what the students did in the phase one was they had some training from Microsoft professionals, some of them even set for certifications. And then in phase two, the goal was to try to apply some of those web services for this particular problem. The last phase, phase three, is when the top teams that performed the best in the Kaggle competition would come on campus, present their solution, and then show how they could follow the informs, the seven informs, CAP JTA's to architect their solution. That's how it all came about.

Awesome. Yeah. And that definitely brings us right into the Microsoft involvement with this competition, which also pleased to have with us Mark Tabladillo, who is a cloud architect with Microsoft. Yeah, thanks for being here and being part of the competition. Also Microsoft's involvement in this. It's been interesting, as we've seen some presentations already, even to hear how students are making that realization about, hey, I've been working on my laptop, solving maybe data science toy problems in my courses or something like that. But I got to this problem and even one of the student groups said, hey, I bought more RAM for my laptop to try to solve the problem. But then they're like, that's not doing it. So then they started thinking about cloud services. So how do you think, Mark, as. I guess my question is, as students and maybe people getting into the field are kind of making this realization around the resources required to solve actual business problems, what are those ways in which they can kind of start dipping their toes into cloud and experiment with things to kind of expand their horizons in terms of what's possible without knowing about. Maybe people don't know about Docker and Kubernetes and all that stuff yet, but they want to start dipping into more resources. What's a good way for people to kind of get into that? As you've seen these students kind of do that.

Sure. And I think there's more resources available more than ever to do self learning. And it was maybe 10 to 15 years ago where it was very common to go to a bookstore and find these big thick books with animals on the front. Yeah, maybe they, you know, they would be training books for technologies. And of course, the publishers are still out there, O'Reilly's still out there producing books. And I have a friend, by the way, who's coming out this month with his book on practical machine learning and AI, Jeff Proses. And I'm so proud of him to write a new book. But the point is that so many things are online and in the Microsoft ecosystem, there was a time when you had to pay to even get the proceedings from a conference. Like you didn't even go and you couldn't even get the recordings. Well, now Microsoft's making a lot of that available for free and in a way that people can find it. And so I think for the audience of this podcast, would love to have them look at YouTube. What's available there. Microsoft's got a few channels of content on there, and that's a good way to get started. Sometimes they're short sessions, between five and 10 minutes. Sometimes they go to an hour. But that's definitely one way to get started.

Yeah. And could you help us? I think it's good for people to kind of organize certain categories in their mind. I've heard students talk about like the Microsoft Azure kind of studio environment. And then there's other things like these, like cognitive services and like managed AI services. So how do these things differ and like how, how might they be used or how have you seen them being used either in this competition or other places?

Okay, so the unifying thing is either. And you can pronounce either Azure or Azure. Both correct. Both correct.

Good, good. It's good to have the definitive answer on that.

This is the definitive answer for. For all time. I tend to use Azure, but just because I don't have it. The unifying factor is Azure Active Directory. So that's the main authentication path to going into an Azure subscription. And the subscription itself is, think about it like a credit card. If you sign up for subscription, you would have to put your credit card on there to pay the bills. Again, another free path. Microsoft offers a lot of things. We have free subscriptions and we even send our customers to go get them. And I even tell my customers, I say, you know, it does run out. So I say, well, go make a new email@outlook.com and then just make a new one. All right? We want you to get hands on because there's no substitute for experience. And that's even kind of the point of this competition. You know, you can study it in the book, you can do class exercises. But two things are true. First of all, putting into practice and working as a team. And that's what we're doing in this competition. So back to your earlier question, now that a team may all join the same subscription. The Azure Machine Learning Studio is our flagship technology for machine learning, and it can run on regular CPU or GPU instances. We have regions available around the world. I happen to work in the federal space, so we also have specific clouds just for that use. And there's different sovereign clouds now. And Microsoft's now beginning to build out specialty clouds on top of our regular clouds and also call out to for students. We have a lot of promotionals for students where they get things at discount rate. And also nonprofits. Nonprofits will get special treatment inside the Azure ecosystem. But let me go back to the original question. All right, so there are two focuses that people will have. One is machine learning and it tends to be thought of now as building a model that is the way to think about the products. AI is the marketing term for all our technologies. Now if you go to the main website, everything is AI. However, inside the Microsoft technology there are cognitive services and those are considered mostly APIs. They're rest APIs and they're already pre trained models and they do certain things. And we've seen some of the teams here at the contest, they've been using maybe computer vision or text to image or image to text. You know, those types of technologies are already out there and Microsoft's not unique. Other vendors have these and Microsoft is now behind the scenes supporting all open source technologies. Some come pre built inside Machine Learning Studio. Like we have a certain version of a Python kernel that will run inside there and another technology. Okay, so going back to the Azure Machine Learning Studio, we have doubled down on McLflow as the way we are organizing our workspace. So on the new version of the API that is the path forward and that is open source.

And could you just describe a little bit what that is? Mlflow.

Mlflow. So it is a way to organize experiments and training and models and model deployment. And it has its own syntax in terms of vocabulary and API. The way to work. Microsoft's not alone in using MLflow. There's other vendors that use MLflow in their technology. But it is a way to organize the technology and the assets. Microsoft decided to make that native to our own API and how that works.

Great. Yeah, I think one of the things as we've been in the presentations here today, I think when I was in grad school I programmed, but it was like I'd use matlab or Python or whatever and I did some things. I had no concept of how infrastructure worked in industry or the whole thing about doing programming in academia. It's just not always a parallel to industry. So coming from the Microsoft perspective, I found it really encouraging to see students, you know, saying things like object store or you know, model registry or like these things and thinking through like the architecture. I know that's one of the things in the informs like they emphasize is that sort of model deployment and model life cycle management. So yeah, do you have any words of encouragement for maybe those, you know, listeners who are, you know, again getting into this space or maybe their students in terms of getting hands on with actual infrastructure that people use in industry and how that benefits kind of, you know, your understanding of how to create value with what you're producing rather than just creating a cool model.

Right. So let me call out a few things. Now, some people aren't so lucky to be even admitted to Purdue. And if you were, I would certainly want to come to a program such as here, and you can participate in all these cool events. But short of having that either undergraduate or graduate experience, Microsoft has, and this is what I believe is the front door, we have something called AI Business School. It is a series of courses that show how to tie in the value of AI in a business context. And a lot of the videos were done by our own leadership and they've shown how we've used AI inside the Microsoft business. Now, it's not intended to be a catalog of all possible ideas, but it does kind of COVID the landscape of, you know, along the lines of the INFORMS domains, it covers the landscape of here's we have a challenge. Here's how we're going to use data modeling and then put in production. And here's how we're evaluating use. And it just gets people started. So it's something I do recommend to my customers because people do have different roles and even, you know, what I'm working, we're working on internally and we're now rethinking through who are the Personas of people who touch data projects. And we talked about the domains, right. But we're going now to thinking about, all right, so who's that person? What does that person do? Are all modelers the same? We don't think so. So we're now beginning to think that because we're now serving a large internal community inside Microsoft in terms of our programming. So that's the first thing I think about is the AI Business School. And then also in terms of getting started, we have, inside all our technologies, we have tutorials and samples to get started, little sample data sets, notebooks that run quickly, they don't take hours, but they show you a variety of things. They're guided toward specific outcomes. They begin to get better. I've seen Microsoft examples and how they have grown in the last 15 years. And they're just getting better and better because they're getting better minds thinking about them. But I'll also call out, you know, Microsoft also is always looking for partners that want to share their stories. And we have a lot of case studies of companies doing things in different industries, whether it's for profit or non profit. I'll call out, you know, one example, we work with the Metropolitan Museum of Art to Digitize their entire holdings. Now, I don't know if they did 100%, but just they had the same challenge as a lot of art owners. And that is not all their collection is on display. And some researchers want to have access to those products. So that's an example of, you know, anytime we do work with major organizations, we put those ideas out there. But more practically, and this may even be helpful for students or users, we have what's called the Azure Architecture Center. And inside, we see many architectures, very similar, by the way. We're only looking at the top presentations, but we are seeing architectures presented. It's the type of thing that I do in my own work. And the architectures will be in there, the diagrams, and also the case study of what it did and, you know, kind of what the use case is. So it gives people, again, a catalog of different ideas of how do you use the different resources that are available. So between all. That's a lot.

Yeah. Yeah. Thank you so much, Mark. I will definitely link both to informs, what they're doing to Purdue and the BAME program, and to these resources from Microsoft in our show notes for the podcast. So make sure and check all those things out. Thank you again, Mark and Matthew, for what you're doing on this and looking forward to hearing the rest of the presentations. Thank you, Dan.

Thanks.

All right, well, I'm here with the winning undergrad team from the competition, the image captioning competition, which is from Butler University. I've got Chris Stein, Andrea Markey, and Aaron Pinner with us. So congratulations on winning the undergraduate portion of the competition.

Yeah, thank you very much. We are thrilled.

Yeah. So your solution was really interesting, actually, all the undergraduate presentations, I was surprised. They seemed like graduate student work to me. But tell us a little bit. So the task again was image captioning. So just tell us one highlight about maybe one of the challenges that you faced in the competition.

So I think that one of the big challenges in this case was about the data set, because for certain languages, like the Hausa languages that we have to work on, the data set was kind of small, and also there were not much variety into the pictures. So that was the biggest challenge. So we kind of overcome that challenge by either artificially augmenting the data set or adding new pictures to the data set.

Great. Yeah. And specifically, part of the competition was thinking about different languages where maybe image captioning isn't supported. And I think one of the things I appreciated about Yara's presentation as well was thinking through the business implications of something like this technology of image captioning that could enable new or expanded possibilities for local language communities that don't have this technology. Could one of you comment on maybe what you envision in terms of the impact something like this could make in terms of image captioning for a language where it's not supported yet?

Yeah.

So our idea was almost to create a web app or a mobile app that small businesses using, like Kyrgyz or Thai or these small languages could go on this app and submit their photos there. Right. So everyone's got a cell phone, you know, in all communities nowadays, and if they can utilize that cell phone to almost leverage it and, you know, upload those pictures right then there and get a caption, you know, that is handy for the small business and SIO and their mission. You know, a small business would want to use this because, you know, a lot of people are drawn to websites because of images. They click on images in Bing and Google. So if we can help small businesses, especially if they have a user base with a minority language, that helps both SIL and the company. So really there's a monetary win, but really we're helping the world in a way, so it's really neat.

And also, sorry, if I can add something like, last night we had a brief talk about the languages around the world. And so also there is not only a business implication to this challenge, but we know that also we are losing a cultural heritage like you mentioned last night, that every two weeks, one language is lost forever. Right. So there is no way that we can keep those languages alive. So if this can also help make the world more open towards, you know, those small communities, you know, this is also a good thing also for the world because it makes the world a more interesting place. Right?

Yeah.

Yeah.

Awesome. And maybe a comment from Aaron, as far as this competition, maybe what's one of the highlights of something that you learned throughout the competition that you view differently now, either in terms of the technical challenges and that side of things, or the business problem or something that you'll carry with you throughout the rest of your work?

Yeah, I think one thing I learned was just. I think the challenge really opened my eyes to this problem that existed. I would have never thought about using AI or machine learning in a way.

That directly impacts languages.

So I think that was definitely something I learned and was really interesting.

Great. Well, congratulations again. Hope your travels back home are safe and. Yeah, congratulations. Hope to stay in contact.

Yeah.

Thank you very much. Thank you.

Okay, well, I'm with now the winning graduate team from the Purdue using analytics and data science for good competition. This team is from Georgia Tech. Here I have with me Harsha Varun, Ravi and there was another team member, Sanchita, who couldn't make it to the competition here in person, but want to acknowledge her and her contribution. So congratulations. First of all, you all are the first out of, I think it was 170 something teams in this competition to come up with an image captioning model that performs well on three sort of diverse languages from around the world, Thai, Kyrgyz and Hausa. So first off, congratulations. And I think one of the things that was really interesting to me about URLs solution is one kind of looking to state of the art models like Clip, which was something that featured in your solution, but then also using sort of a multi stage approach where you actually determined if a caption existed already that you had in a database of captions and then if it didn't exist, generating a caption. So could you, one of you describe a little bit about how you kind of eventually got to that solution, how you considered using Clip and got to thinking about that direction.

So I think the problem itself was quite challenging. When you look into the data set and actually see what the data is, you can see poems, you can see philosophical statements, moral statements, parts of stories. And if you want to predict these kind of statements, you need information about the previous part of the story or the further part of the story in order to even build a model, a zero shot captioning model. It's very difficult to achieve a good zero shot captioning model for such kind of prediction tasks. So the next step that we thought was maybe we could do some sort of classification model that was the original thought process that we could select from a corpus of sentences. Can we select a sentence that best matches this? And from there we started researching basically and when we went through hugging face models, we found the Clip model and then we research further, we found a multilingual clip model that could handle different languages and it sort of went through that process and when we actually used it, it was decent. I wouldn't say it was perfect, but it certainly improved our overall solution quite a bit.

Yeah, yeah. So when you were thinking about this idea of looking to existing captions and using those, when you could, how often in the data set that you were looking at, which is this Bloom data set, how often did you have to generate image captions, excuse me, versus maybe looking to a list of captions and using one that Pre existed.

So even in the training data set where we had the images and all the captions that we needed, when we actually used the multilingual clip model, it was more about like 20, 30% that were matching and we had a very low threshold by that itself. And we didn't want to lower the threshold because we didn't want to get more false positives in a way. And basically we just decided on that threshold. We didn't do any optimization on that, particularly from there. When we actually used the model on the test set, we suddenly got a huge jump in the score. That was basically it. So we were covering about 20 to 30% of the images. Even when you had all the captions for the images, only 20 to 30% were actually matched by the multilingual clip model. All the other images went to the generator model.

Yeah, and you mentioned sort of clip hugging face. These are all, you know, kind of the industry standard state of the art sort of things. As a team getting into this problem, what were the kind of challenges that you faced in terms of maybe even finding where to start or maybe it's computational challenges or other issues.

So when we initially started with this dataset, we were like, we were stumped, honestly. Like we hadn't even heard of a model that could generate contextual information with as much depth as was required by the solution here. So as I said, we did our initial EDA with say Microsoft Azure using their computer Vision API and translator model. So when we actually used that, we thought, okay, these are reasonable guesses, like okay, human would make these guesses. But we had to go deeper so we had to match. We had other ideas as well. We thought of clustering common images, maybe they belong to the same story or they're part of a single book or something like that. Yeah, so that's how we got started off. The EDA that we did helped a lot, like understanding that they were poems and stuff mixed into the data helped us look for more deeper models that could generate context.

Great. Yeah, thank you for that info. So I think as you, as you look forward to kind of, I mean, I know all of you will be going very far just with the innovations that you've demonstrated here. I hope that maybe when you own billion dollar startups, you'll hire me to like sweep the floors in your startup or something. But how do you think kind of working on a solution like this from start to finish has influenced how you'll think about maybe AI or data science problems in the future? Any input.

So the amount of good the AI can do to the real world people. I have looked at lot of things the SIL does. It's actually improving a lot of language proficiency among the students and also increasing the educational rate among the people who are not studying so much. So the amount of good the data or AI can do is will definitely influence our thoughts, influence future as well. So it's like the kind of use cases all these things can have on the lives of the people are definitely going to stay home. And we definitely try, we will definitely try to contribute wherever we can by keeping this in mind. So this is going to stay with us forever.

Great. Great. Well, thank you for your participation and congratulations again. I hope your travels are safe back home.

Thank you.

All right, that is our show for this week. If you dig it. Don't forget to subscribe. Head to PracticalAI FM for all the ways and if Practical AI has benefited your life, pay it forward by sharing the show with a friend or a colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets to Fly IO for back in our dynamic requests to Breakmaster Cylinder for the Beats and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.