Content and Collaborative Filtering

What I am going to be talking today is the conceptual framework of what a recommender system is. Okay. I'm pretty sure all of you have come across multitude of recommender systems, right? Like Agastya is a Barcelona fan, right? So now if I start selling things which is Real Madrid, what is the chance that Agastya is going to be interested in that stuff? Or if Ronit is a CSK fan, right? And then I tell him, you know what? This is what Kolkata Knight Riders helmet is all, you know, you do, you'd like to have it. And Ronit is like me, I'm a CSK fan, right? Like a person like me who just wants to watch Indian cricket team play. And you keep sending IPL information to me, I might just get irritated and like I don't want anything from you, right? So it is very important, right, that you get the right kind of recommendations. Okay? Now this might look very funny, okay, this picture that I have here. But this picture gives you a lot of information. Can you all see this picture? Is that top bar kind of. It's all fine, right? Okay.

Okay. It covers a little bit like, but I can understand what it is.

Okay, is it better now.

You change the screen here?

Yeah. Is it better now? Can you see?

No, that like viewing Nikki Verma's screen comes up like there. So it's like that.

Oh, there's a bar there, right?

Yeah.

Okay. Yeah, I put it on full screen. That's why I was asking. Otherwise usually the bar would not be covering any view. But what I'm trying to show you over here is a recommender system right now this is a bizarre. This is supposed to be funny. I didn't see anybody smile except for Namrata. But this is, this is not something that you would see in a daily life. Yeah, but you do see recommended systems. Neil, can you just. Yeah, you do see recommender systems in a lot of things in your daily life. Like you know, a multitude of things that you interact even as kids or even as grown ups or even as, you know, retired people or working people in your workplace while you're driving, while you have the GPS on, you know, all kinds of things that we interact with. Especially when we are trying to buy things. We do get a lot of recommendations. If you look at your Google page and if you have your own Google account, there would be certain recommendations that keeps flowing in which would be very unique to who you are. Okay. It would be very, very, it would be designed for your perspective. Anybody can give us any interesting example of recommender Systems that you face. Yes.

Well, so like in YouTube, like if you watch something like over and over or something similar, it will give you like a recommended bar where you can see stuff. Or like even like in Google, like, it gives you like news. If you like view certain types of news or something. So it will come up there.

That's good.

Spotify has a very good recommendation.

Spotify has a very good recommendation system anymore. Yes.

You go first.

Yes.

Oh, so on Amazon, when you buy something and you scroll down, it says customers also bought. So if you bought like a couple of paintbrushes, if you scroll down, there might be other brands are like paints and other supplies. So that's. And like for example, you bought some sort of toy car or something. Like, I recently got like three toy cars from Hot Wheels from Amazon and with it it showed customers also bought. So that pack of cars, there was six of them. And then there was some other toy cars.

Not necessarily the same ones. Okay. And I had Shakti then. Alia. Yes, Shakti.

Also, when you go on to YouTube just to watch something that will help you, it also shows people who watch that. This channel.

Yeah, so. So it gives you more different types of views. So besides recommending, it also shows you some channels also. Yeah. Alia, you had something. You had your hand up. Alia, I can't hear you. Your mic is not on. Okay.

Who else Neil wanted to.

I just want to hear from everyone. N wants to say something. Yes. Nan. That's Ahan. Oh, sorry. Sorry. I'm so sorry.

Okay.

Yeah, yeah, yeah, yeah.

So yes, in Alexa, when you tell her to like a sing a song, she also tells other people who are listening are listening to this song. Like what other songs they listen to?

Okay.

So then I was playing Believer on one on the Alexa device. Then you would say other people also heard like thunder, like other sounds.

And also Netflix, you get people are watching. Yeah. On Netflix also we do get. Yes. Watched. Neil. Because what you watched. Yes, Neil, you also had your hand up.

He was going to say Netflix, but I had an idea. Facebook, it recommends friends depending on where you go and all.

Okay. I think yes. There are not many people who are here who could be in Facebook legally here. But yes. Okay.

I saw my mom's phone.

Good one. Okay. I know what you mean, Alia. Yes.

So when we got my. When we got my knitting supplies, we my scroll down and there was a knitting book and that's how, like that's how we. I learned to knit a little bit.

Okay.

We saw the recommendation and you Saw.

The recommendation of the knitting book. Yes. Yeah.

There's things so like a TikTok. It like gives you recommendations that if you watch like something it will also give you another thing. Like my friend was telling me that.

Okay.

And he was telling. That was like a really good system.

Yeah. Well, Trump isn't Facebook and all.

Huh?

Yeah. Yeah. Trump is in the hospital. So you can say tick tock but actually you shouldn't be. It's. It's kind of banned in US Although there was a restraining order for a bit. Yes, Shakti.

So sometimes I go on to sites to scroll for music books.

Okay.

Also I click on a. I see the PDF of the book. It also gives me recommended books such as if I choose the Suzuki book one, it shows me the Suzuki book two. Three.

There is one thing that is still missing. I want to hear that. Yes. Veer, Maybe you don't see it as a recommendation. What?

YouTube?

Yeah, YouTube. Yes, we heard that. Yes. Something new.

Like people spread recommendations about like shows or classes or something like that.

Yes. People around you telling you. But there is one more thing which is not exactly Easy, easy, easy, easy, easy. One at a time. Yeah. There is one more thing which doesn't show up as a recommender, but is actually a recommender. When you, when you type messages and all. Yeah. Then it gives you emoticons that you can use those emoticons. Oh yeah, that's interesting. In our phones whenever we type any messages.

When you're searching something up on a Google bar, depending on what you search before options come up.

That is also also gives auto typing and it also typing is also. Yes. That's all machine learning based. Actually there's a decision that it uses.

There's also design ideas.

Yes, yes. Run it in.

In Microsoft ppts when you like for like you just type anything and put a picture. There's also.

It gives you design ideas. That's a recommended system. Very good. Excellent. All of you. See you have. And. And I'm. I was specifically looking for Google because see Google comes as a search engine, right? It you are searching something. But actually we don't know. But Google is also making recommendations to us and thank you Ria. Always to thinking outside the box, kid. Amazing. Okay. So.

So Nick Kill sir.

Huh?

I know another one.

Yes, go ahead.

Like. Well like whenever you're. Yeah. Gram. Grammarly is one.

Grammarly is one. Oh wow. Wow. You guys are pretty fancy here. Okay so now since.

Since.

Okay, okay fine. We. I know we'll have like we can discuss this. This for Two hours and you'll still come up with new ideas. I know all of us has so much of exposure to so many different things, which is pretty awesome. And, and you know, one thing that we need to know is always when you exceed things, you also have to have the ability to process it and ask the question why. Right? Ahan. Yes.

There's also Google Docs so they added this new thing. So when you write something they try to recommend your sentence.

Autocomplete. Autocomplete. Yes, yes, that. That also causes a lot of problems in our phones when we. When you start typing names and it start giving a kind of weird name to people. Especially with the, you know, people who have a very non English name, which is all of us mostly. So we get into a lot of trouble with that. And it doesn't learn. That is also what I'm seeing, that it doesn't train with me. Anyways, all these are very interesting examples, but I want to kind of emphasize as to why. Why do you think you need recommender systems? What is the purpose of a recommender system? Yes, Sakti, you raise your hand first. Augusta, you're next. Yes, Shakti, go ahead.

Are important in our everyday lives is it gives us more exposure to things we may not know more about, much about.

Interesting. Deep. Okay. Okay. So there could be certain things that we are not aware of. And then a recommender system tells us, hey, why don't you try that? Maybe you like it. That's what you're suggesting, right? Great. Good, good. Very good point. Smart point, Shakti. And then Agastya. Agastya. Then we'll go one at a time. Yeah.

So we need a recommendation as we keep seeing. For example, in YouTube we see a video which is related to 5 minutes crafts. So it gives us more videos according to that and it helps us explore new things and it gives us the same recommendation videos of what we like to watch. Like if this video is related about entertainment and horror or comedy, the same.

Content they will try to provide us.

In a different way.

I got it. So basically it kind of classifies you into a box and says, oh, people in this box like this. So why not try this as well? Yeah, good point. Who was next? I'm sorry, I've lost track. Okay. Samax, Ronit, Ria and Alia. Okay. And. And yes. Sorry. Yes. I don't mean to. I. I saw you. You were on the other side of the screen. Okay, go ahead.

Okay, so like a recommender system, it can also make it like you will spend more time on that site. So like on YouTube if you're like, it recommends you more stuff so you spend more time on it and like the youtubers make more money and like YouTube makes more money from that.

Interesting.

Like even Google, like they need like the money stuff and then they also wanted to like help you so, so that you know what to do, what, what's best for you, like what's the machine think it's best for you.

Yeah.

Samarksh has already seen the social dilemma.

Social dilemma is something if you. You have to see all. I'm actually going to recommend it to everyone over here. It's. It's a scary movie.

Normally I, I know a lot of people do this because I do it normally. We just like go on Google and you just type anything and see the.

What.

What Google recommends and you just click on anything so that you could. It does help us find. Learn more about different parts.

Excellent. Okay. Okay, good. Who was next? Yes. Rhea. Go.

It makes your life easier because if you're. I was doing a research project, for example, and let's say my project was on hummingbirds and I searched up how can hummingbirds fly backwards? They're the only birds that can fly backwards. Okay. So then it gives me an answer. Then I search up health food for. And then the option already comes hummingbirds because it knows I'm researching about that. So it makes us easier when you're on YouTube or Amazon, it hooks you on, it gives you more options. And so because it knows what you like and what you don't like, it recommends stuff that you do like and then you just have to watch it and then the next one and then the next one. So like said, it also can be used to like help the actual company that.

Excellent. I get it. I get it and I understand. So it is more of a support system for us because when we are looking for something, we have an assistant which is actually doing the job for us on the side. Is that right? Yes. Okay. Yash, you wanted to add something to this.

Yeah. It can recommend you video games that you like. Like for example, I like to play racing games and I like to play sports games. So it'll show up like racing games and some sports games like FIFA 21 or Forza Motorsports and stuff.

So again it's based on who you are. It's giving you something. Right. Okay.

But it's not necessarily help.

Yeah, yeah. That's what I'm going to actually go to discuss now. Namrata. Actually it's very important. All of you. Alia. You have your hand up. You wanted to add something very quickly. This is last. Okay, go ahead. You're not. Your mic is not working, Alia. Okay, go ahead. It's not working. Okay, so what I want to also mention to all of you, and I'm actually not surprised by how all of you see this, most of you have this whole perception that actually it is supporting us, right? Actually, it is. It is something which is good for, like Ria said, research. Excuse me, who's speaking? Somebody's mic is on. Yes, who's my. Who's my. Okay, so what I was saying is that maybe you perceive this as a support system. As a system that helps you. Right? As a system that, you know, is, you know. In fact, Shakti mentioned that it opens a new perspective to you sometimes, right? Like, you know, something that you might not know, you get to know, right. Sometimes you would have not gone that path just because somebody recommended that to you and you're like, huh, it makes sense, right? Maybe I like it. All of a sudden you start liking it, right? So there could be multitude of things that this system is doing for you. Okay? As long as I know why it is doing for me. Why is this system doing this for me? Okay, hear me out very clearly. It might be slightly complicated. It is fine. But there are many reasons why these systems do these things for you. Samax did kind of scratch the surface by saying money, right? So what happens is in most of these systems that you're talking about, okay, you, this service that of recommendation, like, you know, this service that gives you a recommendation comes for free. Yeah. You don't pay for that recommendation, right. Are you all getting me? You're able to hear me clearly? Yeah. You are not paying for that recommendation. And when you are not paying for the recommendation and that product is giving you a free service, then you are the product. So essentially all these recommended systems, although we are going to study it, we are going to understand that underpinnings. Yes, just let me complete my thought. I'll. I'll come to you. Okay, so we have to. We have to study it. We have to understand what this is, because this is an important thing. However, there is an ethical aspect that you need to be aware of. Because money is not everything, right? If I actually change the thinking process of a person in a direction which is driven by a motive just to kind of benefit me, suppose, you know, as I all, all of you who are listening to me over here, like, you know, you have some trust on me that he's teaching Me. But if I drive your thinking process to a direction where I am going to benefit from it, you understand what I'm saying, then that would be unethical of me. Right? And, and if you are in a Google search and you are saying, who is the best movie star? Suppose, you know, people do all kinds of searches, right? And then Google says, I am going to say the best movie star is Shahrukh Khan. Because Shahrukh Khan invests $500,000 on me every year, okay? Just to be able to bring his name higher on the rankings. So this particular search is not wide, right? This, this particular search is essentially a corrupt search, right? Manipulation. It's manipulation. Absolutely, Amit. So this is very important, okay? This is one thing that I want to tell you. When we're doing data science and we believe in science, we also need to have ethics. Okay? So this is a very important thing. Ethics is another course, a complete course in itself. But I just want to make all of you aware of it and understand that when you get a recommendation, don't get super happy about it. Also question that recommendation. Why am I getting this recommendation? Yeah, what, what is the reason why I'm getting this recommendation? Yes, you had your, your hand up when I was speaking. Yes, tell me.

Like they could collect data from you and like understand it and apply it later for you and for like other people if like they know your age for like Google, they'd ask you for your age. So they'd know like a teenager your age kind of area would be like, I'll search this up at this time or something like that.

Yes.

Yes, they're also collecting data.

Yes, they are collecting data. And collecting data about you might be something that you might not be even aware of. Yes, Rhea.

Our system recommend based on like clusters.

There are multiple. That's exactly what we are going to get into. So, you know, I wanted to make sure that before I teach you something, you are aware of why and also the ethical aspect of it. So let's get into the. Actually let's take a jump into the technology. So now I am not going to be talking much about the ethical aspect. Although, you know, I just want to emphasize that whenever you are going to look at an ethical system or ethical system design, always think from the ethical aspect. Because if you're designing something which is unethical in, in its whole systemic design, then, you know, what's the point? Suppose I say, you know what, I put a camera outside my house and now I can just message everyone and tell them, you know, What I saw you were walking at 3am in the morning. Like, you know, that's none of your business. I walk at 3am or 6am or 6pm or 7pm you know, just because I have a camera, it gives me a trigger. This person is walking.

Right.

I mean this is just a simple intrusion. But you know, I, I don't want to get into very personal things, but people do all kinds of things which are not fair. Okay, that was a slide that came by mistake. Okay, so what you can see over here is that when we look at a recommender system there are, and this is what we are going to study today, we are going to look at different types of recommender systems. And I want to equate, equip all of you with a strategy as to what recommender system you are going to design in which scenario. Okay. When it comes to suppose you are launching a new product, what is the kind of recommended system you would want to have in your product? Okay. When you are designing something for a web page, what would be the kind of recommender system? If you're selling a product, what would be the type of recommender system? If you are, you know, just buying things, what would be the kind of recommender system? So there are different types of recommended systems and they are designed for the same purpose, to give a recommendation, but they have their own salient features and those salient features are what makes them like very interesting. Okay, so I'll start with the first recommender system which is called a content based recommender system. Okay. As the name says. Okay. As the name says. And this is what most of us have seen what happens here is, you know, you, you consume one content. Like Agaste was mentioned about the content that you know was he kind of goes to a site or any place and all of you, actually even.

A.

Few others mentioned that you go to this site and you, you look at a maths video and then it starts telling you, oh, there is a similar maths video over here. Why don't you also look at it? Yeah, yeah. You must have seen all of this. Like it, it doesn't tell you that other people, it's not talking about people here, it's saying oh you, you watch this video. There is this video also which is close to it. Yeah. What do you think is helping us identify which video is close to it? I have taught you. It was two, two weeks back. So I just hope somebody remembers. Yes, yes.

Clustering.

Huh?

Clustering. Like the clustering.

Okay.

And like, and like also like the Content. So, like, if the content is similar, it can cluster it in one group.

Excellent. Okay, so classification. There could be a classification also Amit. But classification would require. It is possible that there is classification. Clustering. Clustering and classification have a very thin line of difference. Right. So generally we do classification. Does clustering precedes classification process. So you build a cluster and then you build those classification models. Right. And you say, this is my class A, this is my class B. This is my class B, C. But that has actually sometimes does not necessarily has to be like a genre, like an action movie or a horror movie. But that would be a multitude of different features that you would use to build those clusters. Yeah. So where you do clustering and then you do classification.

Yeah.

So it would be a cluster like people who do data science on Saturday morning and are excited about it. Yeah. Or it would be another group, people who do data science in the morning, but they're like, oh, yeah, okay, whatever. And then there is. So you could have all kinds of complex clusters. Yeah. And then. Okay, people who do data science in the morning and say, whatever. This is what I want to recommend to them that data science is fun. But people who are excited about data science on a Saturday morning, I would, in fact, tell them, you know what? Good job, continue doing this. You understand what I mean? What I'm trying to convey to you is that these examples that we have. Right. Is based on this whole perspective that, oh, you are always keep wearing a blue shirt. So which means blue is. I. I guessed that blue is your favorite color. Yeah. And because of that, I will always recommend something in a shade of blue to you. Yeah. I just need some acknowledgments to say whether you're getting it, not getting it. Yeah. Okay, good. Thank you. Thank you. Okay. Yeah, it's important because in a classroom setup, it's much more easier. Virtual setup. I'm. I'm looking on the other side for you guys, just for the. The act. Yeah. So now when somebody is just looking at my behavior patterns and are putting all the things that I'm doing into one box and is trying to make a new recommendation based on which box I am situated in, what do you think could be the benefits or what you think could be the negatives here? Both. I want to hear benefits first.

Any.

Any thoughts on benefits? Yes. Go. You have to turn on your mic before we. So that we can hear you. Yes.

I think that one of the benefits of what you just said is that maybe, like, it'll help you find something Else which you like in a topic that you like.

Yeah, well, that is the whole purpose of recommendation system.

So it might be, it might also be some accessory and not something similar. For example, if I'm buying a DSLR and I'm recommended this particular lens, which would be good for wildlife, which I might not have thought of while buying. So like in that way it's like another product.

Yeah. Usually that, that complementarity that you're talking about, Namluta is slowly, we are slowly adding that feature to content base. But in philosophy, a content base is actually not exactly what you're saying. What you're saying is that, you know, if you buy this, then maybe you should try these things because that might work for you. It that is the negative aspect in content based Samax. You wanted to add a thought before. I just, I just don't want to keep going around in circles.

Yes.

No. Okay, so what happens in content base is very interesting. Content base. It actually identifies who you are. Okay. Identifies the kind of content that you are consuming. And he keeps sending you the same content. Yeah, I mean, same content as in. Same family of content. Like, you know, it is sometimes very irritating. Have you all seen something like subscribing to something, a website and then you keep getting notifications in your phone? Yeah. It becomes very irritating after a period of time. Right. You just look at some news and you subscribe to it and it keeps sending you the similar news on a daily basis. Right. It's as if, suppose I say like I was reading one day, how can blockchain change society? And I've got like 50 articles of how blockchain can change society. Why can't you talk about, you know, what are the problems in society that blockchain solves? But it says how blockchain can change society. Like, you know, 20 different people, probably two of them are plagiarized anyways, but they send me that same information. Okay? So what's happening is it's not expanding the horizons for me. It's saying, you live in this bucket, you know, and you, you, you, you kind of operate only in this bucket and I will send you things that come new, which is of this bucket family. Right. So it is exactly the opposite of what Shakti said when, you know, talking about expanding the horizons. Yes, Shakti, you wanted to add something here.

So not all sites follow the same concept as you had said mentioned before. Some concepts, some of the concepts, especially in YouTube, they say a person that subscribed to this channel watches this channel. So. And sometimes they're just. They're not even related.

Yeah, yeah, yeah. So, so. But YouTube does not use content based. Just to Let you know, YouTube uses a much more robust, complex system. We'll get to that. Okay, so let's get to the content base. Okay, so when what happens, it's also called content filtering, by the way. Okay. So what happens is it is essentially looking at your user profile and contextual parameters, what are the different things that you like, and then looks at the product features, and then based on that, it gives you a recommendation. Yeah, it gives you, oh, this is who you are. This is what you like. And you'll be like, oh, yes, I like it. Now, this can work in scenarios. This can work very well in scenarios like going to a hotel. Yeah. So when I used to be a consultant on the road, I had certain preferences. Okay. I, I had a preference that I used. I want a hotel room on the top floor where there is not too much of noise, far from the elevator. I wanted, you know, I did not want a soft pillow. I wanted a hard. Whatever, all kinds of stuff.

I.

A hard bed because I had a back problem. And, and, you know, I mean, preferably the corner room so that I would not be disturbed by people who would just come there for partying. Okay. And what they would do all the time is they had my content, and every time they would match and say, Mr. Verma, would you like to go to the fifth floor or the sixth floor? That would be the recommendation because they already kind of sorted out who I was. We have two rooms that you. We that would suit you, and we have the fifth and the sixth floor for you. Okay. Now this is a very simple process because all we have to do over here is to be able to identify the product attributes. Yeah. When we talk about a content, content is any kind of information that can be identified from a set of keywords. Okay. If that's called a structured content. Because when you have those keywords or description, which is, you know, which repeats from one place to the other. So, you know, like, mostly in Netflix, we say the, the, the. The. The genre is of the movie is horror, comedy, family, kids, so on and so forth. Right. But if you say, oh, this is. Genre is Edison resident movies or, you know, residents in. In Singapore or Dubai, people would love this movie. Yeah, that might not be a very structured way of putting it. It's an unstructured, unscientific way of classifying things. Yes, Shakti.

They. I know something similar to this happens in Prime Video where they just Start grouping it based on language or based on where you live.

Yes, yes, it does. I mean, it does. Where you live is, could be interest, could be not a very scientific way of doing it. But language is definitely a scientific structure, right? Because based on the kind of languages that you speak, you know, that's the kind of movie it can tell you. Or maybe if you don't speak that language, it can tell you there are subtitles in the language that you speak. And it's a different language movie. So it still kind of falls into the category of something that you could watch. Right. And I'll just show you how this is done. This is very interesting, of course, just to be able to look at the mathematical modeling of this is outside the scope of the course and would be very, very, very complicated because they use mathematical functions like cosine differences and other stuff to be able to do it. Now for, specifically for this, I have a notebook that I would be showing you in a minute which does something to be able to do the content filtering. So suppose I say I like shirts of this type. It would run through all the shirts in the database and tell me, Nikhil, maybe you would like shirt 5, 13 and 27 because you like shirt whatever. 75. Okay, so that is what content filtering is all about. Over here we are just working with the attributes of the, of the content. Okay, Now I'll just show you how this works. Yeah. So what we are going to be looking for over here is you just, you know, whatever is the user profile. This is the user profile. Okay. This is this person who likes these different things. Detective, Murder and New York. Okay. What they are going to do is they are going to use the profile of this person. Hear me out, okay? The, the keyword profile of this person with the keywords of the items and then see what would be the good ratio. Okay. If you just kind of don't get into the nitty gritty details of this. All they are trying to see is how many of my keywords are matched by, by, by the items. And whichever item matches the most with me, I will make a recommendation. Yeah. So whichever item matches the most with me, I'll make a recommendation to you. So it's, it's, it's, it's a, it's a very simple mathematical model. Okay? Now tomorrow, if there is something which comes closer, it'll tell you, you know what, I recommended something which was about 70% similar to you. Now I have something 90 similar. And he'll be like, okay, let me know. Right? However you have to understand the scope over here is not expanding the horizons. It is important that I emphasize this and it's important that you get this, okay? The, the scope that we have here is not to expand the horizons. Not to tell you like Amit, you know what? Maybe you would like this because you've been liking things like this. Maybe you should try this out. It is never going to tell that to you. It is always going to tell you that. Mahak, you know what, you've been always using this particular product and I have a new product which is very similar to that. Right. So the difference has to be something that you understand. The difference is that it is only going to think within the categories of who you are and how it has classified you. Am I making sense? Yes.

It basically does not go out of the box.

It doesn't go outside the box. Absolutely. Okay, so the. That the. There is a big limitation to this. Right. The big limitation is that you would always kind of suppose I have classified somebody into a category. You would always make that person live in that category. Yeah. And you would always be able to market only things. Market or position things only to only in that particular category. Okay. And that can be. Not a very optimal way of doing things. Right. So I am going to show you a simple example which I have for content filtering. Can you all see my screen?

Yes. It shows like the Jupiter thing.

Yeah, it does. It shows the Jupiter hub. Yes, our good old Jupiter hub. Hold on. I just want to show you this data set. So you can actually appreciate and see how this works. Yeah, so this is my.

Oops.

Come on. What is happening with my. Yeah, this is my data set. Okay. Can you see the data set?

Yes. Yeah.

So over here I have some id. Okay. I have description of the product. Okay. I have some English words that define the product. Like a sentences like the reason why or whatever. Okay? So every single product that you see over here, I have about how many 500 products here, okay? So I have 500 products here. And there is a description of each of these products. Okay? Now if I say that cap one graphic T shirt is what Samax likes, okay. And bucket hat is what Ashwick likes. Okay. And compound cargo pants is what Agastya likes. Okay? So Agastya tells me I like this.

Okay?

I have a list of 500 things. So now when I get Agastya's information or Ashwick's information or some of his information, what is the recommendation that I can make to some of? Because I have never met Ashwick before. I Just met Ashwig right now, or I met Agastya right now. And I met Samax right now. And they just told me, I like this. And then he's like, okay, you know What? I have 500 things. What are the other things that I can show them? Yes, Ashwick, you want to say something? Okay, yeah, Shakti, go ahead. You had your hand up. Okay. It was just a.

Okay, so if you know the exam, the definition for most of the products, you can start comparing the definitions of the products and then try recommending them to the person.

Yes, exactly. In simple terms, you exactly define the code. This particular code that I have here. Okay, so what we have over here is. This is, again, as I mentioned to you, how this works can be slightly complicated. And that is why I will be giving you these software code for you to be able to have a look yourself. I mean, not yourself for your project. You will be able to use them. Okay. Of course, it is kind of outside the scope of this course to be. For you to be able to write these codes, because that would require a course in itself. But when I show you these. Okay, what's going to happen is I. I am going to start making recommendations for each one of you. So for graphic T shirt, right? Who like the graphic T shirt? Okay, let's say Merino Crew T shirt. So Merino Crew t shirt is 87. Yeah. So I'm going to put the ID number 87. How many items do you want me to recommend to you from my list?

Five.

Okay, so I have five items over here. So this is my function which calls it. Yes, run it. Just give me one second. Okay. And what this going to do is this is going to look at that data set, pull up all the information, and then make these recommendations that I'm looking for. Now I have the Merino Crew T shirt. And because of the Merino Cru T shirt, these are the different T shirts that you would probably like because you see there is a score. This T shirt is closest. These are closest to the Merino Crew t shirt. By 0.649 is the score of the Merino T shirt, which is exactly close to the Crew T shirt. There is also another Merino Crew T shirt which also has similar functionality, similar features. Graphic T shirt, which is similar. Merino 1T shirt and Tank T shirt. Okay, so these are closest to the Merino Crew T shirt. And that is why it is being recommended to us.

Yes, Yash, can I also try this?

Yes, absolutely. Just give me one second. Okay. I just want to Explain a few things to all of you so this can be an interesting thing for all of you to look at. Maybe this data set is just populated hypothetically because it doesn't have. Or maybe we pulled a few things which are real. Okay, but suppose you are able to create a data set like this. Suppose I'm able to create a data set like these. And what I am going to tell you is I am going to ask you what you like and based on your keywords, I am going to match with other products and make recommendations to you. What do you think about this? What do you think? Is it of any value to you? Is it something that would make you feel like, yeah, okay, whatever, run it. And then. Yes.

No, I didn't have a question about this.

Okay, go ahead.

So sometimes I've noticed, like when I go on Google and I search something, it. It comes. And then I, I shut down Google and after something, after a few minutes, like when I Google the same thing again, it's either. Either it says you can't, it can't find it or, or the tap shuts down on my whole laptop. Is there something wrong with the recommendation system or is it my laptop?

No, it would not be a problem with the recommendation system because that would be pretty bizarre. It's something wrong with your settings probably. But. Yeah, yeah. Yes, yes, yes.

I was just.

Okay, have a question ready because we have like 20 people waiting here. Okay, go ahead. Anybody else? Any, Any, any thoughts on this? What do you think? With this kind of a recommendation? I put this Melanoku Crew T shirt and I got like five. How many? Five. Five recommendations for Samax. Yeah. Yes. You missed your chance. You'll have to get it. You're now lower in the pool now. Okay, go. Yes. So much.

Well, so like, is this basically what we are going to do for homework?

Don't worry about the homework. I'm asking about what you think about this.

Well, like, so you basically give us a code and then we put the ID in and then we put how many recommendations and based on that, whatever like the keywords are, we get the, we get like the similar things for that.

Yes. What do you think? Whether this is of any value to you? That's my question.

It is.

It is. Okay.

Yes.

Good. Okay. Anybody else? Any val, Any other thing than more than yes or no? Yes. Shakti, go.

No way. It does have some value, but since it's not the. It's not what you wanted and it's not what you expected and it's may. It may not be what you're going to purchase. So in a way it doesn't have value.

Okay, so can you elaborate why you think that is the case? I know you have a. A stream of thought that actually made you say that. I want to hear that piece. Why do you think it does not have value?

Because it's not exactly what you expected and it may not be what you're going to it.

Okay, fine. Anybody else wants to add to this? Before I take a jab at this, Amit wants to say something. Yes, Amit, I think you might be recommended something but you actually don't have a need for it. You would only buy. You're out there to buy one shirt. That's a very boy talk. Yes. If I go to a store I want that they recommend they are cute with me, I get mad. I agree with you. But you know, let's kind of see it from a different perspective. What I am saying is I pick up something and I say I like this. Then this person tells me, oh, you like this. You know what? I know five more things that you would also like. Right. So it kind of expands the horizons from the perspective of giving you more options because they're like 500 things which are all stacked up there. You get my point. So it. Yes, yes. Come on, quickly.

You might just like something which is not good for you.

You might just like something and then.

You have like five more things. For example, whenever I upload a Facebook story and then I go off Facebook for another two weeks, I continuously keep on getting recommendations that you should upload a story. It's been a long time.

That's true. Yes, yes. Come on, quickly. You want to say something?

Yes. So what I wanted to say is that it does have some value because like for example, you don't like. Well, you like the one that you like said that. Oh, I like this. And then it recommends five other options and you liked one of those options more than the other one. Then it can help. It might even make you have a better choice or decision.

It could have a better choice or decision. Fine. So essentially what happens in this kind of thing. See, this is where, you know, I mentioned to you data science and this is where what I want you all to be as different from people who are just sitting there and programming is to be able to understand why I need it. Always remember the seven chakras. Okay? If you don't know why I am building this recommender system, you would always say I do this. This is closed function. This is this. This is that. Right. I have met many people who do this and I ask them, what is this score all about? Oh, I use the score with all the keywords that are matching. Oh, good for you. But if, what if I write multiple keywords in my product or your product will always be recommended, then so I know I can easily break the system. You understand what I'm saying or you're not understanding? Are you getting it? Suppose I say I go and do a search in all these 500 products and I find all the keywords that are appearing the most times and I write a 501st product. Okay. 501st product, which has keywords that are from most of the products. So like maybe 200 products have the same keyword and then other 200, I'll add another keyword. Right. And then when you do a search, my site will. My, My. My product will always appear. Right. This is how people trick search engines. This is how people trick recommender systems. I'm teaching you negative things. Don't go and do this. Okay, but this is how you can actually fool the system. Right. You know, in the past, this is, this is, was a very common thing. People used to make their, their cvs, their resumes. Yeah. And when they used to make their CVs at the end of their name, they used to put like 50 keywords. Okay. What used to happen is a person would just do a search with keywords and their CV would always appear, so they would always be called for a job interview or whatever. Right. So there are ways to trick the system. Because, you know, inherently, if we don't use our brains when we are designing, these systems will end up doing something really stupid. Yeah. Now over here, the idea was to show you how this works. I am not even trying to suggest that this is relevant, but I am also trying to emphasize that this could be relevant. Right? This could be relevant. As some of you pointed out that, yes, I might be going to buy something. Like, yes, mentioned that I might be going to buy something. And I get three more things and I get excited and say, oh, you got variety, variety of stuff, which I like. Right. But at the same time, there could be things like what Amit said that I want this. Why the hell are you giving me five more things? You're just wasting my time. Right? So it is important that these recommender systems are tied to some kind of a strategy, some kind of objective. Yeah. Again, always remember this. Never do things without an objective. Your parents tell you go and study this. Then you tell them, if I study these, what Am I going to get out of this? Oh, you are going to be a better person. Fine. That's it falls in the big picture of things. Go and do karate class for one month. What will I do with karate? Do you want me to be a fighter in the street? No, it's for your self defense. Oh, I'll do it. Yeah, that makes sense. Yeah. So you have to. You have to question things from within before you do anything. Yes, yes, go ahead. So I'm going to be putting this up for all of you to play with. Okay. And I will be defining something very interesting. I am going to ask you to be a hacker this time. Sometimes being a hacker is a lot of fun. I want you to make a 501 product over here. A hypothetical product with your name where you put in all the keywords that you think would make it relevant. And we randomly 501 and 502. Maybe two products. Yeah. And suppose we randomly pick any product. I want your product to also appear along with the recommendations. You get my point? So I want you to break this system and show the system that you're smarter than the system.

So like how, where do we create it?

In this data file, that Excel file that this system is calling you write two more lines. Okay. Samaksha's crazy product one, Crazy product two. And you say some actually is so smart that he has broken this system which has boundless. And you write something which is gibberish, doesn't mean any English, but you're just putting in the keywords there. Yeah. And then when we do this search, product one or some product two should appear. Good chance. Not always, but good chance. Suppose we try this for five random numbers. If you are able to do this, then you are a real hacker for me. You get my point. So this time it is a hacking project. Trial and error. Trial and error. Trial and error. Okay. Okay. I am just going to quickly present.

Yeah, I don't want to be a hack hacker.

You don't have to be a hacker. You, you're going to be a data scientist. The data scientist has to be a hacker too. Yes. Okay.

Okay.

So now I am the. We are 1103. And last time also I went way over the time. So I don't want to tire all of you. And it's IPL time. I know some of you are already itching in their seats to go and see whatever game is on today. So I will just quickly introduce the next thing which is called the collaborative filtering. Collaborative filtering is the next group I'LL just show you here one more time. This is what collaborative filtering is. We looked at the content base. Now we are going to look at the collaborative filtering model. Okay. Excuse me. So collaborative filtering is where we use human judgment. You use ratings, you use lots of other things. Okay. To be able to provide some kind of a recommendation. Okay. What over here is very interesting in collaborative filtering. Okay, suppose we have three people. Joe, John and Al.

Okay.

And Joe rates Star wars with this much Hoop Dreams and Contact. These are different things that they have. He's washed. Yep. Now we will be able to predict what Nathan is going to rate Titanic has based on his ratings of Star Wars, Hoop Dreams and Contact. Yeah. So this is what collaborative filtering is. Okay. And it helps us to be able to make different kinds of advisement based on information that is coming from other people. Yes.

So as we said before, for the shirt thing. For the shirt we need to look at the keywords. I haven't watched any of these movies, so I don't know what they're about. But if we know the keyword of each movie, so if it's similar to Star wars in a way, we could predict.

So the shirt example, Shakti, was where we just took the attributes of the different things that we have and we compare things. Right over here we have other people also involved. See, in that case, we didn't have other people involved. It was like, you know, you have a shirt which is a blue shirt, you have a cargo pant, you have a merino T shirt, you have a polo shirt. Yeah. And based on the keywords, you try to see which one is closer to which one. Yeah. But over here we have like four, three people before us. Yeah. And what we are seeing is these three people before us rated these things likewise. And when I, Nathan watched, he rated them with these scores. So what is the chance that Nathan is going to rate Titanic as 4 or 3? So it appears that Nathan is closest to Joe if you look at this data. Right. So Joe, watch Star Wars, Hoop Dreams, Contact and Titanic. Yeah. I'm sorry, the spelling of type. Okay. Yeah, it's good. Why does it appear that ways in my screen? Okay, so the chances of Nathan rating Titanic as 4 is quite high based on if you look at how Joe has rated Titanic and Nathan is closest to Joe. You get me. In the other case, we did not have a Nathan and a Joe. We just had Samax. Yeah. Or we had. Yes. Or we had Ashwick or we had Agastya. We just had one person. Yeah. But over here we have multiple people based on whom? Whose data you can baseline things. You get me. And that is why it's called collaborative. Okay. We are basically collaborating with other people. Yeah. So what we could also do is we could use this to help find new items. This is going to expand new horizons for us. See? Interesting. Yeah. So I have a friend and you know, Mahek is going to smile. My friend Manish Arora, he's a very good friend of mine and he travels around the globe. Yep. And every time he tastes or does something new he tells me. Okay, I'm joking. Not after calling his name because I'm missing him. But it's just that I have a throat issue. Excuse me. So when he travels around the globe he experiences things and he knows that I also have a similar taste as him because we've known each other for around 25 years now. Even more. Okay. So he's like, there's a good chance that this guy is going to like it. Right. So when he sees a movie he says, you like it? I'm sorry to the guys who don't understand Hindi. I was just, you know, just to kind of give you the right feel. But what I'm trying to make you understand is these recommendations that come from people is, is collaborative. Imagine we build a system. We build a system. We build a machine learning, a data science backed system which is going to do exactly that. So over there there is no Manish and Nikhil. What's going to happen is it's going to identify a family, a group of. Yeah. Who's that? There's going to be a group of people. You get me? And that group of people is where they would say, ah, these people seem to be of similar interest. Yeah. Because they have similar income, they have similar taste, they have, you know, same family size, they live in the same part of United States or you know, wherever in the world. Yeah. And they seem to have the similar interests. So if one person does something new, it'll tell me, oh, this person does something new. Maybe this is something that we can recommend to the other people. You all get me. And in this kind of a system what's going to happen is you are going to, you are going to get new things recommended because it is magically pulled out from a group of people. Right. In content you are never going to get new stuff unless and until you trick the system. Essentially. Yeah. But over here you're not tricking the system. It's like, you know, one person is doing something new. Oh, why not? This, these groups Might definitely like this. A new Amazon, a new Netflix movie is released and people in my community or people who've seen similar things like I have, has go and see this movie. It tells me, nikhil, you should see this movie. Maybe I don't belong to that genre. That's quite possible, but it can still make a recommendation to me. Are you all getting it? Yeah. So now you see there is, there's a big difference. Yes.

Squad also. So like Netflix, YouTube and these kind of sites they use like collaborative and not content. Right?

We will get to it. We will get to it. We'll get to it. We'll get to it. Okay, so Amazon is mostly not content based. Okay. Amazon is mostly collaborative, but there is content also. And we will talk about it in a minute. It's called a hybrid system. Yeah. So if you look at this, this is how usually you see like, you know, what are the recommendations for this person over here? And this recommendation is drawn from a multitude of things. What this person has searched, which category this person has been put in. Okay. So just to quickly go over the differences. Yeah. When we look at collaborative filtering, it is users interest. Content is item info. Over here, the similarities in the item info or the text document. Excuse me. Oh, okay. So imagine if I was doing it in a class now, you would have all ran out of whatever doors which were there. Right. So over here you're not scared. Doesn't get spread by microphone. Right. Okay. So content base, you have the item info and you have a text document. And that is why it is very easy to break this system. In collaborative you have users.

Okay.

And you have user profile. Right. Okay. Sometimes what happens is if you get into a user group where people are not kind of, you know, as, as, as, you know, people who try out new things, like my friend Manish or anybody else who's, you know, if your friends are like, oh, okay, you know what? I'll go to work at 9 o', clock, I come home at 5, I watch TV, I eat dinner, I sleep, I wake up in the morning, I go to work, I come home at nine, go to work at nine o', clock, come home at five o', clock, I watch tv, sleep. If somebody does that and you get into that category, good luck. Right. You're not going to get any information. Right. So you have to be in some kind of a feedback model from other people also. Yeah. And then if all of a sudden I have a friend like Amit. Amit, don't mind it. Okay. I have a friend like Amit who just goes out and tastes alligator meat and says, you know what, this is really awesome. And then I get recommended alligator meat. I'll probably not be able to sleep for two days, you understand? So it's important that this system also has some kind of relevance. We have to control. Right? Yes. Beta, you want to say something? Oh, he's just. Okay. He's practicing his bowling. Okay. So you, you what I was explaining to you so far, I just want to be you to understand. And this is the basic science. I'm not taking anything off from what I would teach this in a master's course except for the algorithms, because that would be too complicated. But collaborative filtering and content filtering is, are two different things that we use. Okay. And they have their own pluses and minuses. When we look at collaborative filtering. Okay. You know, some of them, you know, user based nearest neighbor algorithm. Yeah. And there is some kind of a probabilistic modeling also we do some kind of a probabilistic modeling. I have done this, my PhD dissertation was in this area. So I did this probabilistic algorithm for collaborative filtering, but with, with another approach which is called hybrid, which will show you in a minute. Okay. So how do we measure these things? How do we measure whether our recommended system really makes sense, Whether it is doing good? Because if you are not able to measure things, you cannot manage it. Right? So suppose you say, okay, this car is very good. Now the next question would be define good to me. Oh, this car is very good. Because the annual maintenance cost is this much or the speed of the car goes from zero to 60 is in five seconds. And it's always like that. So in this kind of a system, in a recommended system, we look at something which is called a prediction accuracy. Okay. Then we look at, you know, precision, half life, utility. What are the. When the system is running, what are the different things that you get? So these are word. These are terms that we use to be able to measure the type of prediction. So whenever you're defining a recommender system, these are the things that you have to address. The other thing that you need to look at is what is the novelty? Like this is what Shakti mentioned. There are some things that I might not be aware of completely. Then all of a sudden it opens the door for me. There are some things that I would have never discovered if it was not for the recommender systems. Okay. And then what is the coverage? What are the things that it can show me? What are the things that it will be able to tell me? Okay. There are things which is also called the learning rate. How fast is the system learning from people? Yeah, we. This is what we define in artificial neural network. We define how fast this network learns from one another. Okay. What is the. How does it define the quality? What is the user satisfaction measured? This is what, this is one thing that I used to measure with my. In my PhD dissertation. When I designed a system, I would always say my measure is user satisfaction. If the user is happy, my recommender makes sense. If the user is not happy, too bad. Okay. This is something that I started the class today. Okay. Which is privacy. Okay. It's very important that all this information is personalized. But you. You cannot break the trust of a person. And that is where you know the, the. There is a very interesting Netflix. I think it's an episode, right? It's. It's. It's a. It's a. It's a movie, right? It's a Netflix original movie which is called the Social Dilemma. I recommend all of you to look at them watch that movie. All of you. Social Dilemma Agaste. Good job. If you watched it, you all have to watch that movie. Social Dilemma, please. Because it is important for your education, for your awareness and for your protection, okay. So that you don't get so much into your phones and data that you consume digitally that you stop living the real life. You have a person standing next to you. You don't talk to that person. You're working on your phone, phone. That is the dumbest thing that you could do to yourself. That is the. That is the worst thing that you could do for a human to human relationship. It is always nice to talk to a person then message a person, isn't it? Right. I have a few friends who've already telling me, oh, talking is so boring. You just message me. I said, you know, you are a boring person. I don't want to be friends with you anymore. Right? It is always nice. Okay, fine. It is not a good idea to always kind of disturb a person. But you know the value of talking and talking in. If it is possible in person. Of course in Corona times is difficult. Is far better than messaging. Just to let you know messaging. 70% of emails are misunderstood because you don't understand whether the person is angry happy that lol. You cannot write in an email.

Right? Right.

So they are misunderstood. Okay. So you have to be very careful and it's always nice to just say excuse me, can I just talk to you instead of sending a message? Always be. You know, that might sound like old school. But that is actually very scientifically valid. Okay, this is just a few additional issues. How do we kind of setup. A recommended system has to have transparency. It has to be able to tell you how I recommended this. I use this algorithm to recommend it to recommend this to you. Okay. The other system, which is called a hybrid system. And with this I will finish today's class. Just give me two more slides. Okay. Actually, not that much. It will finish very fast because the example that I am going to show you after this is going to be the start of the next class and then we will get into the problem project. Okay, So a hybrid approach is what most of the things we have in the current times, we don't have a pure content filtering or collaborative system. Okay. We have systems which have both the things in place. Okay. And that is called a hybrid approach. And in a hybrid approach we have something like the FAB system in Stanford University Digital Library. What it has it. It has the user profile also and it does some things with the content. So that ways it is able to use the power of both of these items to be able to make you a better recommendation. Okay, with this I will stop for today's class because the next example that I have for all of you is going to be very, very interesting. It is in the world of Harry Potter and we will be using hybrid approach to be able to make a determination as to something that is done in the movie Harry Potter. We will start next class with that. But I just want all of you, I will be posting the example that I just showed you today. You have the Excel file and you have the software code. So the Excel file is right here. I'm just going to show you, show it to you one more time. It's called sample data. Okay. In sample data, I want you to add 501 and 502 as two new products with your name. And then go to this code that I showed you and try to recommend an item and say, I make 10 recommendations and one of the recommendations should be your product. You get me? So this is basically what I'm asking you to do, is to hack into this system and forcibly put your product there. Just to make you understand how, what are. What is the under undercomings of a system like this? Yeah. Does that make sense? And now I'm going to stop recording.

And does the description have to make sense or can it just be like all the words that you want?

It need not make sense. It could be completely. Just give me one sec. Okay?