Enabling Future Ready Youth and Communities Everywhere - A Conversation with Karishma Galani

Welcome to the Future Learning Design Podcast.

If you went to an Ivy League school, you were somehow vetted and had a very clear signaler to the world. But that is changing and it's being turned on top of its head today. And so in a world that we are now entering, how do we develop verification rather than content creation? And I think that that is what ATM kind of lends itself to.

Hi everyone, welcome back to the second part in this mini-series looking at the ways in which artificial intelligence is impacting the lives of young people and teachers around the world through the lens of the recently announced partnerships with Anthropic. In this conversation with Karishma Galani, I explored the way that Pratham, one of the largest NGOs in India, is integrating AI capabilities and platforms into its work, reaching millions of underserved young people with quality education. They announced the partnership with Anthropic just a month ago, but that is growing from much deeper roots in the vision of Pratham's co-founder, Madhav Chavan, that you'll hear Karishma talk about. Karishma is the co-lead of Pradigi Innovation Center. She leads a lot of the digital innovation and AI work at Pratham, building on her long career in tech startups, research and development, and venture capital. Karishma founded a deep tech company developing educational assessment powered by machine learning out of Singapore and London. And she has been an active researcher at the MIT Media Lab. Karishma is also the author of two books, Maker Minds and Making a Shift: Social Entrepreneurship in Schools. Perfect. Thank you so much. So perhaps for listeners, the first thing that would be really useful is just to orient people around the broader work of Pratham, because as I've understood, it's just doing some incredible work across a huge area with a lot of people across India and then beyond India as well, I think. So yeah, for people just to understand the Pratham piece, and then we can get into the more of the Anthropic partnership later.

Sounds great. So a little context, Pratham is a nonprofit based out of India. It currently today scales up to about 60 million kids and then across like 20 different countries. So Pratham has really, really grown over the last 30 30 years. We just celebrated our 30-year anniversary.

Oh, amazing.

Yeah. But Pratham is actually known for, it started as a literacy numeracy, like a focus on foundational literacy and numeracy. And they're most well known for the program called Teaching at the Right Level, which is essentially a 30 to 40 day intervention program. It's like a bootcamp offline, fully offline, that basically focuses on literacy and numeracy. You look at about grades 2 to 5. And Pratham also runs something called the Annual State of Education, report that comes out for specific to India. And in it was stated that children at grade 5 level were not even able to read a grade 2 level text. And so that is largely what inspired Teaching at the Right Level. That's also largely what drives us to kind of focus on this concept. And obviously it's now been many, many decades at this point. Teaching at the Right Level specifically has been evaluated through 6 RCTs, which are randomized controlled trials by J-Kal, which is out of MIT, and consistently there have been an increase in learning outcomes post that 30-day intervention of about 40%. So any child that comes into the program is, yeah, 40% better off after.

That's significant.

It's massive. And so that's teaching at the right level. And Pratham is most well known for that. Obviously, over the last 30 years, Pratham has grown across schools, life, and work. So like focusing not just on school students, but also looking at, let's say, youth and focusing on the upskilling aspect. So there's a massive YouthNet program and also also working with early child caregivers, so working closely with mothers and mothers' groups and kind of working through that process. It's co-founded by two folks, right? But we work closely with Madhav Chauhan, and Madhav Chauhan stepped down in 2015 as CEO and focused largely on technology. And I think his big question was trying to figure out how can we use technology to actually integrate and enable within all programs. And that has been, that has largely been the work that we now today see. So it comes from a deep-rooted question and a deep foundation focused on users on the ground.

Yeah.

And how technology can help that user base because it's different. It's very different. Building technology for someone like me is very different from building technology from a student that is in rural India, for example, or a student that is based in rural Africa and How are those products different? And I think that's the large question that we've kind of set out.

Amazing. If I could ask you just specifically, I noticed Pratham Open School and then some of the work I think you're instrumental in is the Pradigi Innovation Center work. So yeah, could you perhaps just give a little bit of a description of those?

Absolutely. So when Madhav started this Pratham Digital, largely that was the component, but Pratham Digital today has been morphed into what we call today Pradigi, which is an innovation center. And this focuses on technology and its use cases across programs. And so if you imagine it as a, you've got Pratham Education Foundation sitting as a hierarchy that is very specifically delivering programs that are showcasing learning outcome improvements. Then you've got Prodigy as an entity that's feeding into that, only bases the technologies that actually have been piloted, have been tested, and then are being sent to actually deliver at scale. And so that's how Prodigy, the entity, works. Open Open School was actually started within Pradigi. It was a massive initiative to be able to take everything digital. Like the first layer of building anything technology needs to be the content needs to shift into digital content. And so that is what Open School is.

Oh, okay. Sounds absolutely fantastic. And then, and then obviously that kind of opens up the space then for us to talk together about Anthropic. And then, cuz firstly, perhaps thinking about, obviously, you know, there's been a big announcement for this partnership between Pratham and Anthropic, but clearly that hasn't come come from nowhere. That will have come from a significant amount of co-development together. So perhaps, yeah, could you just speak to how are you seeing that partnership develop? Where is it coming from and what kinds of work are you trying to do together?

Absolutely. I think it even comes before the partnership with Anthropic because we've probably been working with them for about a year now, slightly less than a year, but largely ATM or Anytime Testing Machine came from a vision of Madhav Chauhan from about 12 years ago. It's a very longstanding vision. It was very revolutionary, I think, as a thought process for the world, I think, at the time. And, and it's very interesting to kind of see and work with someone that is so forward thinking because to read a paper that he published about ATM, even though it was an internal document, to read that from 12 years ago and then today to be working on it and leading it.

Interesting. Wow.

It's crazy. And I think his idea, obviously ATM as we see it today in the world is a very small percentage of that large vision, but really it was this learning engine and this verification engine that needs to be developed given how the pace of the world is changing and how we've kind of approached education. Can we flip the system and really focus on the student and put the student at the center of the learning engine, essentially? And then today, if you bring in the, I wouldn't say unfortunate, but unfortunate aspect of where the world is going and how upskilling is going to become critical. And in that upskilling process, verification verification is going to be critical. In the past, verification has always been done through universities, right? If you went to an Ivy League school, you were somehow vetted and had a very clear signaler to the world of who you were and what kind of quality you bring to the table. But that is changing and it's being turned on top of its head today. And so in a world that we are now entering, how do we develop verification rather than content creation? And I think that that is what ATM kind of lends itself to. Let's put the student in the center. Let's ask them what they know. Let's cater to them, but then let's also try to figure out how verification and all of this works. And so that was his large vision. And I don't know if you've, in India particularly, maybe in other parts of the world, you have an ATM system and you go to an ATM to withdraw cash. We may not use it as much today, but you go to an ATM to withdraw cash. And I think the term ATM came from that concept. Can a student go to an ATM and then just ask to be verified on something that they're good at? So if I know something about, if I'm a master at parakeets, which many kids can be, kids go down rabbit holes, trains, for example, very popular one, right? So I'm a master of trains. I know trains very well. Can I walk into an ATM booth and be evaluated and get a certificate at the end of that to make me understand exactly where I am in that pipeline? And I think that's the grand vision that me and my team were working been so hard on for at least the last 2, 3 years, but even more so over the last year and a half. And so the product launch that you see today, ATM, focuses on 1.0, which really looks at building that infrastructure and that pipeline. And that's actually around the time when Anthropic came in and you must be following the news with edtech, but India has really come under the lens for AI companies. Similar to you, we have been very, very cautious in how we approach that. So when we did speak to Anthropic, we had decided to take a step of caution. And so it was interesting because I think that at least the team that we were working with was very education-focused. They were all education-first. They had all had that experience on the ground. And so they understood the nuance that we were dealing with. And they were willing to come along that journey, which I think they have really done. So they have really took place in that process of trying to figure out what ETM's larger vision is. How do we build that infrastructure? How does it impact the user base that we're working with? But I would say Pratham has led that pedagogical aspect very, very tightly. And so, and they're very clear about that as well, that we are technical experts. We understand our system, but you're the one that understands the use case.

Sure. Yeah, yeah. And the, yeah, the specific context. No, interesting. And just to clarify, are you already running 1.0, version 1.0, and it's out in, you know, in the field? Let's say it's being trialed with young people.

It is, yeah.

And I mean, are you, is it too early to speak to any of the learnings?

No, no, I can't. I mean, it is, it is early to make like, you know, massive statements like this talk, but we have tested it out in two programs. So one is the Second Chance program, which is a program that is run across 20,000 women across 15 states of India, and it focuses on women who have dropped out of the 10th grade, who never completed 10th grade and are coming back to finish 10th grade, which is a massive signaler in terms of confidence, but also employment. And these are women anywhere from ages 18 to 55. It's a very wide age group. We tested up with about 44,000 students in this particular program, and they've taken two assessments. Through this process, so about 8,000 assessments. And what we've seen is that there is a, anywhere from a 40 to a 60% increase between assessments 1 and 2. And each use case for ATM is slightly different. Right now I can get to that as well, but what is ATM? It is an end-to-end system that works as generating curriculum-aligned questions that are personalized to particular content that you as a facilitator or as a teacher are looking to assess yourself on. And you can generate that basis your difficulty level, your grade level, your chapter level, and further personalization is possible, but these are the ones that we stuck with. And then after that, you can take your student responses handwritten, take an image of that, upload it, and then digitize that, get a grade for it, and then get personalized feedback for it. And so that end-to-end pipeline, any one of these pieces of technology in and of itself is not necessarily unique, but together they act as an engine that tries to really fit in and solve the time gap issue, but then also the personalization issue that we are dealing with, particularly in rural areas, right? The student-to-teacher ratios are very massive. So you're looking at maybe 1 is to 40 or 50. And so in that scenario, if we pedagogically know feedback is critical in order for improvement, but we also can see that that may or may not be happening in reality, can a system like this actually enable the teacher to buy back time, but also give proper feedback back to the student at a timely manner.

Fascinating. And are you— you gave the example of parakeets and trains. I mean, is that level of personalization the future vision, whereas what you're currently describing with the Second Chance program is more of a formalized assessment based on a current existing curriculum that all of those students are going through?

I think that's a great question, just because we grappled with it a lot internally as well, because if the vision is to be able to really truly meet a student where they are at, and those examples include trains and parakeets, why am I asking curriculum-based questions? You know, and I think for us, the approach that we took, and everyone can take a different approach, but the approach that we took is we need to be able to build the infrastructure that meaningfully connects each skill set and concept with each other.. And I can only do that when I have a curriculum-based setup. So when a student comes into a booth, for example, and says, "Assess me on trains," I need to now be able to understand exactly where trains sit and how do I generate specific questions that come from specific curriculum that can be personalized now to this student and what they're asking for. So the system needs to have an awareness of how curriculum works in order for us to be able to break out of that curriculum. And so that is the approach that we've taken is can we first build the technical infrastructure build those nodes of communication between each subject, each topic, each skill set, but then also go down to micro skills and then make those connections and then say, okay, now assess me on trains. Then is it able to pick up question A from a CBSE curriculum, then from an NIOS curriculum because you're looking at more freeform thinking around trains, then more structured thinking. Maybe you use an, like a train example from Africa versus a train example from India. But in order to do that, you need to understand concepts and curriculum India and in Africa.

Yeah. Fascinating. So it's, it's almost, it's actually, it's actually called a knowledge graph.

So there is a technical term for it. And I think Anthropic has, I think released a, a version of a knowledge graph that is very specific to one use case that may not tie to, let's say, the user base that we're looking at, but knowledge graphs are done. But in order to formulate that, it takes time.

Yeah, no, fascinating. So the, I mean, in my less specialist knowledge, I was thinking, as you were describing, almost the curriculums become part of the training of the dataset somehow that the agent is then drawing from.

Correct.

Limits of my technical knowledge, but—

No, but it needs a foundation, right? And how you treat that knowledge or that foundation matters, and then you there are multiple layers to this because the ground layer is the textbooks and the curriculum. The secondary layer, which gets more complex, but it looks at scoring. How are rubrics built? And we may look at grading as a very numeric and quantitative term, but it's actually very qualitative. There are very, very specific nuances to it. And for example, the curriculums that we, we worked with was NIOSC, which is a very independent Indian curriculum, and then CBSE. Which is the Central Board of Secondary Education, both have a set of handbooks that tell you exactly how the scoring happens and exactly what those 4s mean and what does a good answer look like and what does an average answer look like. And we have that sense, but it's so fascinating to then go to the ground and see how teachers are grading and how are teachers giving feedback and how are they interpreting a lot of this. And then how do good students get formulated from this?

This.

And to capture all of that through a system, you can imagine, is, is complex. But these are the separate layers that you end up trying to codify. And that's actually largely what Anthropic was helping us do, which was how do I bring what I see on the field and how grading and feedback and question generation actually happen? How do I at least bring part of that onto a system? Because you will always need a human in the loop, right? We are very big believers of that. A human needs to be part of this process, but how do you then short circuit almost a little bit and allow for quality to come out at least by 85 to 90% and then help have the human kind of complete the process.

Yeah, sure. No, it's fascinating. And then, so just to clarify then, Claude is the kind of system on which you are building or with which you are building all of this additional—

Yeah, I mean, Claude is a very intelligent sort of system that can actually understand, but it is unique and learning its interpretation of how it is taking the feedback and the curriculum and then providing that back to our teachers. And then also that additional layer of how useful is this feedback to students and teachers, because again, keep in mind the user base we work with is unique. And so if you are not able to understand that feedback, I can't give a student blocks of text. It needs to be digestible, essentially. And so in our case for ATM, We have a mix of Hindi and English feedback so that it's more interpretable by the user base that we work with.

Yeah, interesting. I was curious about language, but because given the diversity of language in India, but yeah, I mean, absolutely fascinating. And the other thing that strikes me as quite interesting for is like clearly for the verification point, there's a lot of quality assurance that you need in order for the verification to be taken seriously, right? I mean, the credibility piece is huge. If you're going to be claiming to develop verification systems that rival the current hierarchy of high status in, you know, institutions like universities, as you said. Correct. So clearly that is a key piece, has to be, right, in the conversation.

Absolutely. Absolutely. And, and I think for us, there's a delineation between formative assessments and summative assessments. And the way that we've built ATM, today very much focuses on formative, the approach for summative is going to have to be very different. And so I think in this case, you do need to build in an additional layer of verification. And we've started thinking about it and we have a thought process as well as some work for ATM 2.0 that has come on the process, but we are using psychometrics. So we are going back into the psychometric field to try and pull out exactly once the LLM has generated all of this, this, all of these questions or these grades or this feedback, can you apply a layer of psychometrics to it to be able to address some of these inaccuracies and this post-processing that we know we will need? And then after that, actually provide it to a human to be able to verify. And I think eventually you get to a place which we have in, in 2 or 3 of our pilots, things start to repeat themselves. Teachers start to make very, very similar changes. To feedback, for example, they start to make similar changes on the one question.

Interesting.

Value point, you know, and then you start to capture that and you're like, okay, this is interesting. And then you kind of tweak the model a little bit to understand that nuance and capture that nuance and then kind of move from there.

Yeah. It's a, I mean, it's a fascinating situation because I mean, I'm so conflicted because I can hear that so many amazing, you know, the access and the ability to, as you said, scale like beyond the limits of available educators in rural areas, for example. I mean, it's huge potential. And at the same time, I have this part of me that's really nervous about this direction of ultimately allowing the machine to be the final arbiter of—

Absolutely.

—an educational outcome, whatever that might be about trains, parakeets, you know, GCSE maths, or, you know, obviously there are some things that are much easier into the—

Yeah, I think when we describe it, it feels very— because you're right, you have to— there are multiple things to consider here, right? There is cost. Claude is an extremely expensive model. Most LLMs are quite expensive for the kind of scale that we're looking at. So anyone you speak to will, will mention cost. We've launched another product called Parhai, and we've spent— it's a reading assessment app. I'm not sure if you I don't know if you've heard of it, but it's a reading assessment app that works completely offline. And we put in like 5 or 6 years of work building very, very specific children audio-based datasets to be able to ensure that this model works offline. And that's largely to get around scale, internet, all of these challenges that we know we regularly see on the field. And we continue to see with ATM. Obviously ATM needs an LLM. It needs internet. It requires a lot of these. Constraints that we are introducing on the field. So I think that's one component, but then also on the pedagogical front, right? How do we ensure a human remains in that loop? Because the idea is never to replace the process, but it's more so to aid that process. And I think that we try, at least for us, we try to keep that human very much part of the process of what those questions are. How do we phrase them? How does the system grade? How does it provide feedback? And I think these elements are what we watch out for. The other thing is that we've built it in a way that actually highlights where we see doubt and where specific human attention needs to go. And so in that way, at least there's a guidance rather than an override of decision-making.

So I hear you. That's super interesting. Okay. So yeah, it's seeing patterns and, and it can target where the human intervention is, has more value perhaps, or, you know, is more needed. Yes. Yeah. Super interesting. Early detection system, essentially.

Those are the future, but yeah.

Yeah, no, sure. No, it's, I mean, the other piece for me is interesting that back to the trains and the parakeets is that there may be a recursive impact as the tools get smarter that provides an opportunity to value more of what young people already care about, can do, know, because that to me, clearly you need to orient around the mandated curricula, etc. But the fact that if the tools get so good, they can with credibility value things that young people already really care about. And so they can get some kind of proof and evidence and they can show some of these amazing things that they already are passionate about that then can become more valued in the system. I mean, there's dangers there, of course, turning them into commodities to, you know, to trade. But at the same time, there's a trade-off there also that you, you know, you actually really do get to value things that young people care about. Yeah.

And be able to, for the first time, actually provide evidence that this is actually useful information. Right.

Yeah.

And then signal it to the world. That is gonna become, it's not only gonna be nice to have, it's actually gonna become critical as people, as unemployment rises in traditional jobs. Can we actually look at people currently where they are at and actually help them with that process of moving on?

Absolutely. Because I remember a conversation I had with Sandra Milligan, who works at University of Melbourne in the Melbourne Metrics team there. She leads that work, amazing work on assessing competencies. But one of the things we discussed was how uncredible so much of the kind of the badging and the micro-credentialing has been. Because it just isn't, the assessments are not rigorous enough. They're just, you know, employers, for example, or future education institutions won't trust them as valid, credible signals of genuine depth of knowledge and ability or, you know, of skill competency. And so this is part of that puzzle, right? I think of— Absolutely. Trying to get to a place where they have real credibility more broadly. Yeah.

Even with Pratham, independent of anything to do with technology, they have a YouthNet program and a lot of that focuses on trying to credential these students in jobs like, you know, you're looking at hotel staff, you're looking at agriculture and, and these various aspects. Can we credential them in a way that actually gets them that job in the real world and connect that aspect? And we've been able to successfully do that and place students very successfully in actual industry. But then can you add in this layer of verification that happens at further scale, given the fact that all of this happens offline?

Amazing. Absolutely. It's so interesting because otherwise the monopolies on the credential capital stay with the big curriculum organizations and there's a distribution that can happen at scale if the tools get good enough to be genuinely credible.

Yeah, yeah, absolutely. And it's an issue that, yeah, I mean, all credit to Marvell, but yeah.

No, thank you so much, Krishna. I mean, completely fascinating. And yeah, no, I really, I mean, I appreciate your caution. In as well with the ways that you're talking and thinking about this, you know, moving quickly, but not too quickly that we forget some of the important humans in the loop and values and ethics and all of that. But no, so, so interesting. Thank you so much.

No, thank you. I'm looking forward to hearing more and everyone else's perspective.

We hope you've enjoyed this episode. Please feel free to continue the dialogues with our guests, with us at Good Impact Labs on our or on social media, or within your own networks.