The AI App Seeking to Scale 'Magic' in the Classroom

Hey, Michael.

Hey, Diane. It is good to see you as always. It's been fun because we've had this arc of really digging deep on AI with school models, and then we've been moving into the edtech tools that are starting to define a lot of current models, the new models we might see, and so forth. And we've got another big one, dare I say, for today. This is going to be a good conversation.

I am very much looking forward to it and looking forward to digging in. But before we do that, we have a quick ask for all of our listeners. Will you all rate or review Class Disrupted wherever you're listening to it? And of course, please subscribe. We've never asked anyone to do this in 7 seasons. And, you know, this is, it turns out this matters. So we'd really appreciate it if you would do that.

Yes, we would appreciate it. And we know we get a lot of feedback from listeners. We certainly get a lot of emails and texts and things of that nature. So we know y'all are listening, but we need to see it in the ratings, reviews, and subscriptions as well. It helps other people find the information too. As folks know, this is a passion project for us and we want it to actually be influencing the conversation more broadly. So look, we know we get a lot of folks tuning in from our great partners with The 74, Substack, Apple, Spotify, YouTube, you name it, we know you're out there, but subscribe, rate us, leave us a comment, and we'll just be appreciative for that.

We will. And we will continue to be grateful for the direct feedback as well, because we use that, we take that in. And in fact, this conversation today is something folks have asked for. So we, I will not waste any more time.

Yeah, no, it's true. Well, I'm thrilled because we get to welcome Adeel Khan to the show, known obviously for Magic School, but before that, we'll talk about Magic School in a moment, but Adeel worked as a teacher, assistant principal, founding principal for DSST, the Conservatory Green High School in Denver in 2017, and then left a few years. I think it was like 3 or 4 years in or something like that to coach school heads. And my sense Adeel can correct me if I'm wrong on this, but my sense is this wasn't the calling. A year later, I think it was, ChatGPT bursts on the scene into people's consciousness in November of 2022. And Adeel says, there's something here. This can revolutionize the teaching experience. And, you know, quickly sees like, hey, teachers aren't going to figure out how to use this productively. Let's help them do that. Builds Magic School and it takes off like a rocket ship. rocket, no exaggeration, I think. Over 7 million teachers now use it. That number's probably out of date as I say it. Partnered with more than 10,000 schools and districts, I believe, over 160 countries. So we've got a lot to dig in here. The use cases are broad, but Adeel, first, welcome. It is great to see you again.

Thanks, Michael. It's great to see you too. And Diane, pumped to be on the podcast today. And thanks for the kind introduction, mostly accurate. What

did I miss? Yeah. What did I miss?

You got it mostly there, I think. Certainly when I got to coach principals, I didn't, it wasn't my favorite thing in the world. Not because it wasn't a great opportunity. I worked on a great team and principals need support. But rather, I think when I was founding my school and building it, I thought, if you'd have asked me if I had the most important job in the world, I would've answered yes. And I just missed that feeling. I didn't have that kind of builder chasing really ambitious dream. feeling when I was in that new role. And it's sometimes good to know that about yourself, right? I think that like sometimes trying something and knowing that, you know, where you are, 'cause, you know, in some sense at the end of my principal career and being graduated our first class of seniors as I kind of built the full school out, I was exhausted, right? And sometimes you think about, you know, what's the other side like, you know? And, you know, being an advisor sounds less intense than being in a school building every day and, you know, chase this big dream and, And then, you know, knowing what the other side was like, I was like, oh no, some people are wired for this, right? Being in the builder seat and being—

You are.

I learned that about myself. And even when things get hard today, and they do, we have a lot of challenges, I know that this is what I want to be doing. And I would be, you know, I wouldn't be as— there's no grass is greener thoughts anymore. Even when those seep in, I'm like, nope, I know this is the thing, even when it's having my most stressful moments. So I'm really good at it.

I love it because you're willing to make the trade-offs for the thing you really want. And obviously we hope all our young people start to grow that same muscle right in their lives.

That's the work I'm doing right now. So it's an inspiring story. Well,

I was going to say, right? So like we have 2, you know, Diane, Adil, you know, founding school heads become founding edtech company CEOs. I think the story of the founding story of Magic School is somewhat well known, but maybe what's less known a deal is like how you think about the problem that you're actually solving, both originally and today, because you all have expanded your scope quite a bit. But let's start at the outset itself because I think your framing tells us a lot about why you grew so quickly. So, you know, how do you describe what Magic School first solved for teachers and why that approach was so important at the time it came out?

Yeah, so I think that there's a part where there's maybe a little revision on the way you described the problem that I found was that not that I didn't think teachers would be capable of figuring out how to use this and use it productively. It was that the tool itself was a general purpose tool. It wasn't built specifically for the domain and therefore it was clunky to use. And at that time, even really good prompting didn't yield a really good result for teachers. It feels like ancient history that you had to have massive prompts for an LLM to understand what you really meant in the prompt. But at that time, it really was. So if you were prompting GPT, it was like you could get something quality, but your prompt had to be a page long for it to really understand your context, really understand the format of your output, just all those different things. And if teachers were going to use it right away, we had to do some of that work to get them to a magic moment with the technology. A lot of teachers would report to me when I was training them on ChatGPT, which is kind of the way the product idea started was, you know, I was prompting it so much, it wasn't saving me time, it was costing me time. So there was like kind of a really acute problem of, you know, promise that this has an opportunity to assist you, save you time, and also make you more efficient, augment your abilities, but it's not working for you right now, or the vast majority of teachers in that moment. But I think of it as, you know, when I started thinking about building the product, It started with, I actually thought I might be training teachers on ChatGPT and that just didn't work. I experimented with it. I went to my school and teachers didn't really adhere to the product. So I was like, there's got to be a better answer to this. I was savvy enough to kind of be looking around to see products that were coming out that were generative AI products. And there were this first set of products that were hitting the market. There was Harvey, which was AI for lawyers. I remember that really vividly. And I don't think they even had a live product, but it was announced. As a product and they'd gotten some funding and there's news stories about it. And I was like, oh yeah, there's going to be an AI that is made for each kind of profession. And that makes sense because each profession has its own quirks. It has its own domain knowledge. It has its own workflows. It has all of these things that are specific to that industry and a generalized large language model isn't going to be the answer for every single one. So that was kind of the clear vision from the start is we make a really domain-specific intimate product for teachers and the product's evolved a ton since then. But that core of it was like, it's going to be really domain-specific. Even in those early days, I thought to myself, we're going to make some tools in this platform that if you've never worked in a K-12 school, you would have no idea what that even means. Like that was kind of my bar for myself is like, I want it to be so intimate, so clearly made for teachers that They scroll through that initial dashboard of tools and they see a couple things and they say, you know what? Somebody who's worked in schools and like built this. I still hear that today and it is one of the things that brings, brings me the most pride. Teachers don't necessarily get built products that are so clearly intimate for them and know their work. And that's what we hope to do and continue to do with the platform.

This is, this is such a good place to jump in because one of the things Michael and I are doing this season is trying to help people see how AI is truly being used in schools. Like today. And so building off, um, that origin story and where you started and where it is today, I mean, uh, let's get into like how Magic School actually works and what, where the AI is in it. And, you know, this is the moment we get to kind of nerd out, which is fun. It sounds like you're, you're in that space. So like, just to give people some context, when you, when you enter as a teacher, it seems like there's basically 3 big categories. of support, teacher tools, writing feedback, and the Reina, the chatbot. And so, and then when I look at the list of teacher tools, you're right, I open that up, I'm like, oh my gosh, there's like 75-ish tools. And, and we're getting everything from like a worksheet generator to report card comments, unit plan generator, standards unpacker, IEP generator, quote of the day, classroom management plans, and even tongue twisters, you know. So like, help us unpack like Literally, what's going on here in this tools section? How's AI, what is the role of AI and what, how are teachers using this?

So one thing that, that, uh, I think Magic School does exceptionally well as a product is it, it, while there's a breadth of tools and sometimes we get the feedback that it's overwhelming, it's like this tension we, we feel between it being overwhelming and also really Familiar for teachers. We've gotten that feedback since the very start, and I always hesitate to change the UI or make some meaningful shifts there because it has kind of become our calling card where there's this list of teacher terms almost, right? The UI itself has been replicated by 20 other products at this point because it works, right? It's like when you say AI to maybe a skeptical teacher or a teacher who's like, technology's not actually been a meaningful part of their story in the classroom, or a teacher who's Maybe been sold on technology being a really big opportunity for them and they've been burned. And there's all kinds of reasons why teachers and technology have not gotten along particularly well. I will be clear, I was not somebody who, like, as a principal saw incredible value in technology. My school didn't subscribe to any boxed curriculum type technology tools or platforms. We built all of our own internal, we built all of our own curriculum. We didn't subscribe to a single edtech product outside of the LMS that we used. And that was a kind of a requirement. And the reason was mostly because I just didn't know that they were going to drive outcomes for my kids. And I was an outcomes-oriented principal. I wanted to make sure our kids' literacy and math scores and their SAT scores were— that was kind of our focus, was preparing them for college. And I wasn't sure that any of the tools out there would make a difference. And I might have been just wrong. Maybe they just weren't in my world and there's some great tools out there I've learned since joining the edtech world. But nonetheless, I think there is that skepticism amongst folks and there are tools that have come into schools and have done just about nothing. I mean, everyone who's worked in a school district or building will tell you of like, remember that like 2 years ago initiative we had? What happened to that? That is just like a common story.

Mm-hmm.

Uh, it'll be like a new curriculum that's gonna revolutionize the way that you teach and gonna completely dramatically— all those kids that are behind in your class, they're gonna catch up in a year. Uh, or here's a new tech tool that is promising to do this. So there's all this baggage around like products that come into schools. And the baggage often is, I invested so much time and energy in in implementing this thing. And I don't know how much value I actually got out of it. And then the district gave it up in 2 years. There's frustration here. So if you think about Magic School coming in, really novice founder in me, I was just like, let's get people to value really quickly. Let's just get them into what this thing can do so it's believable. So you jump into Magic School, you see rubric generator, you click on it, it asks you some pretty simple form fill type questions like what grade When attached to the document, maybe of the assignment you're going to be assessing with this rubric, all very simple, super easy to use. And then you click the generate button and you get a rubric that looks like a rubric. It's what you expect it to be. It works the way you want it to work. It meets your expectations. Uh, whereas like at the, at the, you know, in early days, uh, ChatGPT and even still today, like it's even the simple friction of if you went into a regular chatbot and you tried to build a rubric, of course you could. But like, you'd have to type in like 4 sentences. I want a rubric. It's gotta be 6 columns. It's gotta be, you know, like, uh, make sure it's like a boxed format. And then like, you know, you do it and then it, then the LLM will respond something like, oh, did you, you forgot this detail. Can you gimme this detail? Like, wait, wait, what? Like, I just gave you, like, it's a frustrating experience for somebody who's been told this is gonna save them time and have to now funky think about prompting. Nonetheless, like, that's delightful. A rubric generator that gives you a rubric is like a zero time to value, incredible experience for the AI. I see how this is going to be really helpful to me in my daily work. And the flywheel it creates is also a secondary part of value, is if generative AI can do this for me, then what else can it do for me? And we have like a chatbot on the platform, Raina, that is just like a ChatGPT, but built for education with a couple bells and whistles and education-focused context. And if you use that, like, if you know that, you know, Magic School could create a rubric for you, you know, we're not hiding the ball here. Like, you know what's happening is there's a prompt, there's inputs, it's coming out with something. You know, you can probably prompt Rana and do something completely original or new, but you need to be able to see the value first for you to be really sucked into it. And like, I think you do an amazing job of getting people in the door with something really low stakes that should like, Turns out to be really high quality, saved you a bunch of time, and it makes you want to use the platform more. It makes you want to think about, okay, what are the other tools? If it did this thing for me, what else can it do for me? And I think that's the way that we see the flywheel start. We think about the user journey as like they start with these like really simple tools. They might graduate into like their own freeform prompts in Rena and trying things. And then they might decide to use like an agentic workflow, like our class writing feedback tool, connect their Google Drive and have writing feedback dropped on each of their documents. Then they might wanna think about, well, if I'm using this in my work and it's really valuable, it's helping be a thought partner to me, I wonder if it'd be helpful for my students too. So then we—

That's super, yeah, super helpful to, to understand your thinking and your flow. And it seems like that is working. Let's go back and nerd out a little bit on the rubric generator. So where, what's, what happens when they type that simple thing in? Like, how are you using AI? Literally what's going on behind the scene there? And as a person who's written a lot of rubrics in my life and used a lot of rubrics in my life, like how, how do I know that's a good rubric? And like what's going on behind the scenes and under the hood?

Yeah, so great question. When Magic School started, it was just a prompt. It was like me with a prompt in the background. I had created rubrics as an English teacher in my life and the best instructions I could give and the judge was me. Like, I would determine whether or not the quality was high enough or, or, you know, would meet my standards. And then it was a group of users who were doing the same things upon using it.

Right. And you're prompting one, one of the big LLMs essentially.

And that's 2 and a half years ago. So just, just know that's like, it's radically, radically different today. So today we have a trust, safety, and quality team run our own evaluations on the platform. We have pedagogical experts on the team who are reviewing for quality. So there's a human part of it. Then there's a thing called LLMs as a judge. Some of our evaluations are using models like Claude to judge the quality of the rubric. I was actually on a panel yesterday with one of the teammates at Anthropic. His name is Nirav, and he was naming that Claude itself at this point is probably the level of a PhD in every domain. Mm-hmm. So you can have Claude, PhD in a domain almost nearly and getting better every day, judge the quality of the rubric too, and then iterate on the prompt model we're using and change those things out. So we actually have a wildly more complex version of judging for quality internally now. And it's a combination of like LLM as a judge, human in the loop, user feedback on the platform, And then when new models come out, we are on a regular basis, once every week or 2, we're saying, is this new model better performing against our criteria? And so each one of those essentially roll up to a score and we have like a score that determines whether or not this model can exist in the platform. And sometimes, you know, if it's a 94 from one model and a 95 from one model, but one model's like 110, the cost, we will choose the one that's a little bit cheaper because we have to keep our platform affordable for schools.

Mm-hmm.

The vast majority of them, we were able to say the absolute best output is the one that we go with. But there is, they're judged across a lot of different quality markers now to ensure that the output is something that, uh, that we can stand behind.

So just so I make sure I'm understanding, when I go in there and I pick a teacher tool, like, like, um, I think it's report card feedback or something like that, you all have kind of established what good report card, the elements of good report card feedback. And then you've created prompts that are prompting the LLMs to do that. And then you've gone through and tested those responses to make sure that they're quality. So, because literally the humans aren't testing while I'm the teacher here asking for that, you know, it comes back like that, right? So, so you just have confidence in those responses I'm getting.

Yeah. Yeah, we're essentially running out like hundreds of queries against them and then judging queries and, and then orienting around what the best kind of set of all of those variables are. What's the best prompt? What's the best model? What's the best output?

Got it. Okay, got it. Now help me understand like the, that rubric. So now like I'm a teacher in my classroom, it's often been standard that the teacher's sort of like in their classroom doing their thing, using their own rubric. How do you think about it from like your principal's Like the whole school. So, I mean, you know, for example, when I was leading schools, we built a longitudinal rubric that, you know, over time the kids were. And so one of the things as I play with Magic School, I'm like, oh, what's the bigger picture here? You know, what's the, what's the high school arc or what's the full arc? And is this kind of more of an activity in my classroom base or is it the full big picture, the backward planned? Yeah. approach? How do you think about those things?

Yeah, it's a really great question. Right now, I think that Magic School's teacher side is best judged as kind of an assistant that helps you in the moment that you need it in your daily classroom activities. You can use Reyna as a thought partner when you're struggling with something, or our chatbot. You could, you know, really build out like a full, like, unit, then subsets of lessons and like a generative thread. All the tools are threaded. So for example, if you started with the unit plan generator, you could then say, build me the first lesson in this unit. It keeps context of that entire thread, and you could give it input about, hey, my student's performed this way on it. Can you generate the next lesson with that context? And you could do it that way, but transparently, few people do, right? It's really like they come in and they kind of get what they need and they keep going. We want to move toward more of what you're describing, which is like, Hey, the entire cycle is brought through in the platform. You give the platform your intent and you keep these really rich threads with memory, context, and knowledge of your classroom. So we're moving toward where you can kind of have entire context of the classroom experience built into the platform. One of the key components of that is assessment. Yeah. Like if the platform understands how your students did on their assessment, let's say for example, Today, you could build an assignment in Magic School and you could, you could actually, you could build an assessment in Magic School. It could be taken by your students in Magic School. The results of that assessment could produce some really interesting insights about how they performed against the standard. And you could then create a material based on that assessment. So that's where we think it's going is that, hey, you're going to build things based on the assessment results that take into context your students' areas of strength and areas of growth based on the insights that are in aggregate across those folks who took the assessment. And you can imagine that like there's even more you can do with that, right? Like there's on the student side, on an individual level, if you had assessment data about a child and you also had them taking assessments in the platform, you had them doing activities in the platform, you could trace a student personalized learning profile that understood kind of their academic needs. It could be continually updated with memory around the way that they interact with the platform, not just like from their academic strengths and how they're growing, but also their stylistic preferences, their post-secondary desires. And you could really make these incredible persistent learning experiences for students that they always can tap into that's associated with a really rich profile to serve them right in their zoning proximal development and in a style and in a way that they will engage and learn more.

I'm curious off that, just Because that becomes a very big vision that shapes around the student. Starting from a teacher workflow to the student lifecycle almost, if you will, how much do you need to know and collect about individual students and how much do you need to know about the specific, like take the assessment question, right? So I'm going to give feedback on your grappling with a particular poem or a book or something like that. How much does the model then need to get trained, not just on the teacher rubric, but actually on the content itself, as opposed to just the standard, which is probably a higher level statement of ability to do X, but ability to do X in one context might be very different from another. So like, just help us think through the scope of that, of getting to that vision you just painted.

Yeah, it's a really good question. I mean, we are at the early stages of understanding how this will work without student data. But as we build, so one One of the things that we are doing, or are in the planning process of getting some of these ideas off the ground, there is a really interesting process where you can kind of use LLM agents as sample students. And then you can also assess the quality of their interactions with the platform based on specific profiles that you create for the AI agents. And then you can kind of get a really good dataset on what you're describing, Michael, is like, is it working? Is it actually meeting your needs? Or is that agent, that student who's an agent who has a stylistic preference around visuals for their, that's the way that they learn best, or audio is the way they learn best. Their reading level or their reading test score diagnostic was this, and this is what that means. You can kind of create all these profiles individually, and then you can run them against the interactions they have with the platform. And then you can judge how high quality the interaction is on the platform. With like an external observer. So like almost similar to where I described that LLM as a judge, the judge can actually judge the experience. So you can think of it as almost like a principal judge, right? There's a principal who's an agent who's watching the AI interacting with the student and determining, is that a high-quality interaction or not based on the child's profile, based on what I know about what high-quality instruction looks like? So you can start building these like kind of recursive loops and running them hundreds of times and get to some pretty pretty impressive verifiable outcomes pretty quickly. And of course, AI is not kids and they're not going to be perfect.

And AI is not going to act out for us.

Yeah. So I don't want to kind of anthropomorphize that this actually is the answer, that we can simulate everything through generative AI, but it's a great way to, in a basic sense, feel like, okay, it works in a simulation. And then when we bring it to students and we work through the pilots and we see schools and districts who embrace this, then we can see it in action. We can combine those insights with the insights we got from our simulated experiences and make something that we think is really powerful. Some of those generative experiences, we're going to be starting this summer in summer school with some partner districts to see how it works in practice and is it actually the needs of the kids. And we're going to use that data as data to inform the product as it gets into a more live cadence.

You just said so many interesting things in there. Like, I'm of like 5 minds right now where, what's the thread that I want to pull? But let me pull on the one because you started by saying like, we're playing with what we can do without student data in there. And best I can tell based on my experience in Mando, school, you're not connected to an SIS or anything like that. So you're not pulling in any data as the teacher around your students. That said, I did create, I did use the IEP generator and I generated an IEP and I, you know, I look, I didn't do it on a real student. Obviously I've got like 25 years of experience, so I sort of built a persona in my head and like input the information. And I will say I was like kind of blown away that a fully formed, detailed, and dare I say, very confident IEP like came out, you know? And at first I was like, oh my gosh. And it looked like an IEP that you would kind of read in a school sort of thing as I was skimming it. Then I started like really digging in and I was like, oh, as someone with 25 years of experience, like This is not the IEP I would've written necessarily, 'cause I, in my mind, you know, the kids I'm thinking about, like I have intimate details and I'm like, wait, that feels a little, it felt a little AI-ish, right? Like sometimes AI gives you really, like seems like compelling results, but they're not very specific or personalized or whatnot. And so that was one thing I was curious about. Like, how do you think about some of the tools that are like that and, and how they get used? And like, I was thinking, what if I'm a first-year teacher? Yeah. And I use that and I don't have 25 years of experience. Like, can I, how do I do that? And then the second piece is you're out with teachers, like, do they just pour a lot of information about kids into the platform? And I'm sure you get a lot of questions about like privacy and security and like what's going on there. So I'm curious about those 2 angles.

Yeah, so I mean, we do a lot of professional development. I'll start with the second one. We do a lot of professional development around making sure that Teachers do not share information that could be sensitive into the platform. So there's upfront training in that tool you used. Actually, you'd see that it actually actively reminds you in the tool itself to not, because that's a tool that's more susceptible for you to maybe submit accidentally that data. Of course, if any data is brought to our attention that was submitted to the platform that was PII, we remove it immediately. And we have incredible data handling practices and Safe things are all available on our website. So we've done a lot of things to make sure that schools and districts can trust us. And so that's certainly something that we consider. And we just think training and enabling teachers is the best way to prevent that stuff. Because even if Magic School has really great data handling practices and is kept safe, they might bring it to another platform. And we want them to know how to use our AI, but also any generative AI. And keeping student data safe is incredibly important when you're using these tools.

Yeah.

Second part, Diane, I think is really interesting. So I was a special education teacher too. So for me, I can think back to when I was a novice teacher and Magic School would've been a godsend for me. That IEP would've prevented me from being anxious. Uh, it probably would've helped me save a lot of late nights and it would've made me feel like I had real strategies to, to support kids because like, you know, it's a, it's a good point. So, so your question, I think I might challenge this, like, You presented it almost as like a fear that then they're not 25 years, so maybe they're just taking this robotic IEP that's not as good as the high-quality 25-year IEP. I want you to know that like I've been a principal reviewing IEPs and I have seen teachers, wonderful, hardworking teachers submit IEPs with the wrong names in them because they were just copying goals and pasting goals from one student to another because they were just trying to get it in by the deadline. Wonderful, hardworking, incredible teachers. So do I believe the world has gotten better because of MagicSchools IEP generator? Yes, it has. The floor has been raised because there are actual— because now the barrier is you understanding the student and getting some high-quality things. And even a novice teacher will see the goals and at least they have formats. They know how to write goals now. I didn't know how to write goals, right? And I didn't necessarily have someone to go to to help me write those goals. So I would say net-net, MagicSchools raised the floor dramatically for the way an IEP is written. I think a family would be much happier to see a Magic School written IEP than the ones that I was editing, if I'm being totally honest. And it's again, not because teachers aren't wildly hardworking and talented, it's because there is no time. And so I think that's kind of the reality. Now, in an ideal circumstance, they have a really great draft and then they have an instructional coach like you who they can go to and say, what do you think about these things? And they can question and challenge them and push them and make them even higher quality. But I do think that sometimes we miss the reality of what happens in schools when we're critical of the tools that teachers are using to center their practice. And sometimes you just need to trust teachers. In that case, I'm like, actually, I even trust the first and second year teacher to use this appropriately, and especially if they're educated and they're told the way to use it, not to submit private information and things like that. But I always challenge people. It's like, yes, we have schools and districts that hide the IEP generator because they're so scared of it. And, you know, we respect anyone's decision on what they want to do, and they're allowed to do that. And like, you know, our enterprise can do a one-time product, and we support them in doing that. But I, on a personal level, I always challenge them. I say like, look, like, ask your assistant principals who are reviewing NAPs.

Yeah.

What they see, are they, are they higher quality because you took this tool away? Or are they higher quality when Magic School's in the loop? I think that's like the question to ask is, What, what's the before and the after rather than like, what's the ideal, right?

Mm-hmm.

I want the ideal. Um, but I think that there is like a, this go-between I think is, is, uh, is super strong. And I think that the tools are getting even better over time. And I think that the, the better the context that you share with it, the better it'll do, of course.

Yeah, that, that, um, that makes sense. And I think I, I understand the perspective that you're coming from and certainly your lived experience. And I do think sometimes people, you know, have an ideal Of what is happening in school versus a reality of what is happening in school. And so I appreciate that. I was also really tuned into you saying that as a school leader, you all developed your own curriculum. That was my reality too. You know, we ended up building curriculum. We ended up building Summit Learning, which was like a full massive curriculum. And so I'm wondering how, but a lot of schools have adopted curriculum, as you know. How, how do you recommend that teachers sort of deal with the world of like, I have an HQIM that I'm, I'm given by my school and I want to use Magic School. How do those 2 things play together or do they, or what does that look like?

Yeah, I think that, you know, again, lived reality of an HQIM in a district. Uh, I've lived in that reality as well. We, we, we didn't box any curriculum, but sometimes we would get, uh, for a certain subject, we would have like, you know, pre-prepped lesson plans or whatever it might be. And I think that the lived reality of those things are teachers are always modifying, changing, supplementing, making those things work for their classes, and that's good. So I think that that's what we'd encourage them to do is like, yeah, keep the spirit of what your school wants you to do. And obviously there's a research base behind the curriculum that you're using, hopefully. And we want to be a great supplement to that. We want to make sure that you're able to build the supplementary materials that you need to make sure your class functions and works, and Magic School could be used alongside There's a world where we build knowledge into our platform, so the schools and districts can upload things like standards, curriculum they've built internally that are not copyrighted by the publisher. And there's a world where we partner with publishers too, and we bring those knowledge bases into our platform and call to them as well. And as you'd imagine, those publishers aren't super excited to work with large language models because this is their proprietary IP. And nonetheless, I think that we think that there's a future where there's kind of a win-win for both of us in a world where teachers are finding that they're starting their day with Magic School and it'd be really convenient if they can pull in some of that information and make those curricula more flexible with generative language models. But yeah, I support that. I think a lot about curriculum and I make spicy posts on curriculum on my LinkedIn if you haven't seen them. Yeah. But, uh, I am, there are a couple curriculum that are incredible and obviously they've driven meaningful outcomes for kids. And I don't think they're all that way, to be clear. I don't think they're all super research-based. I think that a lot of them are, uh, are not particularly valuable. And I think, I don't think, I know a lot of teachers hate being put in a box. They hate having to be told that you can't be autonomous in your classroom. You must follow a script because The script is better than anything you would create, which is like the, not the intended message, but the unintended message that a teacher might receive when receiving certain curriculum. And of course, there are curriculum that teachers love, right? Like they tell you, no, this thing's amazing. It's changed my classroom. So not painting with a broad brush. There's also like really amazing ones. I will say that I have my lived experiences share that like if you trust teachers to go find the right resources and give them the tools to do that and know their kids and coach them really well, you can get really extraordinary outcomes. Mind you, most of my experience is secondary, but we have incredible results for our kids. We served a high-needs population and had dramatic growth. And so that's my own experience. So I don't know. I think that there's a little bit of like, I would not call myself as a personal, like on a personal level, I do not ascribe to the church of boxed curriculum. Yeah. That is not my ministry as we used to call it in Atlanta. Uh, so, uh, do I think there's really good stuff out there? Absolutely. It's not my ministry. I trust teachers. Yeah,

that makes sense. Clearly you are outcomes-driven. I know that based on the school that you started and all the language you're using here. How do you think about, how do you at now at Magic School think about, uh, what outcomes you're per— how do you hold, what's, what's your bar? What are your goals? Like, what are your outcomes that you want Magic School to drive to and how do you measure them?

So there are a couple of ways that we're thinking about this. So outcomes are at the heart of our mission. We have named goals in our company about how we are going to drive student outcomes in classrooms. Right now, the way that looks in our platform is the amount of what we would call feedback delivered to students. So generally what we've seen in the first 2 years of the product is that the things that teachers have said have driven growth, the experiments we've set up or have not set up, we've just been reported are that when students get things like feedback on their writing that they've done on their own aligned with a rubric in the platform, that is powerful. That is something that drives an optimized classroom. We have actual classrooms that have shown state exam scores change over those things. We have enough data to say that when Magic School gives student feedback, that teachers is controlling and aligning to a rubric or the way that they'll assess students, that's a great thing. So we measure right now how many instances of feedback is Magic School giving to a child under the supervision of their teacher through either our assessment platform in the product, which gives students just-in-time feedback after they take an assessment, or in their feedback portion of the platform where they tune in. So those to us are pretty hard. We don't look at it as like, hey, you talked to a chatbot, so you learned. That's not enough for us to count in something that we think is definitely going to drive an outcome. It certainly might drive AI literacy, and we certainly think about that and measure that as well. But in terms of outcomes, that's the way we think about it now. Where we want to go is we want to be able to provably say that students have learned and judge it in the platform itself. So one of the ways we might do that is through having a diagnostic assessment students take at the beginning of the year and then at the end of the year, or tracking time in platform as we have more persistent students. profiles in the platform and simply asking the district to themselves compare their users' data, which students spent the most time in platform and how much did they grow, and then just give us a report back. There's like ways that we can kind of like say it's not a perfect correlation, but if students are spending more time in the platform and they're getting better results at the end of the year, they're growing more than their peers. That's usually a pretty good indicator to us. There's an experiment that was run unbeknownst to us in our first year I actually shared this on a panel yesterday. It was at South by Southwest, but I think it's a powerful example. And I think what a lot of organizations are doing around generative AI right now in the world. And I actually think this is fun because like a school district was ahead of like enterprises. It was like they're ahead of companies in the way that they're thinking about generative AI. And our first year that we had like a real enterprise product was 2020, um, 2024 to 2025 or 2023. Our first partner, which you imagine our very first partner at Magic School would be pretty innovative, and they certainly were. Aurora Public Schools, one of our very first ones. At the end of their first year using Magic School, they got a printout of their MGP scores at the district level. MGP in Colorado is basically like a growth score that is associated with each teacher in the district based on their state exam scores. If their grade has a high-stakes exam. So the district quite literally gets a list of all the teachers who had the highest growth in order of this calculated growth score. So they kind of have which teachers are having the best results in their classroom. It's a pretty sophisticated calculation. It tracks what was their expected growth based on their prior year's performance to this year's performance. So it's a pretty solid number. At the end of the day, it's like the kids actually grew and it's based on some pretty hard metrics. What they did was that they The way the story is told is the academic, um, one of the assistant superintendents called the technology director and said, I have all the teachers who have the highest scores or the highest MTP scores in the district. Can you pull up MagicSchool's, our user dashboard, which shows which teachers are using it the most? And, uh, they said like one for one, it was like best growth scores were in the top 20 users, best growth scores, users. So the way industry is doing this now, what we're doing at MagicSchool is we, we have an enterprise version of Claude. that our employees use. And we're, we have a real big focus on like agentic coding for our engineers to move faster and ship with more velocity, build a lot of really amazing things for our users. And one way we're measuring like the efficacy of generative AI right now is we're looking at our leaderboard, like which engineers are using the most tokens in, in our version of that enterprise dashboard. And then we're asking the managers, is it, would you call that your highest performer? Like, is that engineer who also is using the most tokens in Claude performing better? Are they shipping more? Are they kind of meeting the goals that you've set for your team in a better way because they're using AI more? And if the answer is no, then we need to rethink about, like, you know, is generative AI actually helpful? But if the answer is yes, then we need to spotlight that engineer. We need to have that engineer teach the other engineers how they're using it and how it's making them more productive. And Aurora was doing the exact same thing 2 years ago. So I think it's a really, really cool way to think about how this is actually amplifying productivity in a meaningful

way.

Adil, I think that's a good place to leave the conversation for now. I appreciate how much you've geeked out with us. And also I appreciate that you've told us like where it is now and where your sort of your vision for it is. As you know, a lot of entrepreneurs, they sell the future present as one package as opposed to distinguishing the two. So I appreciate that in this conversation as well. Before we move to our last segment, as listeners know, I just want to thank the Learner Studio because this season of Class Disrupted has been sponsored by Learner Studio, a nonprofit. It's motivated by one question, which is what will young people need to be inspired and prepared to flourish in the age of AI as individuals, in their careers, and for civil thriving? Learner Studio has been sponsoring this season on AI and education because in this critical moment, we need more than just hype. We need these authentic conversations that we've had today, asking the right questions from places of real curiosity and learning. You can learn more about Learner Studio's mission and the innovators who inspire them at www.learnerstudio.org. And with that, we'll move to our last conversation, which is, it's just a fun segment we've had, and people track this and so forth, Adeel, on LinkedIn and elsewhere, believe it or not, about about things that we've been watching, listening, or viewing, and would love to hear something that you've been tuning into, what's either on your bedside table or on your playlist.

I think I'll go with a Netflix show. So it took me quite a long time to get to it, but I finally watched— I just finished the last season of Stranger Things, which I was late to the party on in the first place, but kind of binged it a few years ago. And so I was eagerly anticipating the final season. None of it got spoiled. for me, and I got to watch the whole thing, and it was delightful. So, uh, I felt like I wrapped the bow on a, uh, a really special— I feel like Stranger Things is so awesome if you guys have seen it, but I feel like it's such a special cultural, like, phenomenon that everyone kind of experienced together and watched together. So that was— that's my, um, all right, cool. Speaking of new shows, so excited to hear your guys', uh, well,

I'm not gonna give you a show today. I apologize in advance. I'm gonna give you a book. It's a little bit ironic to have AI books, I think, but this one feels special to me. So it's called Co-Intelligence: Living and Working with AI by Ethan Mollick. And I will say when it arrived, my kiddo who works on the models said, oh, that's perfect for people like you. I was like, what do you mean? He's like, you know, for regular people who don't understand what we're actually doing, but like, have some sense of it and it's really useful for kind of who are really trying to make sense of AI and the world and what that looks like. And that's what it feels like to me. And so it's, you know, I'm not telling you anything new by sharing this book with you, you know, a bestseller, but it is, it's short and it's thoughtful and I think useful. And anyone who's really trying to grapple in this space, I The

one question, Diane, I have, I love Ethan's Substack as well, and when the book came out, I bought it as well and read it, but I'm just curious, like, does it still feel current given like the race?

You know, yes.

Interesting.

Yeah. I think so.

Okay.

I think so.

Cool. All right. I've got a book as well. So Adil, I'm also striking out on the show watching, I think, at the moment. But I still haven't done Stranger Things. Diane, I think this was on her list. I can't remember how many episodes ago.

I started early, but I haven't finished off the season. So you've inspired— you've finished. I

was about to say, you just inspired her to finish it. Yeah. The book I've been reading is Heart of a Stranger by Angela Buckdahl. She's a rabbi at the Central Synagogue in New York City, which I think is the largest synagogue in the US or top 2, I guess. And it's, it's a— she's a Korean American rabbi. And so it's like a very interesting story about where she, the circle she has not belonged in, and then making sort of this, I guess, momentous movement into being a rabbi and sort of what that's been like and her life story through it. So it's been a very good read. Diane knows my wife's Korean American, so like it's, and I'm Jewish, so it's like hitting on multiple levels in our household. TBD if anyone else in the household reads it beside me, but I'm enjoying it quite a bit. And we'll leave it there. Adeel, huge thanks for coming on, joining us, geeking out with us. And for all of you, keep the questions, comments, notes coming after this episode and in general, and we'll see you all next time on Class Disrupted.