Season 4, Episode 9: Shake Up in the Assessment Market

Foreign.

Hey, Diane. Wacky weather is continuing to roll out across the country, it seems. And I hope, speaking of that, you've all stayed dry enough in California, which, frankly, is not a sentiment that I normally need to express to you.

Well, Michael, the persistent rain has been a little bit strange, a little bit unfamiliar, but I will say also welcome, partly because we don't have any personal complaints, you know, haven't experienced significant destruction or risk, and others have. So I want to be mindful of them. But I will say, Michael, it does seem like our schools are never going to catch a break because with these storms have come power outages that have caused us to have to close several of our schools for several days. And honestly, at this point, we are kind of starting to wonder if we're ever going to be able to consistently operate our schools with everyone in them.

It's just insane to watch all this, Diane, and I've been following some of this with my friends in the Bay Area on Facebook and so forth. But aside from weather and the normal question of our students in the schools themselves, which has been a recurring topic on a lot of our podcasts, I've been looking forward to talking with you because there was a major announcement in the world of assessments and obviously assessments and accountability and the challenges they pose to the kinds of innovation that we both want to see are something you and I have discussed a lot on this show. But given the latest news, I'm just wondering if we can go much deeper on this topic. I have a bunch of thoughts loaded up, but I'm trying to sort out that are informed from the innovation side of my work on what this means for education. But I'm. I'll be candid. I'm really excited to get your on the ground perspective. Just to check my instincts on this.

Oh, you know how much I love geeking out on assessments, so I'm totally up for this. And before we dive in, though, I'm pretty positive we're thinking about the same story. But just to be sure, are you talking about the one where Houghton Mifflin or hmh, one of the three big textbook companies, acquired nwea, which as many of us know, is a nonprofit that's most known for the, what we call the MAP test, which is basically an adaptive benchmark assessment. And I think it's used in about 10,000 school districts across the country. And I will say personally, I know used in a lot of charter schools.

Yep, that is the announcement that I'm talking about, Diana, grabbed my attention.

All right, well, mine too. I'm excited you're bringing it up today because, you know, I've got a lot of feelings and thoughts about this topic in general, and honestly, this acquisition. And so I'm excited to dive in. But it sounds like you have, you know, some real questions that we should, should unpack. So I'm curious about that. But before we do it, do you. Do you think I should give a bit of Conte and to help folks out?

Yeah, I think that makes sense. Just so everyone's on the same page before we dive in and geek out.

Okay, well, let me just. I'll talk from my experience and I was thinking, you know, I think my first encounter with MAP was or NWA was probably around 2011ish. If you remember, this is around the time of Summit's redesign. It's also a period of sort of explosive growth for charter schools. And as a result, there was just a bunch of school leaders and philanthropists who were looking for ways to see if innovations, and I would say both charter innovations and school design innovations were having an impact. And interestingly, this is also the time period when Common Core is sort of driving the full assessment conversation and what's happening in the country. And so at least where we were in California, there was a bit of a space or pause on those sort of big assessments while Common Core was coming into being. And so people were looking for things to fill the gaps during that time. Anyhow, folks started using the MAP assessments. And a fun fact, back in that time frame, I remember giving the assessment to our 10th graders for the very first time and realizing that only a little over 2,000 10th graders in the entire country had taken that particular particular test. I'm sure we'll get into this as we start talking, but the number of people taking the test actually matters a lot given how it scored and what this test does. And certainly there were higher numbers in earlier grades. But the point is, NWA has grown pretty explosively over the last decade, at least from my perspective. And so we fast forward to today and NWA is either a math or an LEA assessment. It's administered three times a year. And I think that's important fact by schools and teachers. It's adaptive in that it's on the computer and it, you know, sort of feeds kids the next right question based on where they, you know, how they did on the previous one. And its purpose really is to show school year growth for a student and to give teachers and schools, you know, quote, good information at the start of the year in the middle of the year about which students are kind of on track to meet the grade level standards or to pass the end of the year assessment. And so conceptually, teachers and schools can make grouping decisions or curricular decisions or tutoring support decisions for students based upon that information. But to be really clear, these aren't really formative assessments that are embedded in curriculum itself. It's not even tied to the curriculum. So they're not driving instructional decisions. And the absence of being connected to the curriculum. Well, I'll just say it. As the company says, they really seek to be content, agnostic, and so there is a real distance there between what you're doing in the classroom and these assessments. Let me just give one additional piece of context, and that comes from the perspective of a parent, because my child took these assessments three times a year for many, many years, and they don't mean anything to students or families. I don't, I can't sugarcoat it. Like, there is literally nothing that happens for students or families as a result of these tests. I mean, the reports and results are basically nonsensical if you're a student or a family. And there's just not any information in them that I can act on or use. And so in my experience, what happens is kids rightly start to ask the question of, like, why am I taking this test so often? What, what is this for? What's it doing? And you know, as a, as a school leader who was also that parent, I, I can tell you, I spent a lot of my time trying to convince my own child and get him to convince his friends to do their very best on the test because it mattered for the school. And in reality it did because we were using those results for everything from charter renewal hearings to grant applications. And so I'm going to stop there because that's a lot. And you haven't even introduced what you want to talk about yet, Michael. But as you can see, this is a juicy topic and I'm getting fired up already, so. So I'm super curious to hear what it's provoked for you.

Well, I love this, Diane. I think it's super useful context and I love the passion and energy. I thought I would tap into it on your end. And so we have. But, you know, assessments more generally. Right. Is something that I, as an outside the classroom, as a non educator, have really had to come up to speed over the last 15 years to better understand how different kinds of assessments can or can't be useful, what their function is, what they are capable or incapable of. Telling you, and I will say I feel like I'm constant work in progress on this, which is why I'm excited to check with you, but it is not really obvious to folks outside of the classrooms and schools. Oftentimes I think we. What an assessment says, and as a policymaker or someone like that making law about this stuff, it is very opaque, I suspect. So that's just further context. But as I look at this acquisition, one of the stated reasons I think for it, and this is from the companies, not from me, but from the companies, I think the purpose of this assessment is that, look, you have this established assessment system that is used in lots of schools and districts across the country and, and HMH now will basically be able to use those assessments to connect them with their curriculum and make personalized recommendations for each student. The theory I'm, I'm assuming that they have at work is that they'll be able to sort of do what you suggested or implied at least would be helpful, which is that assessment should be used to actually be embedded in the curriculum and drive learning choices themselves and in turn help bolster more personalized learning opportunities. Now, to be clear, they've been, they've been clear about this. Each set of products will still be sold separately and under their own brands. But I think a major rationale that they're making is that the possibility now exists to combine these, to combine the assessments with the curriculum and buy them as a package, if you will.

I have to admit, Michael, I'm, I'm deeply skeptical. I'm really curious about the theory that suggests this is going to be sort of a happy and product coming together of these two very disparate things.

Yeah, well, look, this is the theory or the theoretical reason, right, for the rationale behind the acquisition and, and why I assume as a nonprofit, NWA probably said, look, this is for the good of our mission for the education sector. I imagine that's what they're saying anyway. But for the moment, I kind of want to put aside one of the structural things that you mentioned that causes your skepticism, which is that these assessments aren't at present anyway, actually embedded in any curriculum or tied to any specific curriculum or given in an on demand way to drive student learning. We're definitely going to circle back to that, but I want to instead introduce a totally different theory on my end to analyze this move and its impact. And it's called the theory of interdependence and modularity. And it's one that we used in disrupting class as well as My most recent book, From Reopen to Re and even in Parts of Blended. And it's a theory that sort of combines engineering with business is the way I would think about it. And as a refresher for those that don't know and don't geek out on the stuff routinely, the first half of the theory essentially says that when something is underperforming what customers or users need from it, and the way that two elements in the system interact are unpredictably interdependent, meaning the way one works and functions is dependent on the other one, the way it works and functions, and vice versa, then to make it good enough so that people will actually adopt it and use it, you have to do it in a proprietary structure and really have a proprietary design. And that's because the two sides really need to be designed interactively. So, quick example from business and then I'd love you to sort of reflect on what that means for education. But way back when, in the 1940s and 50s, when IBM came out with their mainframe computers, no one knew what a mainframe computer should look like or operate or anything. And so IBM just couldn't exist as a standalone assembler of computers or even a seller. They had to make every single part inside the logic circuitry, the core memory, the operating system, assemble it, sell it, everything. Because each of those stages interacted in unpredictable ways with every other part of the process. And so if they had just thrown the operating system sort of over the wall, if you will, to engineers in the side assembling it, well, that would have impacted in unpredictable ways the performance of the fundamental computer. And each side would have had to make unacceptable trade offs that just wouldn't have been good enough and no one would have bought it, basically.

Okay, let me make sure I'm getting this before we go on. Let me try to bring it into this context and see if I'm understanding it correctly. So in this case we have curriculum, so textbooks, you know, lessons like all of that and a benchmark assessment. And they're separate, they're in separate organizations right now, but, but they're interdependent because when you're educating a child, you're, you're both assessing them and you're doing this curriculum, which is kind of confusing because curriculum has assessment in it as well. But we'll leave that aside for a moment. They're, they're interdependent, but, but they actually depend upon each other. But the key right now is it's really unpredictable how that happens because, well, I guess I'll just describe from my experience sometimes like it often felt like when we were giving NWA that the results of that were totally disconnected and unpredictable based on what we were seeing in classrooms and our knowledge of students and things like that. Is that. Am I getting it?

Yeah, I think that's exactly right. That there is a. Certainly a dependence and in some systems it's a well understood dependence. It's predictable, we can get into what that means later. But in many systems it's unpredictable. Right. You make a change to the curriculum that should impact the assessments because otherwise you're just not going to be getting reliable information and feedback on what a student actually has learned or knows now and can do with that information, and vice versa. Right. If you make changes to the assessments, well, that should mandate changes to the curriculum. And when they're at arm's length, they are not really feeding off each other. And when you think about all the personalization that we want to see in the world because we know students have different learning needs at different times, those interact in very unpredictable ways right now, particularly as you get in, like think about math or something like that, and the sequence of learning and the dependence of different topics on other topics, et cetera, et cetera, these things weave together in very interdependent ways. They, they impact each other. And so my sense is what you just said, which is that when you have the state of curriculum and assessment at arm's length separated from each other, they're basically underperforming what schools need. Certainly Title 1 schools. I think you could make the argument in Lexington Public Schools, where I am, that's not the case. But in Title 1 schools in particular, those that serve lower income populations, for example, they really need these to be connected to help drive choices for the students on the ground themselves. And I think the evidence for this, Diane, actually they don't just have to take my word or maybe your word for this. It's actually in the market itself. We've seen incredible traction and extraordinary growth of a different company over the past 15 years, which is this company called Curriculum Associates. And they make both a curriculum, the Iready curriculum, as well as the Iready assessments. And they use them as benchmark assessments. But they're intricately tied to each other and they essentially in a closed loop system, what you get on the assessment system determines, you know, what you do next in the curriculum and vice versa. And interestingly enough, Curriculum Associates over the last 15 years has basically come from essentially nowhere to become one of the digital Players that I would argue is disrupting much of the textbook market over that period of time. And I think it's driven by the fact that, that their assessments and curriculum have been designed in an interdependent fashion. And I think there's further evidence for that in the struggles, at least relatively speaking, of a company called Renaissance Learning. And Renaissance, for those that don't know, basically has historically offered the STAR assessments, another benchmark assessment instrument. And they've been sort of stuck, at least my senses, with assessments where they've been trying to sort of find partners, Diane, in the curricular space that they can sort of align to, if you will, to provide information around. Around. But it's been kind of kludgy because you're integrating these curriculum that haven't been built with the assessments in mind and vice versa. And you have a curriculum agnostic assessment. And so they've acquired or partnered with providers like nearpod or more to essentially like, have both sides of the equation. But because they weren't designed interactively, the feedback loop between them, or at least the theory would suggest, and full disclosure, right. Like, I can't make the judgments because I don't use them. But my sense is that the theory suggests that the recommendations and integration of the two just wouldn't be near as eloquent, if you will, as what Curriculum Associates offers because of the way they've built both sides of those items. And so to be clear, that means in a not good enough world where these things are underperforming, that favors interdependence.

Fascinating. What you're sharing is bringing up two things for me. And so I want to just keep checking my understanding of this with you. The first one is it seems to me that maybe another example of this is the partnership between the College Board and Khan Academy. You know, as folks began sort of pushing back on the SAT test as sort of these standalone assessments that are super high stake, and they just felt like they're disconnected and inequitable. The College Board partnered with, you know, Khan Academy to offer sort of the learning curriculum part of the equation. The benefit there was, it's free, it's to anyone who wants it, you know, bringing the equity piece. Is that, is that a good example of this theory?

Yeah, and I think we'll get into more how it is in a moment, but it's sort of. Right. It's this modular, trying to build up into an integrated way, I think. Yeah.

Okay. The. The second thing that comes up to me, which probably you're like, of course it comes up to you. It's just what we did at Summit Learning. And you know, it seems as you're describing the theory that it, this is what we sort of did intuitively. You know, we, we built as you know, an entire system of learning and assessment to be interdependent and integrated from the very start. And we, we just thought it was the only thing that made sense. And I know we're not alone in this, this, in this realm, but it's, there's a reason that a lot of people don't do it. It's, there's a whole bunch of reasons. That's probably a whole other episode. But it seems you're, you're offering evidence for those instincts. But I'm wondering about the modularity part of it because we haven't talked about that yet. And that how does, so how does that fit in?

Yeah, I'm so, I'm glad you asked because there's a trade off in this and, and there's not like one right place to be in this continuum. And because when you're optimizing for raw performance by creating a really proprietary interdependent system, the trade off is that you can't get affordable customization. And this is a really important point and it's interesting actually, because Summit Learning, you optimized for curricular modularity to create an interdependent system around parts that were underperforming. And so you made these sort of trade offs. And that's the other piece of this, is that in practice every system has some modular components and some interdependent ones. And you basically want to be interdependent where the performance isn't good enough and modular where you're overshooting so you can get that customization. And so in essence, what the theory says is that when you start to overshoot, that's where you need that customization, because people demand it. They want affordable customization. They want to be able to mix and match different parts and just pop them in. And to do that you need modularity. So the quick example again we'll go to computers is, you know, Dell computers say in the early 1990s, if folks remember, you know, Apple, IBM, they had sort of started to overserve what people needed. They weren't willing to pay for some of the improvements in that paradigm. And here comes Dell and they don't make any of the parts inside of the computer. And that means you can just jump on their website and be like, I'd like this much memory. I'd like this kind of drive from Seagate, I'd like this kind of monitor, et cetera, et cetera. And they fit together in very well understood ways. And so Dell just quickly snaps the parts together and 24 hours later you got shipped out an affordable customized computer. And so I think you at Summit Learning know these desires very well because you know, you chose to avoid purchasing on the open market a proprietary, highly interdependent curriculum and instead, as I said earlier, create something far more modular on that side because you wanted customized playlists and so forth for the students. And interestingly enough, I'll try to make the argument for you. I think you, you, you said that we're, we're, you know, the systems out there are really underperforming is in the integration or the interdependence, if you will, of the knowledge, skills and habits of success that are often broken apart and perhaps unfortunately so. And you said that's where we really need the interdependence in the Summit Learning system. Yeah, go ahead.

I think I, it's all just clicking for me as you're talking. We've never wanted or bought a textbook because it's, it's so not customizable. And if you've ever been a teacher or know a teacher, you know that is the number one thing a teacher wants to do and needs to do, which is why those products don't get used with fidelity. And so we're just, just, I think as practitioners, so keenly aware of what actually happens on the ground and like pragmatic about that, that that's where we wanted customization, but super tight on the, the assessment part of it and the connection to it. And like you said, the connection of the, all the factors of learning, not just, you know, a single siloed content area, if you will.

Yeah, and I think that's the big aha here, right? Which is that you can't actually just jump into a modular design. Like everyone wants customization. It sounds great, but you can only move to it once the interfaces between the different components are predictable, specifiable and verifiable. And I hear this all the time, Diane, in education circles that we want an open system, we want an open this, we want an open that, as though open in and of itself is a good thing. And the only way, though, you can move to that kind of modularity is you really have to understand how each side works and functions and the impact of changes on one side with what it'll do to the other. And that means you have to be at the level of predictable causality. You need to be able to create specs, in essence for exactly how those parts fit together at the interfaces. You need to be able to verify that the parts in fact meet the specs. And then in the early stages of creating something new when it underperforms, that's just really hard to do. And it's, it's only when you get better and better and better, you start to overserve people and you start to understand how these parts work. Then you can start to shift to modularity because you have a better sense of how the overall system is interacting. And the science of studying them essentially allows you to get more predictable. So a decade hence, maybe we could start to modularize summit learning, but we're not there yet.

I'm having a reaction that I commonly have when we are talking putting theory to practice. And often, you know, in my mind, I'm always putting it to our practice. And at first when you are describing the theory, my mind immediately goes to, oh no, we didn't do that. We need to go back and do that. Oh no, we messed that up. We weren't good innovators. And then it's funny, as we talk more and more and I really like calm myself down and think about it, I realized that a lot of what you're saying, Michael, we do or did, and you know that the theory really describes our behaviors, which I think is good. Because you don't want a theory that's actually not describing what's happening. Right. And I'll be honest, the outcome of that work is strong. By designing a learning and assessment system from scratch that clearly prioritizes each and every student and incorporates the learning science and the full understanding of the needs and leverages technology, you know, we really have created what I think is a very elegant and interdependent curriculum and assessment system that is highly modularized in the right places and thus customizable for the folks on the ground, both students and teachers, I would argue, and schools for that matter. And as I say that, I'm cognizant that it sounds a little bit like bragging. But the point that I want to make is that it's possible that what you and I are talking about, it's not a fantasy, it's not a future thing. It literally is happening right now in hundreds of schools across the country. And I just think that's important because we always want to take this sort of third way, future looking perspective. And what I don't want this episode to sound like is just us criticizing a deal that happened without another solution. Because I do think there are other solutions and it doesn't feel like we've compromised in the combined offering. And honestly, this acquisition feels like a compromise to me. I suspect that a lot of people who care about seeing big rolled up data and tracking and reporting on the percentage of kids in schools who are supposedly learning or not, they might be happy with this direction. MICHAEL but for those of us who care about literally every single child and if they can read or think or solve problems, and if every child's motivated to come to school and knows how to learn, for those of us who think it's unacceptable that school systems are unable to simply tell families if their child's on track to be a skilled adult in areas that families care about and their life goals, and instead we send them these incomprehensible test scores in the mail months after a child took the test. Well, look, I'm going to stop there because you could see I'm getting fired up again. But, like, this is possible what, what we're talking about, the theory says we should be doing. And I don't think that this deal represents that.

Yeah. And this is where I ultimately want to go as well, DIANE and I think there's another issue as well underlying this, which is it gets the multiple purposes that assessment serves, right, from driving learning to continuous improvement to accountability and transparency. And I think what's also interesting on this is that the interdependent approach, frankly, it's always going to struggle to serve a public accountability function in my mind, in which we're making judgments about the schools themselves, even as the theory shows why it's the best way to optimize the teaching and learning. And I'll say that again with to analyze curriculum associates, for example, and with immense respect for the predictability of I ready assessments on what a student will score in a summative year end assessment. But essentially, Diane, if the I ready assessments, like, I suspect some people are listening to us or just like, like, well, why wouldn't we just use the I ready assessments to replace the summative year end assessments? Their curricular tied. Great. End of story. But you can imagine that then there would be a really slippery slope that would come into play in which curriculum associates and again, I'm not saying that they do this nor that these incentives exist right now, but in this alternative system I just described, the incentive I think would exist for them to basically just just give rosy assessments of student learning to show, hey, look, Our iready curriculum is really making a positive difference for students because there'd be no check on the other end of it. And then if they made those changes, which are no, no longer reflective of actual performance now, that means that the assessments would no longer be as useful for guiding learning for students and teachers. There's a big trust issue here.

In other words, Michael, I experienced this in real life where very few people actually take, take seriously the assessments built into Summit Learning as an outside, valid, you know, way of measuring what we're doing. I argue that they should because they can be completely valid and reliable and all of the things that we need. But. But there's very little pickup there. And I suspect what's underneath it is this trust issue that you're surfacing right now.

Yeah, well, so I think this is where there's another approach from the theory which works frankly though, only in a world of mastery learning that we've discussed in the past in the show, in which students work toward mastery of each and every competency and time becomes the variable, not the students learning as the precedent system holds. But that is, if you have the interfaces clearly specified between curriculum and assessment, and that's a big if, then you could have modular assessments created by a third party, so not the curriculum company itself that an individual school or a district or yes, even a state like, you know, as Louisiana has, could select based on its alignment to their curriculum. And in that world, those assessments could both serve to inform instruction. They're learning informative, but they're also summative because the measures they're giving are inherently of learning, which is what your system does it just to your point, it's not third party, so people don't trust it it. And the point being in that world, if I demonstrate mastery, I move on and that can be reflected. And it's more robust than a summative assessment because I can demonstrate it when I as a learner am ready to raise my hand and show mastery. And so to come back to nwea, this is the curveball it seems to me that HMH is walking into here. And you've alluded to it several times. But just to say it from the theory perspective, on the one hand, by controlling the assessments, they theoretically, theoretically different from theory, will have the opportunity to make them in a proprietary fashion alongside their curriculum and benefit from, you know, the same way Curriculum Associates has. But to do that, they have to do it from, you know, by starting with assessments that are not curriculum aligned. That's going to be a major Lift. And on the other hand, HMH can't exactly serve the modular use case that I just outlined because it doesn't have an array of assessments aligned to lots of different curricula that can be, you know, customized where the individual school or district or state or charter organization can select to drive learning. And, and it now owns both sides of the equation. So it's sort of caught here, Diane. Now, I think there's some, you know, silver lining here as well. There's New Meridian, for example, a nonprofit. It exists as a newer third party provider seeking to offer modular baskets of assessments that you can pull off to match your curriculum. But I'm just not sure that HMH can make this jump as we've outlined. So it's possible, Diane, that it'll be commercially successful as a sort of in between solution, I guess in the market as, you know, sort of the market maybe transitions over time, but I'm frankly not sure that that's where the puck is going or frankly where it should go. And I really want to know how that lands on the ground for you.

Well, you probably have guessed that the idea of a commercially successful in between solution isn't landing very well with me. This is the thing that just really frustrates me about our sector and our work. I, I agree with your assessment, Michael, and I, I think this is another example of what holds us back in education at the broader scale. You know, we've got these two players who have a massive market share together. Well, I would call it massive, you might correct that, but it seems pretty big. And, and you know, schools have to have curriculum and assessments and they're under pressure to, to have something that tells them exactly where students are, especially coming out of COVID so that they can help them direct resources to them. And just the expectations on teachers in schools continue to go through the roof around what they are expected to do. And so tons of schools are going to use this joint offering and in their mind they're going to be told and sold on the idea that it's better. Somehow it's gonna suck for kids. I don't know how else to say it. It's not better, it's not good and it's not what our kids deserve. And what's possible and what's frustrating about that is it pulls the vision, the mind share, the resources, the energy away from what doesn't suck, which is what we were describing before. Assessments that are embedded in project based curriculum that's authentically assessed, assessed in a way that dramatically improves engagement and learning and self direction for students and job satisfaction and impact for teachers and gives them local control and all of the things that we care about, that we know matter and we have the capability to do that today, do something significantly more meaningful. And I just think that this pulls us away from that work. It pulls everything away from that work. And I think it goes back to the conversation we had last episode, Michael, when we acknowledged that most people don't even think about students as customers. And in my mind, this approach sort of epitomizes that. As I read the transcript of the interview with the two CEOs, there was literally not a single mention of this acquisition in terms of how it was valuable to students. They got down to teachers and said teachers would have all this, you know, data and information. Of course, the way I read that was like, wow, teachers have now a new expanded job and having had that information, know how hard it is to actually work with it. So yeah, I'm going to stop again because. But that, that's how it's landing with me.

Yeah, well, I. No, I think that makes sense, but something you just said in terms of, of all the information we're going to get and all this stuff and so forth. Let me step out of my innovator hat on that for a moment and talk about one other problem, in my mind, at least with NWEA assessments more broadly through an explicitly education lens. And this surrounds how so many of us want to shift to measuring student growth instead of point in time learning. Right. But the reality is that there are different ways to measure growth. And the public and educators, I think, honestly, broadly speaking, don't understand the difference. So when they see a growth measure, they don't actually know what it's telling them. And in this case, NWEA reports a norm reference growth measure as opposed to what would be called a criterion reference measure. Essentially what that means is a norm reference basically evaluates you relative to other students like you at your percentile, quote unquote, of your learning. Whereas a criterion reference is against that yardstick of curriculum. But it has to be curricular aligned to be able to do that. Now, as a result of that, it frankly, the NWEA growth measure offers this incredibly false illusion where it basically, if I'm say a student in the fifth percentile of seventh grade or whatever else, then basically it's going to compare my growth relative to other students, students in that same segment. And so I might grow one and a half years relative to those students. But that's embedding all the lowered Expectations and malaise of our education system in that growth measure. That's not what, when, when a parent hears one and a half years of growth, that's not what they're thinking. Like, they're, they're thinking, oh, my kid grew from, say, the third grade to well into the fourth grade. Right. That's what policymakers, I think here. That's what I think. Educators and curriculum companies that use this stuff, like, I think that's what they're hearing too, and it's just not what it's measuring. Plus, I think this gets back to what you said at the very beginning. And, and maybe we'll. This will start to wrap a bow around this. You mentioned that when you gave it to your 10th graders, only 2,000 kids in the country had used that. So now it's driving a growth number relative to other students like them. I'm not going to be able to do the math that quickly in my head, but, like, 100 percentile points, whatever. It's not a lot of kids in each percentile. You're driving a growth measure off of that. That's even less meaningful, I think. Diane.

Well, you're, you're, you're opening the can of worms. And I know we, at this point, we should be closing out, so I'll, I'll be brief here, but I distinctly remember that conversation about those 2,000 kids. It stands in my mind a decade later because it was so clear to me, not initially, but when we really, really, really dug in to understand it, when, you know, the company rightfully said, well, there really aren't kids, like a lot of your kids. So, you know, you kind of have to take these results with a grain of salt because there's only 2,000 kids. And I mean, Michael, we could do days on this. But, like, let's just start with the concept of, I mean, you've reinforced why these results are meaningless to parents and students, because how can anyone in their right mind understand what you just said? When you're looking at a result or expect that that's what you're getting, you would just want something straightforward, right? Like, do I know this? Do I not? Am I good at this? Am I not not like. Yeah. Anyway, but the second piece is this concept of like, bias. So we're gonna just compare you to people who are like you, who at that moment in time, if you're a, quote, lower performing student, what are the expectations? We know what our systems have in terms of expectations. We know what natural human inclination is. I just think this is so destructive. And I will just share one last story about this. Part of those conversations with my son around this were so disturbing because he would come home and say the kids would just share their raw score with each other and they would just stack rank the scores. And what happened every time is, quote, the smart kid, the super smart kid always got the top score. So in my world where I'm caring about growth mindset and like kids believing that hard work actually pays off, I was having to fight against what these results, the messages we're sharing with kids, which was much more of a fixed mindset than a growth mindset. And we were doing battle against that in the context of the school because we were giving these tests, which I see the look on your face and it just is like, it's icky. There's no other way around.

It's really crummy. It's really crummy. All right, well, let's not leave it there because I think there is hope on the horizon. And maybe, Diane, frankly, as we see this acquisition, maybe it represents consolidation. And often when you see consolidation in markets, it creates room for the disruptive plays underneath to gain market share. And so I'm hopeful that we will move to content aligned assessments and have some third party players that working hard to align to the different curriculums out there so you can select the right ones for you and move to this world where it's both driving learning but also giving parents and students the information that they want about their growth. So I'm hopeful we'll get there. But I don't think this acquisition is

the step, a perfect way place to leave it, Michael. And before we sign off, I would love to hear what you're reading, listening to, thinking about outside of our, our day to day. What, what, what? Add to my list.

Yeah, you bet. Well, so I. Sal Khan sends a book every few years, it seems, to folks who are on his advisory board. And so I got the one in the mail from him, which is the Art of Living, the classical manual on Virtue, Happiness and effectiveness by Epictetus. And the interpretation that he sent is by Sharon LaBelle. And I read it sort of, you know, chunked like a few lessons at a time over the last several weeks. And it was really helpful, Diane, in just resetting my own sense of how I think about my own life lived right and what we can control and focusing on that. And so I really enjoyed it and was appreciative that Sal made the time to send it. What's on yours?

Well, that's Awesome. I appreciate that. I have a fun one this week. So, you know, as we're, we're about to head to India, folks have been hearing about that for a long time. We watched a movie that's on Netflix. It's called rrr. It's getting some, you know, sort of positive press in the US it is a Bollywood movie out of India. The hard part is it's three hours. But I will say it is a fun three hours. It was joyful and interesting. It's packed with all sorts of over the top action and it is over the top in many ways, but we deeply enjoyed it. And I think on the sort of interesting part of it, potentially quite controversial, but it really is sort of offering a different narrative about, you know, the drive in India to be their own independent nation and how they engaged and interfaced with the British. And, you know, the normal narrative we all have is surrounded by Gandhi and a peaceful approach. And there are some folks who want to have a more fight, struggle narrative. And so that is presented here for better or worse.

Super interesting. That's a major political current right there. I know. And so I can't wait to hear what you learn from it as you come back if you choose to report on that. But for now, we'll leave it there. Thanks for engaging with me on this conversation and for all of you listening, we'll see you next time on Class Disrupted.