How to know if an edtech product supports different learners' needs

Welcome, everyone, to the Future of Education. Michael Horne, joined by longtime friend in the education space, Vic Vucic. Vic, we've known each other for like 11, 12 years, something like that. Is that right?

I think so. I think it was when I was back at the Hewlett Foundation. I think you came by to pitch us on getting the Christensen Institute up off the ground back in the early days.

Yeah, that's probably. Probably right. Yeah, that probably is right. We probably should be doing more of that these days.

But.

Now you're in a new post, obviously from when we first met. You've been leading the Learner Variability project at Digital Promise Global for several years now. And why don't you tell everyone actually who's tuning in first what the Learner Variability project is, and then I can make the case for why it's so important.

Excellent. Perfect. Yeah, it's now I actually just crossed five years at Digital Promise, which is really exciting. When I joined, we were 25 people, and now we're about 115 or so. So it's been quite a ride. And basically the bulk of my work has been getting the Learner Variability Project designed and off the ground. And it's a really large initiative that's focused on first, I guess, bringing on this concept that it ends up, if you look at especially Todd Rose's book the End of Average and some of his work, we build off of that, where there's no such thing as someone who's average at everything. We all have variability, and we all have variability on many different dimensions. And when it to learning, we all vary in how we actually learn best. So it's not just about what we know or don't know, but it's actually about how we learn. And so at that time, especially five years ago, you know, personalized learning and technology was starting to come out to enable personalization. And teachers are trying to differentiate and the diversity of learners in our school system has grown dramatically and really kind of diffused. It's everywhere. And so I would often ask the question, well, what exactly are we personalizing and based on what research? And too often the room would go fairly quiet or they just say, well, we're just trying to personalize. Whatever we can, we can do pace, that's a first step. And then we do fonts and something else. And the thing is, there's a tremendous body of learning science research, huge on how kids learn. And so what we've been doing is we've been working really hard in distilling and synthesizing this Research in huge ways with a whole child or whole learner lens and creating what we call learner models, which are the synthesis that map what are the big factors that learners vary on that we have evidence has an impact on their learning outcomes. And then what are strategies that have research behind them to support those various factors. And so we have these maps where you can really see what is the variability that really impacts how kids learn. And then we have this whole database of hundreds of both product design and instructional strategies to support that variability. And it's all free and open source on our website and it's in our Learner variability Navigator is the name of the tool. It's Creative Commons. And then we're now getting that out in the field. We're working with major edtech products, everyone, newsela, Lexia, Reedworks, ST Math, et cetera, to embed this in their product design processes so that they're basing their decisions on research as much as possible and bringing this lens of supporting all learners. And then now we're working more with districts and practitioners to bring this into the classroom to help teachers with strategies to support that diversity that's, that's in their classroom every day.

Beautiful. So I'll throw up the learner nav. The. This is, I think the navigation site, right? So.

Yep, that's right.

Perfect. I got the right one. So people can go there, check it out, start to use these tools in the frameworks. I find it personally incredibly important right now because there's been so much hype over the years around differences in learners. Like for a long time people said, well, learning styles must be the differences that matter, right? Some people are visual learners, some people learn by hearing, some people by I guess, playing things out or something like that. There's a variety of theories on that. And when you look at the actual research, while we might have these preferences in any given content area, it turns out it's not the best way to teach someone or to personalize the learning. And so you all have really gone into the research below the averages, as you said, and pulled out these strands of, okay, like this group of learners or the dominant group of learners, like this traditional phonics based approach might make the most sense. But like, you know, if you have this set of characteristics with, with working memory capacity and perhaps English language as a second, as a second language in the home, when you hit this skill like this is actually the ways you ought to consider tackling it. That's maybe different from say the 60% of students in the classroom and Things of that nature, which for software companies, building product, I think is incredibly important for the teachers and parents trying to navigate this world, I'd also say it's incredibly important because otherwise you're just going to keep banging your head against the wall if you try to just do best practices and not realize it depends on the learner in front of you a little bit. Is that right?

That's exactly right. And kind of to your point. First, why it's important to base on research. Yeah. Learning styles is the most common personalization strategy I see out there, and yet there's no empirical evidence that they exist. That said, I will say, don't throw the baby out with the bathwater. It doesn't mean that rich multimodal instruction isn't great. It just seems great for everybody. Right. But we do know working memory. Right. It's a factor that learners vary on and we have strategies that can support that. And we know emotional regulation is another thing that kids really vary on and impacts their learning. And we have strategies to help with that. Right. So what are these hotspots that you need to understand and want to understand? And the big thing too is when you. Look, we released a white paper, I think it's four years ago now. We've updated some of the data, but I mean, we just took a couple big categories. When you take kids that have experienced trauma within the last two years, English language learners, kids with learning differences, sort of the gifted and talented, and a couple other. If you take five or six categories, you're talking over 50% of kids. Like, even if you're taking the most conservative estimates of overlap, like this is the core student body. And this is part of the argument, especially for product developers. But even more broadly, first, this isn't a 10% remediation group that you can just pull off on the side and deal with. That's not gonna work. Right. You have to design for this in the core. And the second, from a market opportunity, this is the bulk of the users that are gonna be using your products. Right. And so we can't just start from the average. And the trap in product design, because I used to be a product designer, is that you want to move on to the next thing, the next thing, the next thing. And you never actually broaden out for everybody in the process because of the pressure to release the next thing. And so how do we make it where it's easier to say, yes, we're designing this, but we're doing it intentionally and putting in these supports for all these learners so that when more learners come to their products, more of them are likely to succeed and show that learning growth that we all hope happens.

So great set of work, great foundation you all, if nothing else, you've categorized the world in ways that are incredibly helpful, I think. Also starting to make it more and more actionable because it's really complex and so trying to break it down for folks. You are now breaking new news, right? Like you've come out with a certification process to help educators and parents navigate the world of all the EdTech software out there, so to speak, to help make sense of who's actually taking this research and embedding it in their product. Can you talk about this new certification that you've come out with?

That's right. So we're really excited. This is kind of our long term vision when we kicked off five years ago was let's build the foundation of knowledge and understanding, based it on research with experts from across the country and even the world. Then let's make it actionable and then let's try and create some incentives and signals that really make people hopefully pay attention to this and create currency on this in the, in the market, in the field and help people understand what, how products are designed and what are they designed for. And so part of that was launching this product certification and product certification process. And so what we did, I'll just take our first certification was research based design. So basically does your product, was there any research basis to sort of the prime hypothesis of your product and how it was designed? And it's relatively straightforward where what we ask for products to get certified, they need to share what is sort of the core theory of change or logic model, if you will, of your product, how is it going to make that learning growth happen? And then what are at least three peer reviewed research studies that undergird that and that that's based on. And then we ask how does that research show up in your product design itself? So what have you designed in based on the insights from that research? And then the last piece is, have you shared this publicly on your website? We're trying to create some transparency and also that allows us as researchers in the field to understand where products are. And so if you go through that process and do that, then you can get this research based product design certification. And then the new one that we just released is the learner variability product certification. And this one, this one's a little more nuanced, but we're super excited about it. And it's basically have you designed in at least, we put the threshold for now at six, at least six major features, functions, tools or experiences that explicitly support learner variability. And we have a definition of learner variability in your product. Who were those intended to design? How are they intended to support learner variability? Were teachers involved at all in the design process? And again, have you shared this publicly anywhere? And actually there's one other criteria which is are at least two of the features sort of, I would say activatable maybe, or actionable by the learner themselves. Right. Because our research shows when you empower learners to actually use these, they use them quite a bit and it actually increases the usage of these features and the impact of them. And I will explain one thing a little in depth. Our learning variability framework has four columns. The first is kind of the content domain, so like 436 math or PK3 literacy. The second is what we call domain general cognitive factors. Those tie to what a lot of people call executive function, things like working memory, attention, sensory processing. Then there's social emotional learning, which is the third column. And then the fourth column is background, not the student background. So things like poverty and social supports and trauma sleep. Right. Things that are just in their background that impacts how they learn. And we have a lot of research behind the learner variability. Certification actually only focuses on the last three columns because so many, the vast majority of edtech products just focus on the first column. The academics, that's what's easy and that's where they personalize. Right. But this brings this whole interesting, whole learner lens and to be honest, it nudges product developers. And if you want to get certification, it makes them actually address these other things that there's clear evidence have huge impact on learning, particularly for underserved populations. But in reality for all populations.

That's super interesting. And are you seeing, I guess the first question is, are you seeing stickiness in the marketplace where teachers and districts and school leaders are starting to look for the certification and say if you don't have this right, then like we're going to ask a bunch more questions at minimum, maybe at maximum, like it's actually starting to write it into the RFP processes as they select content and stuff like that?

Yeah. So I think that's kind of the vision. I think we're seeing early signs of that. These are both relatively new, but I will share a couple of. When we've worked with, we work with superintendents, our League of innovative schools, over 114 superintendents, and then other folks. So first Thing that's really interesting is a year ago we did a national survey around learner variability and education technology. And one of the things that came out was the top reason that teachers select ed tech is to help support differentiation. That was the number one reason by far. Right. And so, and to support learner variability. So when teachers go to look to choose education technology, that's what they're looking for, support. So we were kind of surprised at how high that was there then I think the next thing is that when you look at the learner variability certification and supporting that and creating that signal, what we've heard when we talk to people is like, this really helps me with what they said with cya. Right. Cover your ass. So it's basically my board's going to ask is this research based? Or my boss is going to ask is this research based? And this just lets me take that off the table, be like, yep, here you go. You know, there is a research basis behind it or does this support our learners? Or how does this design to support our learners? And they can say, yeah, it has a learner variability certification. So we've heard that it's starting to provide that sort of confidence in selecting certain products over others and making it easier for them to do that. And so, and we're going to build out more of a campaign over the next year to really start to get this out there. We've been focused on the supply side, so over 50 products have now gotten certified on either one or the other. Yeah. So we're getting good momentum on the supply side. It's going and it's everything from creative innovative startups all the way to really major market players. You know, Newsela, I think Amplify and some others have gotten. So it's been a nice breadth and really interesting to see starting to get off the ground. That's terrific.

So one other question going down this line, which is are you going to get. Do you envision a world in which you might say this particular software, say Lexio learning, for example, really good at supporting learners, learning how to read if you have this type of variability profiles. Right. And maybe it's a few of them, but they don't do it all. And frankly to your point earlier, that's okay because as product developers like they have to focus on other avid axes of improvement. But we do it from as opposed from a deficit approach, we do it from a positive one to say like this is what they do well. And if you have a learner with something else going on in the Background. Here's a different approach. Right. And here's other software that fills in that. Do you envision it going down that design or do you think it stays much more. You're trying to push a universal design for learning approach for all and everyone should be comprehensive, I guess, within the content or subject domains in which they're working. How are you thinking about that?

Yeah, it's a really good question and I do think, I would almost say it's a little bit of a blend of both. So when supporting learner variability udl, many of the strategies are UDL strategies. Right. And a lot of strategies in our navigator are UDL strategies and those are really extremely strong and research based. That said, there is an intentionality of design that's really important. And as you get really deeper and deeper, there are specific things for specific learners. You know, Carol Connor showed in her research, certain kids do better with a lot stronger deeper phonics at this point versus whole language. But then other learners do much better with the whole language but less on the, on the phonics. So when you get into the nitty gritty, there is that differentiation that has a lot of power. So I do think, you know, the, the dream is that, yeah, products and this is part of the learner variability certification. We do ask intentionally, who did you design this for? What were the supports for? And how will learners, specific learners experience this? And so it is really about that intentionality. And like you said, it's not necessarily that every product is going to support every learner everywhere. To me, that's not going to end up being the solution. And so being able to map who did you intend this for? Who did you base, you know, look at the research and design for. Have you been transparent about that? You know, in an ideal world, yes. I mean the question, the challenge is will an ed tech product ever admit that their product sure. Necessarily work for a certain group of kids? You know, that's. Yeah, that's, that's tough. But look at medicine. Like you have to like, I can't say this medicine, this medicine doesn't work for a group. I gotta say, this is not for you, you know. Yep, yep. And, and here's an issue that really popped up and has been kind of. I had a bit of an aha. And this is a little bit from coming from the broader technology field in the consumer tech world. And it is that if a product doesn't work for a child or a learner, I should say learner because we just launched our adult learner model Which I'm so excited about. Oh, that's cool. We got an incredible quote from one of our research advisors that this is a first of its kind whole learner synthesis for adult learning. And so I have to say whole learners. But what I was gonna say is that. Sorry, I just lost my train of thought and excitement.

No, no, no, no, no. No worries at all. No worries at all. We could buy time where I can ask you a question from Jim Vanities that just came in.

Oh, nice. Hey Jim.

I'll ask you the question from Jim.

Go for that.

And then when you. And then when you come back, you know, I'm totally with you. I do that all the time these days. So Jim says first he gives you kudos for creating a framework to encourage better edtech products. Question he's asking is, does Digital Promise then offer a customized coaching guidance to product teams developing new ed tech solutions?

Yeah, it's a great question. Thanks, Jim. And we have to grab coffee soon. Outside at a distance.

Socially distanced, of course. But you are both in the Bay Area.

Yes, that's right. So I will say yes. So this is also interesting. So we have support, luckily generous funders, the Czi Oak foundation overdeck all really supported the core of this. So we are able to provide coaching and some early supports. Obviously different levels. We can do it for free for a lot of it. You know, we're happy to do webinars or video sessions for teams to kind of take them through the tools and the framework and how to use this and do light support here and there. We do a few products. We actually do dive deeper and we've even done research studies with actually we did a really great research study. It's up on our website with Reedworks. Over 1800 kids at Talladega School District in Alabama. Really wonderful results. And so occasionally there's that, but we don't have huge capacity for that. But yes, we help set up and coach in the beginning and then occasionally we can do a check in or something and then certainly, certainly down the line we may build out those more robust supports for people that want to go deeper. I will say one thing to also be clear, since we're talking about product certification, one thing we want to be really careful of is not to be seen as kind of self dealing. Just because you work with us and just because you use the learner Variability Navigator doesn't mean you automatically get certified. Our certification evaluation process is completely firewalled and separate and those are different people and that gets evaluated differently. So you know, we hope it supports and helps, you know, move you in that direction in general, but it's not like, oh, here, you know, pay to work with us and then we'll certify you. Right, perfect. So we don't do that.

That's helpful. So, so I have, I have, I have a million questions, but obviously this is something I've been passionate about that you, that you've been doing for a number of years now. But I'm going to hold myself to two more questions unless other people jump in in the chat. But the, the first question, I want to go a little bit deeper on something you just said about the reading piece because the reading wars, as you know, have been going on for decades. There's been a lot of movement in the last, let's say, couple years around the science of learning behind reading. A lot of the pieces on NPR that I think Emily Hanford did. And then Lucy Calkins, who was more in the balanced style of reading, at least by rumor and reporting in the press, starting to move some of her work over a little bit. I mean, I'd love you to weigh in a little bit here because you said something very interesting, which is, yeah, maybe for most learners, like a phonics based approach along the continuum, phonics first or whatnot is the right approach. But there are certain learners for whom leaning into whole language at some point during the continuum is actually the right way to think about it. Can you unpack that a little bit? Because I think it's an incredibly important concept and it gets a lot of attention and there's very little balance balanced literacy, but there's very little balanced conversation about the reading wars as those it, it might depend on, I suspect the learner, but also the stage of learning, if I'm understanding.

Exactly, absolutely. By the way, I remembered what I was going to say before so we can start.

Okay.

I wrote it down so I won't forget it again. But so yeah, my working memory is getting taxed. This is super fascinating and it's one of the first things we dove into. Our first learner model synthesis that we did was PK3 reading. And I will say I just want to. One of our first advisors was Carol Connor, a professor out of UC Irvine who was just a brilliant learning scientist. So sadly she passed away a few months ago. But she's really one of the leaders in this and did incredible research in Florida and a number of states. And if you dive into her research and she even they launched a platform called A2i that does this, it really clearly shows. So first of all, I think she did a full RCTs with over 2000 kids, low income. And if they took this personalization approach between these two, kind of balancing simplified, but phonics and whole language, they got to 93% on track reading comprehension by third grade. And this was for, I think it was 70% free and reduced lunch. Large scale and really powerful. And what she discovered was basically, yeah, it ends up vocabulary, really interesting vocabulary was the big signal or was one of the big ones, but was really possibly the biggest one. And what it was was that. And if you think about it, it's actually quite logical. Kids that are high in vocabulary, so basically you give them a diagnostic on vocabulary. If they're high in vocabulary and you give them a ton of phonics, it ends up they don't show much growth because they're probably already pretty good at phonics because they have a high vocabulary, they understand words and how they're built. Right. But if you give them whole language, they take off and show tremendous growth. Right?

Interesting.

Kids that are low in vocabulary, you give them phonics, they show tremendous growth because they need to build that. Right. They don't necessarily have the understanding of words and how they come up break apart and sounds and things like that. But if you give those kids with low vocabulary whole language, then they don't show much growth again because they don't have the vocabulary to kind of get that flywheel going of reading growth. And so it's, you know, and some level of precision. And it's not an either or, like you said, it's basically, you know, what the ones with high vocabulary, 20% phonics, 80% whole language. Right. Ones that are low vocabulary, 80% phonics, 20% whole language. And this is what her platform does. It's a couple of light touch diagnostics and then it coaches the teacher on which part of the curriculum for which kids. Right? And it's not that every kid is total individual, but it's like these groups of kids here, these groups of kids here, and you have a couple sections and they go at their own paces and you can actually, you know, she demonstrated, got really powerful results. And so it is, I mean, it's this thing and, you know, we all know learners and even, you know, my kids that are, that are different, they have different developmental fits and spurts, but, you know, at times, yeah, you know, they need structured sentence starters, you know, and some structured scaffolding. And then maybe pretty quickly they start flying but without that, they're just floundering. Right. And so yeah. That the variability is real. And, and you know, I think there is pretty strong evidence when you dive into it. And actually you can go to the navigator, click on the vocabulary factor and if you look at view, every learning science factor has rep view references and every factor has at least 10 studies behind them. You know, defend, you know, basically why we can make this claim that this matters, this variability matters. So super interesting.

I would love to see you all more present in those debates that are occurring because I think it becomes such these either or camps and it misses a lot of this nuance that you just dove in. I have like a million other questions on the reading that just out of my own curiosity right now, but I'll hold back for the moment. Maybe if you'd be willing, we'll have you on again and maybe get Todd Rose on as well. I know he likes this topic as well. And break down like, how does it vary when it's dyslexic and what kind of dyslexia and things of that nature, because I know that can change.

But. And just super quick on the learner variability navigator, one thing we're looking into, what we would love to do is have these kind of overlays that show like, oh, dyslexia is actually just variability in these four factors that to be honest, we all have these factors. Right. It's orthographic reasoning or whatever and working memory and phonological awareness and one other. Right. And that variability in that is what constitutes dyslexia. And guess what? We all have those factors and we all vary in them. It's just dyslexia is a certain permutation of that. Right.

So that's interesting. That's very interesting and more precise than a lot of the definitions of dyslexia, actually. So, so, so last question for me is for parents right now in the pandemic, they're working directly with ed tech products. It's a lot for them to take in all this learner variability. It's incredibly important, I would argue, to knowing your child better. But it's a lot.

Right?

And they're already dealing with a lot. How can they use these tools, these certifications that you've come up with in sort of quick and easy ways to make good decisions for their child in these times right now.

Absolutely. And really important question and it's something that I'm navigating and at times struggling with as well. Right. It's not, not easy. So A few things. One is, you know, yes, if you can go to our product certifications.digitalpromise.org and there you can see which products are certified. And every six months we announce the next batch of products that have been certified. It's kind of a rolling certification process. And so that can help give you some comfort and give you a starting point of products that likely are doing some thoughtful learning design behind their products. And then kind of, I would talk a little bit about the Learner variability Navigator. And so I will say, you know, don't get me wrong, it is complex. You know, my mantra to our team in designing this and developing this is really, how do we respect the complexity of learning? Because learning is complex. It's not a simple thing, right? But how do we represent it with actionability? How do we represent it in a way that people can do something about it? Right? And that's kind of our goal. And so what I would say there's kind of two things in the navigator. So one is just we have these factor maps which have these learning science factors. And most models, I will say, have about 35 to 40 factors. It's a fair amount in these kind of four areas that I mentioned before. But what they can really help with is, first of all, you don't need to be expert in all 40 factors. That's not the goal. That's not what it's there for. Nobody will be able to do that. It is too complex. The analogy I draw, though, is it is a map of what may be happening and what may be going on. And just having awareness of that map is a huge first step. And so the analogy I draw and I sometimes talk about is if I want to drive from San Francisco to New York, it ends up, I don't need a map. Literally. If you type it into Google, it literally says, take the on ramp on I80 in San Francisco, drive for four days and get off in Manhattan, right? And that's it. I don't need a map, don't need anything, right? But obviously there are hundreds of other ways I could drive across the country. And there are many reasons why I may do other ones. Maybe I run into a snowstorm in Tahoe, need to take a detour. Maybe I love barbecue and I want to take the southern route. Maybe I love sports and I need to go to, you know, Green Bay and something else. And again, I'm not going to be expert on all, you know, 500 different ways I could drive across the country. But a map lets me know that they Exist. Right. And that they are there. And it lets me know, oh, I am interested in this, this might be something. Let me look deeper in this one area. Right. And it gives me a direction. So the equivalent in learning is my kid is struggling in phonics. This actually happened with a district that was using the Navigator in an IEP conversation. The daughter was struggling in reading and they were going to pick phonics. And again, everyone always focuses really heavily on that first column of academics and they were basically going to lay out a bunch of phonics interventions. Right. And reading related interventions. The parents saw this map and saw emotion and was like, emotion, what's that mean? And the teacher was like, well, you know, if your kid has anxiety or, you know, outbursts and has a struggle with emotional regulation or expression, she's like, you know what I noticed is when I sit down to read with my daughter, she gets really nervous, she starts tensing up, she starts stuttering, she starts getting really anxious and it's really hard for her to progress. And so again, it brought in this thing. She wasn't an expert in it. But you know what, this is something that actually may be happening here, may matter. And it lets you bring that lens. And then they got some strategies from the navigator that brought that into the phonics. How do you link in emotional regulation support while you're growing phonics? Right. So again, kind of looking at this map to give you a sense your kid may be struggling from for a number of different reasons and it gives you that awareness. And then the last piece I will say is we do have these signature workspaces that's on. If you go into that first site that you posted our initiative Project initiative and go down to the educator focused tab and website, then we have signature workspaces. And these are curated workspaces in specific areas. So supports for distance learning at home is one of them. And guess What? It has 15 or 20 research based strategies to support, you know, a child. Helping them focus, helping them, you know, with executive function, helping them emotionally. Right. And other ones we have, we have for dyslexia, we have for, you know, general SEL type things. And so we have these curated workspaces that are a little simpler. They sometimes have videos of teachers explaining things. And those also can be a great kind of starter where you're not looking at the. This whole huge database of information and research.

Gotcha. That's super helpful. And Vic, I appreciate you coming on and talking through frankly all the work that you do. And I tell people that this is like in the top three of most important, I think, projects in education that are going on because of the I mean, in many ways, it's sort of like the human genome work in the 90s that started to lay out all of the what we know matters along the different, you know, the ways in which we differ. And as if you've pointed out to me, not everything does matter in every single domain. And sort of figuring out what's significant is, is perhaps 80% of the battle. But deeply appreciated. And thanks for being with us today.

Absolutely. Absolutely.

And we'll be back next time on THE FUTURE OF education. So stay tuned.

Awesome.