The Future of Assessment - Special Episode with Melbourne Metrics, Rethinking Assessment and Mastery Transcript Consortium

Welcome to the Future Learning Design Podcast.

What we find is this cascade of change that's needed, and that's why you need new metrics, next generation assessment, next generation recognition, next generation pedagogy, next generation everything, which we call The new grammar of schooling.

Hi everyone. Thanks for joining me for this week's episode. My name's Tim Logan and the podcast is brought to you in partnership with Notosh. This week's episode is a big one on the future of assessment. If you've ever thought that the standardized exam is a block to educational change, well, things might be changing. Three of the largest global organizations who are working hard to rethink assessment join me this week to talk about the work that they're currently engaged in. It was a huge pleasure to be able to speak with Bill Lucas from Rethinking Assessment in the UK, Sandra Milligan from Melbourne Metrics at Melbourne University, and Mike Flanagan and Patricia Russell from the Mastery Transcript Consortium in the US. And it's a perfect time for this conversation. You can listen out for a big announcement that Mike and Trish make about a significant move to scale up Mastery Transcript's work. You can find more information and links for the three organizations in the show notes.

Hi, I'm Sandra Milligan. I'm the Executive Director of Melbourne Metrics at the University of Melbourne, and we work on next generation assessment, credentialing, and recognition.

Hi, I'm Bill Lucas. I'm Professor of Learning at the University of Winchester and co-founder of a coalition of organizations and individuals called Rethinking Assessment, trying to, well, as the name suggests, change the way we're doing these things.

Fantastic. Brilliant. Thank you so much for joining. I know you've had a really hectic few days there in Melbourne with this amazing conference you've been running. So I really appreciate you joining and the chance to have the conversation. What I'd really love to do is spend a bit of time, not so much on the definitional aspects of what we're talking about with complex competencies or general capabilities and all of that great stuff, because I had a fabulous conversation with Sandra previously on the podcast, episode 72. So if people are interested in kind of listening to that first, orienting themselves around some of the definitions and Sandra's beautiful poem. Then please do that. But what I'd love to do with you today, if I can, is just really give listeners a sense of where are we at now globally. So we've got Rethinking Assessment, we've got Melbourne Metrics, we've got Mastery Transcript Consortium, really, I would say, 3 of the top global organizations working in this space to really lead the change. So just to give people a sense of what does that mean globally, where are we at in terms of particularly pedagogy and assessment? I'm interested in I know, Bill, you've been talking about your, your traffic lights and actually we're doing pretty well on curriculum and understanding, but actually where we're less good is on the pedagogy and the assessment. So if I could perhaps start with a question just around, so just to set the context for people for the need for the change. I know Rethinking Assessment is coming out very strongly saying the exam system needs to be completely reformed. So where do you see particular drivers for that change? For the types of competencies that we're talking about, creative thinking, etc.?

I think for probably a decade, those of us engaged in education, especially education in schools, have been aware that the assessment system is no longer fit for purpose. And as you're implying in your question, Tim Logan, neither is the curriculum system and neither is the pedagogy by which we deliver those things. So there are some general trends in terms of curriculum and some general trends in terms of assessment. In terms of curriculum, we're going deeper. We're going to spend longer doing the things that we do, and it's highly likely that we're going to narrow to a slightly smaller core to enable us to go deeper. Or although, of course, we'll sometimes be studying things that have the familiar subject discipline labels, we'll quite often also be working in other ways. That might be called an investigation or an expedition or a project or a piece of inquiry. Or it might be some kind of interdisciplinary study that might involve two subjects, might involve several subjects. And that's a pretty much a universal trend. Even China has interdisciplinary learning in its new curriculum. So does India. Most of the Scandinavian countries do. And as you look across the world, that's the kind of direction of travel. It doesn't mean it's happening, but that's, if you look at the reports promising it, it's beginning to happen. So that's the kind of curriculum, as Sandra may well say, grammar, the footings, if you like, of the enterprise. And then in terms of how we deliver it, that's shifting too. There will be moments, of course, where you need more traditional, and I'm using that not in any pejorative sense, but more didactic instruction. There are simple routines and key concepts that we need young people to get their heads round and hands round and bodies round, because we're talking about a holistic education here. But there will be opportunities, I think, for much greater involvement by learners themselves in the design of their learning experiences. There will be opportunities for them to work collaboratively where, by and large, learning in schools is a solo pursuit. And there will be approaches that typically are referred to as project-based or problem-based or inquiry-led learning. Now, there's a, there's a whole gaggle of different phrases that you might use around that. And then the third part of this, and they're intricately connected is, of course, so how do we evidence progression in whatever we're doing, whether it's knowledge, whether it's skills, or whether it's complex competencies or dispositions? And there are some key principles there, and there are some key trends. I'll start with the trends because we may want to come back on, uh, some of the kind of methods and principles. So the trends are that we're moving away from single grade, single marks. We're especially moving away from that at the end of transition periods in education. So from primary to secondary, lower secondary to upper secondary, and then when you leave formal education to go into apprenticeship, to college, to university, or whatever your destination might be. And whether you're talking to employers, or you're talking to higher education, or colleges, or indeed parents and all those other actors in this drama, there's a growing sense that we need whatever we have at the end of senior secondary to be able to say with some precision and some rigor what else that young person can do. Here in Australia, we've been using the expression ATAR Plus, and in the UK, we're using variants of children, young people are more than a grade or more than a mark. So that's the direction, and profiles and portfolios and transcripts, and sometimes they're called diplomas, are frankly the way we're going. And I, if I had an Australian dollar or two to make a bet with you, I'm absolutely convinced that in 5 years we'll all have a URL that will be our profile for life.

Fantastic. Thank you. Well, that's, I mean, that's a really great overview to start with of the whole, I mean, I'd love to dig into some of those specific topics if we can, but Sandra, maybe could you speak to what Bill was just saying there about the direction of travel in relation to countries who are talking a good game in terms of the commitment to the competencies, but maybe the change is slow and fine. We'll get into some of that conversation, but are you seeing general unified direction of travel towards these competencies? Or are you still seeing some countries just really digging in? I know Bill's based in the UK. There's some particular people digging in there, I would say policy-wise for that much more cognitive knowledge rich, et cetera.

Just reflecting that Bill and I have been working together all of this week trying to find an area which we disagree on. And it's been very— there's not much that we've found. I would say that in Australia, one of the things that's really good is that we're working from the schools out, and the people we're working with, the principals and the teachers and the kids in the school, they're starting with the understanding— this is the hard stats— that around about 60% of kids leave school not with what they need. About 20% of them leave school before the end of school and we never see them. About 20% leave with nothing and about 20% leave crushed or unhappy or not feeling confident of themselves. And the rest sort of paddle along. So the schools we're working with are saying, well, look, what are our ambitions for learning for all kids? Let's make sure they get them. And we call that ambitions for learning. And if you look at it from a curriculum point of view, most countries around the world have got that nailed. You need a bit of content, you need some skills, you need some general capabilities, but more importantly, you need to package it together so that kids are competent and confident at what they need to do in life. So it's not just a curriculum question, it's also a pasting it together question. Now, as soon As you get to that point, you then say, oh, new learning ambitions. Oh gosh, we assess the wrong things. We're content heavy, or we teach the wrong way. We get up and lecture to them. So you're then getting to pedagogy and learning design. And then you think, oh, we recognize it in the wrong way. We give them a piece of paper with scores on it. Let's get a profile. And then we say, gosh, we use the wrong methods to select them into university or tertiary education. We rank them when in fact we should be matching them to what they're going to do. So what we find is this cascade of change that's needed, and that's why you need new metrics, next generation assessment, next generation recognition, next generation pedagogy, next generation everything, which we call the new grammar of schooling. Now, if you look at it like that, you can say, yes, every system's talking about it, but that is a hard walk to walk, and different systems are going at different paces, ranging from glacial to walking pace. I think Australia, like the UK, like almost every country, has the kernel of where we're wanting to go, but this is a long-term agenda.

Yeah, I'd love to agree with Sahra about the UK, but sadly, I think we're taking a little time to draw breath on this. I mean, the Brookings Institution has looked at the growth of competency-based curricula, and the last piece I read had something like 20 or so educational jurisdictions beginning to take this seriously. So if you think that there are 195 countries across the world, and if you're thinking about a place like Australia, that will then divide down to 8 educational administrations, very similar in Canada. So, you know, that's a relatively relatively small. It's 10% maybe on a generous day. And if I'm going to be really honest, this is an ambition in many of these places. It's not yet an actuality. So I think the truth is we have a way to go. But there are a growing number of us who know that the research is suggesting how we might go, where we might go.

Yeah. Great.

I was just going to add to what Bill just said then, because it's not as if a country is a unified thing. So there's all these pressures. And plays and you go a bit forwards and a bit backwards. So we've been really trying hard in Australia for, say, the last 5 years. And, you know, sometimes you feel as though you're in a battle. You've won that battle and lost that one, but the war creeps on. So I'd say, you know, it's always a dialogue. It's always a process in education.

Yeah, absolutely. And there are— we're no longer in a world in which education is solely authorised by geography. So, you know, whether it's the IB or whether it's New Pedagogy for Deep Learning or whether it's some of the other related expeditionary type movements, there are families of schools out there who are choosing to do something different here.

That's great. No, absolutely. And I mean, I would just, I suppose, reflect on my entire career in teaching since the early 2000s has, you know, split screen teaching, for example, was something that Guy Claxton came into University of Bristol when I was training and talked about. And so my entire career in teaching has had this presence, if you like, of how do you do this? How do you think about not only teaching knowledge, but also thinking about competency development intentionally as a teacher? But then, you know, as in our previous conversation, Sandra, you said that the assessment is often the evil tail that wags the learning dog, right? And I think a lot of people are hoping, they're putting a lot of hope in the work that you are doing around assessment reform specifically, because I think a lot of people do see as you also said in our previous conversation, Sandra, that there's a really great lever there for change in that cascading system that you just described, Sandra. So I don't know, could you give us a bit of an update on where do you feel like we're at with some of those things with the assessment piece?

Can I just make one comment on the split screen? Of course. I've been pondering the idea of a split screen for a couple of years. And when you talk to us, I used to be a chemistry teacher. So when you talk to your average chemistry teacher, you can see immediately that they're thinking, oh my gosh, this is extra I've gotta do. I've gotta do all my chemistry and then I've gotta do all this stuff. Oh, split screen. Okay, I'll do a bit of this and a bit of that. One of the things that's coming through from our schools is a development of that, I think, which is, look, what is the objective here? The objective is not to teach a bit of this and a bit of that or to integrate them. It's to, and here's a shift which I'm playing with. So, you know, Love it. Don't take it too seriously. The shift is this: from getting kids to be consumers of curriculum delivered by teachers to being producers of their own competence. Now, as soon as you think about that, you stop thinking about a split screen. You start thinking, okay, I've got to turn this student into a producer of chemistry knowledge. How do I do that? Oh, they're going to have to collaborate. They're going to have to communicate. They're going to have to do this. They're gonna have to do that. So I'm trying to find a way to make the split screen not bi-focal. It needs to come together.

Absolutely. That's amazing.

The split screen metaphor and planning and design process is something that allows people who are starting out to see that we're living in two worlds, but it's not the end game. The second element of learning design thinking from my perspective, and we've not really chatted much about that, this week, Sandra, but I suspect it is in your work, that a complex competence is made up of a whole series of other elements. And those elements can be broken right down, and here I'm drawing on the work of people like Ron Ritchhart at Project Zero, to thinking routines, or making routines, or collaborating routines. But they have an element which you really do need, paradoxically, to learn as a little mini routine before you can then kind of be more fleet, even free-flowing. Sorry, end of a long week.

Yeah, no, I think that's a great point. And I know that it's come up for you this week, that a different metaphor, but riding two horses. And I think Ellen, who's been fantastic in supporting us to pull this together, mentioned that, yeah, that idea of school leaders feeling like, and teachers probably feeling like they're trying to ride two horses. Like, you know, it's more, as you say, traditional academic piece, they've got to keep that going because that's still got the value and the credential. But actually there's this other thing they're also trying to bring in. But I agree, I mean, that idea of the synthesizing those with the and the mastery around them. And what you were saying, Sandra, really reminds me of Gert Biester's work around subjectification. So thinking about, you know, what are the opportunities for young people to step in and take up, you know, that kind of role as subject themselves. And yeah, as we know, not an easy thing to do. But yeah, could perhaps I could come back just practically to the point about where do you feel like we are in terms of the more, hesitate to use the word, rigorous rollout of those kinds of assessments and measurements or credentials, whatever word you want to give to them in terms of beginning to have higher education, for example, really take them seriously. And last time we talked about micro-credentialing and backpacks and all those types of things, Sandra. So I, yeah, I wondered what's the current state of, of how things are moving in that zone.

So, you know, from an assessment point of view, over the last few years we've been, I think, going like rockets really. We've now done 667,000 assessments using what we call the next generation assessment so that we can work out how we can assess these complex competencies in a way that's rigorous, in a way that supports formative assessment in schools right through to credentialing and is trusted by everyone concerned, trusted by the students to represent who and what they are, trusted by the teachers so that they can see growth and see the feedback loop in their teaching, trusted by parents, but also, as you said, Tim Logan, trusted by higher education institutions And for the— over the last 2 years, we've now issued 2 forms of credentialing in Australia at the senior secondary level that is now being accepted by 50% of the tertiary institutions as an alternative to the standard senior secondary certification. And we think that's really significant. We're not saying that we've nailed everything. We work with a wide range of schools on this, and we call the partnership with the schools a research practice partnership. It's been going for 4 or 5 years now, and I can see another 4 or 5 years required before we get to the point where we're as certain about the next generation assessments and their utility as we are about, you know, the traditional ones. Yeah, even if they don't do what want to do anymore. They're trusted. So we want to be as trusted.

Yeah, that's brilliant.

In the parallel world that— and I really do feel that in a collaborative and gap-filling way, Sandra and I have slightly different emphases on what we're trying to find out. In the UK, we've been starting with taking those existing project qualifications and using them more effectively. And that means sometimes explicitly generating new content for them because they're largely content-free zones, but they needn't be. They could be a really good playground for exploring interdisciplinary learning driven by really good questions. And interestingly, in the, in England, out of a cohort of 600,000 young people, only 30,000 take the Extended Project Qualification at 16 to 19.

Yeah.

So that's 5% only. So our goal this next year is to ramp that up to 30%. And to then go down into the next layer down in senior secondary where there's something that's half the value of a GCSE. And it's very easy to timetable because it takes half the time of a GCSE if you want to do it in that very linear way. So our way of opening up this broader, more strength-based, competency-led work is to fill up this embryonic profile with aspects or elements which we're getting much tighter and more careful about understanding how they work in the real world of schools. And another example there would be rather than doing the brilliant work that Sandra is doing, in fact, our hope will be that we'll simply make that more international. On our learner profile, we have 3 C's. So we simplify some of the things that appear on international curricula down to just creative thinking, collaboration, and communication. And that communication may turn into oracy, but the 3 C's is quite a useful mantra. And we'll have some kind of visualization and you'll click through into data. And what data? That's where we're exploring. But the thing we've been working out is, can you do that in the current restrictive English system? We've just completed an exploratory study, and I'm delighted to say the answer is yes, you can. Ironically and brilliantly, our research partner was the Australian Council for Educational Research over in England, and we'll be publishing that shortly. But there are lots of other aspects of the credentialing which are part of this mix, that we have courses that aren't as big as the current courses, so you can have more choice and more variety, that there's more learner agency. So there may be as well as testimonials and film clips and podcasts, who knows? You know, that would be a very interesting part of it. The message is the medium. So there's a lot more to be done here.

Yeah, great.

But it is, we're very clear, prototype, draft, work in progress, research. We're not saying we've got all the answers. I think that would be hubristic. Neither of us is saying that.

Sure.

Yeah. I just, one thing about that. I mean, it's really interesting working with Bill over the week cuz you know, he says, oh, we're doing 3 Cs and we're saying, oh, we do 3 Cs and plus learner agency. And someone else says, oh, we do 6 Cs.

And you know, there's always the Cs.

Yeah.

Yeah. Always Cs. We've got an Al, uh, as on top of the Cs, you know, we, but the point is this. Everyone talks about profiles and video clips and projects and so forth. One of the important features, I think, of this is that it's— that all of that diversity is great, but it needs to be underpinned by a common conception of standards and ambitions that can be used to provide a standards-based recognition of what you can really do and who you really are in a strengths-oriented way. And so the thing we think's important is, yes, you can have 3 Cs or 4 Cs or 6 Cs, but underneath it, there should be an understanding that there's actually a common language here. And from an educational point of view, a common set of standards that say employers can trust. So if you've got 4 levels of communication, that 4th level should mean something for an employer or a higher education institution that gives them an idea of what that student can do in communication. Similarly, Level 4, Level 3, Level 2, it should be behaviourally and standards-oriented. And that's what makes this hard because anyone can do a rubric.

Of course.

Anyone can rank kids up and down. But the thing is, are they well established in standards and language that everyone understands?

Yeah, but isn't that the nub of the problem? I mean, like, I feel like that shift almost— I mean, you could crudely say from qualitative to quantitative, right? Is there— I know it's not only that, but there is just this idea of— and I loved— I do love what you say about utility and trust, Sandra, because I think that's incredibly powerful. That of course they need to be trusted, but they also need to be useful. And obviously those two things are interconnected. I worry personally that with the shift into that standardization again, we come back, we get pulled back into this old frame of basically we want a common language for every context, for every young person. And because we're trying to make it useful and trustworthy, we get pulled back into that standardization trap. I don't know, what do you think?

Now, Tim, if you use the word standardisation in the same sentence as me, I'll scream, because there's a difference between standardisation, which normally means the same content, the same task, the same rubric, yes, an objective assessment, often computerised, you know, all of those kinds of things, Standards do not require standardisation. And so I make that distinction very strongly. Our version of standards, things that any employer can understand, like in collaboration, the lowest standard is when you're working with someone else, you do as you're told. The next standard up is, oh, you actually work out what you should do and do it on your own. Next one up is you adjust what you're doing to what everyone else is doing. And the next one up is that you coordinate people and help them together. So they're standard and they're rich and they don't require anything to be standardized.

Yeah, no, I think, I think that's a really useful distinction.

Their competencies and they're being shown in the real world of school, just as you might show your competence as a safe driver of a car. The levels of things that your instructor and then your tester will be looking at, that there will be a checklist there that will be standard. But the road conditions, the way you're dealing with it, those will all be different. There will be a theoretical knowledge component, a skill component, both physical and cognitive, and there will be your ability to read situations that you've not been prepared for. And you could probably create another Bloom-like taxonomy for driving a car very well. So it is different. It's not overly complex and it's definitely not standardized in the way that many of the educators that Sandra and I profoundly disagree with by using that term.

Yeah. Okay. No, I think that's a really useful distinction. I guess part of my question, I suppose, is coming from, because Bill, I've heard you talk about the idea of taking these models and really contextualizing them in particular school and district context or what, you know, local context. And I think, so there's something really important there for me about, because how broadly accepted are the same types of standards that you're using? How local can they be for the learners in that particular locality? I think there's an interesting question there because are you talking about employers and institutions nationally and all having the same common language? Yeah. So there's that kind of localization question.

These are all the right questions. They're all the right questions to be asking, Tim. And we've got some glimmers of some answers. We're doing a pilot in Doncaster in the UK. It's so it's end to end early years right through to employers and people who've kind of become unemployed and need to reskill themselves. It's a, it's a really rich test bed. And those are the kind of questions that we're asking. And do employers all have the same understanding of even the same word? No. Right.

Exactly.

Yeah.

Do all teachers? No. Do all parents? No. So we are in, and we've been having a discussion today about this. There is a period of advocacy clarifying. Messaging, but we are all human beings at the end of it. So we may well end up choosing slightly different vocabulary, but the more that people like Sandra and I can bring a common understanding to bear on this, the better. If you ask the man or woman in the street, 'So what's science then?' you'd get 5,000 different answers. So we can't apply an unfair test to a competence like creative thinking, say, compared to other aspects of the curriculum. In fact, I'd want it to be complex. That's the point. This is a, you know, a deep discipline. I'd want that to be complex. So we're in a transition and that's back to your reminder that we've been using this metaphor of two horses or two races. That's where it is a little bit uncomfortable just whilst we're going through this period of learning. And some people may feel they're in, in these kind of two spaces, two headspaces.

Yeah.

Can I just again do a little reflection? We spend a lot of time musing and reflecting and trying to figure out, and so our team has philosophers as as well as psychometricians and teachers. So we muse a lot. Yeah. So one of the musings that we've had is, all right, how do you stop standards becoming standardisation, which is sort of what you're getting at. And, you know, because we work with a lot of schools, this is a really practical problem because we work with schools who have 5-year-olds who are living in Aboriginal communities that have very little direct contact with Western community. So that's different to the leafy green school in a big city, which is very rich and has Year 5 students. And you'd think, well, heavens, you know, how can you apply the same standards to them? You can't apply lots of things the same, but what you can do is if you understand the progression of competence, i.e., the standards, you can work hard to work out what are the behavioural indicators in each context that signify the development and also to judge whether or not that is a culturally appropriate concept or construct. What we've found is that for things like, you know, the standard things like agency and learning, communication, collaboration, the Cs and so forth, that by and large the idea of what the progression is is very common. And we're doing some work in the Philippines, and we were thinking, my gosh, that's a really collaborative society, much better than competitive Australia. I bet our collaboration progressions don't work there. Well, they did, but the indicators that people were using and looking for in particular contexts contexts are different. So we distinguish between the standards and the progression and the behavioural indicators that are linked to it, and that's where the rigor comes from because we tie behavioural indicators from different contexts to the progressions, and that's why we say we think that we can have a pretty precise allocation of a student to a level using different indicators of behaviour, but the same progression. And that's the technical side of our work.

That's great. But it also makes me think of what you said in our previous conversation about them not being too overly complicated and specific. And actually, because I think, Bill, your point about complexity and allowing space for kind of local emergence, I mean, one of the principles there is allowing space for that emergence in different contexts. And I think so that idea, when we try to over-articulate and over-describe the outcomes that we want, want, then I think we get much more trapped back into the same standardization trap. Whereas actually, if we can allow the standards to be somewhat, you know, as open as possible, but still to have the utility and trust that we want, there's something there about allowing for that complexity and emergence to develop based on where they're being implemented.

The best way to hurt our feelings is to say that we're standardizing things.

So, you know, I hurt your feelings on a Friday evening after a long week. I apologize.

That was my way of saying we really care about that issue.

Absolutely.

Another parallel look at this issue would be to say that through the work of Sandra's team, I think we'll get to a stage quite rapidly where people will say, and by people I mean all the actors in this drama, will say these things are understood, reliable, and valid enough to be trusted. And we take that on, you know, and we, the warranting by the university gives us that comfort. But most importantly, they will always be useful. And that's what you cannot say about a number of other so-called often standardized tests where, for example, our own qualification body Ofqual in a parliamentary inquiry 2 years ago reminded MPs that grades awarded at GCSE and A-level in the social sciences and many of the humanities could only often be one grade accurate either side. So a B could be a C or an A. So again, I think we just need to be a bit more real world about what we're talking about here and what's sauce for the goose is sauce for the gander.

Exactly. Yeah. But I mean, you pop some sacred— mixing metaphors— pop some sacred cows, or I don't know. You, I mean, you, you really have to challenge some sacred cows, right? Because you really, I mean, you have to question, and again, in our last conversation, Sandra, we did exactly this. You have to question the robust objectivity, so-called, of that quantitative game that's been played traditionally. Can I ask, as you said about the actors in the system, I wonder, are there things that you are learning from the young people as they experience these new metrics and new credentials? Because I, I think for me that's an important point that we unders— we learn how does it feel to be, for example, surveilled and observed in all these different aspects of life. I have some interesting kind of wellbeing questions around that.

That's a good question. And we're learning huge amounts as we go through. I'll just give you two examples. We've been working with many focus groups in a number of schools, looking at how would a young person look after and curate their portfolio or profile? Would it become a chore? Would it be, would it give them joy? Will it be done in what we would call a home group or form group, or, uh, what will that be different in primary and secondary? Because we want whoever the adult in this space is really to understand the person, the learner. So we're exploring all those sorts of issues and we're not clear, but one thing we are getting very strong evidence of is the more you invite learners to take more ownership of the progression of their learning, the more engaged they become. So that's a definite win. The second example I give you is probably more about how the school, especially a busy secondary school, administers this. So we've got some prototypes there. I know Sandra's got a number. Where we're just trying to think of in amongst all the other busy things that teachers do, how do you make this fit? How do you make this work so that the teacher actor in this drama really feels this is time well spent?

Yeah, and Tim, I think, look, there is a risk of surveillance and, you know, everything's metricated or metrified or something, but I think if kids feel surveilled, you've got it wrong. Because some of the things that go against that is, first of all, this isn't a thing that you do to kids at the end of something. It becomes integrated. For instance, in senior secondary, they have experience with these things for 2 years. They become the focus of discussion with their favorite teacher. They have their own versions of the rubrics or whatever.

Yeah.

So that they're continually assessing themselves. So that where we've had the most experience with this in the Big Picture schools in Australia, the kids are agents of their own assessment. They'll talk about it all the time. And what that gives them is a sense of confidence of where they're up to. It gives them a language to talk about it. So for instance, you get a student from a Big Picture school and one who hasn't done that, you know, what's your work experience? Oh, first kid in the normal school says, I flip hamburgers on the weekend, you know, that's my work experience. The big picture person says, oh yeah, I do, I've got a hamburger flipping job, but that's really good because I've been able to use my communication with the manager and the customers to show how my communication skills have developed.

Yeah.

So they get a sense of pride. We say they have agency in the assessment, so they do their own assessment. They argue it to their teachers over a 2-year period. There are no surprises here. You know, at the end of the 2 years when they get their certificate, they probably don't even glance at it because they know it.

Yeah.

So as soon as you— the student feels surveilled, you know that you haven't got it right. So go back.

Great. I love that because I think, and in many ways, even the way that you are going through this process of iterating and reflecting, I mean, as a research process, I mean, we live these as adults. We want this kind metacognitive competency around ongoing reflection and improvement, right? I mean, that just seems like it's part of being a reflective human being, which we need, we need more of, one might say. Bill, you know?

Yeah, no, I was gonna profoundly agree with you.

Okay.

But I would also add that, and this is the question I wanted to ask you about, you know, Marilyn Strathern and the Goodhart Law of when the measure becomes the target, because I think if that process is not happening also within the broader complex ecology of the employers and the universities and all of those things, then you end up with, because we know that people like to game the target that they are being held accountable to, right? We do it as employees. It happens everywhere. We do it as children with our parents. You know, we try and get away with what we can. This is not a cynical view of human beings, but I think that's just a practical reality that that does happen. And therefore, how do we avoid when we set these standards and these really rich opportunities to display complex competencies, how do we avoid people then making the decision to go and work in the hamburger flipping joint because it will develop the things that they are being held accountable to? And I think that's a subtle nuance of difference between those things, and maybe it doesn't matter that much, but I, I worry that if we are too high stakes systemically, then we will maintain some of that gaming. Which can then feel like surveillance.

Every system will have those who try and game it, whatever exam system or whatever school curriculum we have. I think, and we were talking about this today, actually, that rather than Goodhart's Law, I think Campbell's Law is probably more effective here, that if your measure is a quantitative measure and you're trying to make any decisions about that, rapidly people game that. And the obvious point of that is when you're moving from higher secondary into university, university. That whole gamification takes place in Britain with the UCAS system, and most countries have something a bit like this. You get tariff points for certain high-status exams, and you get tariff points for some other things, but not all other things. So for example, we could learn from the uniform groups and their use of badges, not necessarily digital even, because when a learner is ready, she comes to the leader and says, do you think I'm ready for the camping badge or the something badge? And so there's that dialogue. So it's not about Oh, I'm going to go and get my level something by flipping my burgers. It's a constant sense that life is complicated and I'm going to take every opportunity I can to make a little bit more progress. Sometimes it'll be an explicit thing and I'll be having a conversation of a kind of coaching mentoring kind. Other kind, I'll just be logging it in my, as it were, not yet filled backpack. And when I'm ready, I'll say, you know what, over the last 4 weeks, I think I've really made this progress. Do you agree? So there's a whole different kind of conversation going on, and it's not a surveillance conversation, it's a dialogue conversation. And that's, I think, so healthy. And if your child is only learning her communication by flipping, then you might be, as her mentor or his mentor, saying, well, you know, there are opportunities X, Y, and Z. Shall we have a look at those too? And it's often a we, so that you're having a conversation about that. Because if there's one thing I'm particularly— and we've been talking about this a lot this week, Sandra and I— both concerned about, and that is that you wouldn't want want simply to mirror the stratified societies that we live in here, right? You wouldn't simply want to be a child who isn't blessed with the advantages of her or his more, uh, advantaged peer to have this to be something else where, you know, there's even more gap. And there are ways around that. Interestingly, we're working with— in fact, we're working with a big picture additional needs provider in Doncaster where the profile is a brilliant addition because additional needs teachers count in one. So what better way than trying to really understand the all-round profile of an individual learner? In the state sector, in busy secondary schools, we're arguing for an electives premium. And this has been recommended by a number of different organizations. So just as we might have in the old days put some energy into maths and English tutoring, so we need more resource in the system to make sure that the very young people who need additional support here get get it. And that's something else we're trying to explore in the testing that we're doing, Tim, in our work in England.

Fantastic. Sandra, did you want to—

Look, we just haven't found the gamification to be a problem yet. And the people who are picking it up are the tertiary institutions who are getting high failure rates when they pick students based on the old things. And they're saying, what we want to do is match, not rank. So they're looking for a strengths-based profile. And, you know, if a student can choose to be assessed on their strengths and their strengths are represented and, you know, the other bits are de-emphasized, and the tertiary institutions are more now looking to saying what we want is people with these strengths or those strengths, and so the differentiation is possible. I have no doubt, Tim Logan, I'm with you on thinking what can be gamed will be gamed. We haven't yet seen how that might play out. We're on alert all the time, but this matching, not ranking idea really works against opportunities for gaming.

So, you know, just to, just to reframe the idea of games and gaming, I think that games and gaming are, and there's a really good set of researchers in Paris that Todd Lubart is part of, who are exploring the ways in which— you might call them scenarios, you might call them games, you might call them online inquiries— can be used both as enormously fun and effective learning, but also to give you real-time feedback on how you're developing some of these processes. And ultimately, that's what we were doing in the recent PISA creative thinking test. It's essentially a series of open-ended scenarios, yeah, which you're doing online.

Them.

So, you know, I think there is some learning to be had here, and I think there will be a new generation of— as well as observational behavior-led, there'll be some other processes that we can use as part of this mix. I think it's going to be a rich mix.

Yeah.

And I think what we're not going to have is this tyranny of the paper and pencil timed, everybody does it the same time, test of memory. I think that the days of that are numbered. And I say that in the week schools in Australia are doing their NAPLAN test. So I, as I've been in this week. That's been hilarious.

Yeah, there's an irony, but no, thank you. That's amazing. I mean, I guess one thing I would just reflect on is what you're saying about strengths-based, Sandra, is we also therefore need an ecosystem that acknowledges and genuinely acknowledges and values weaknesses in that, because the whole point of a spiky profile is that you won't be great in everything. And I think that's where the gaming has happened in the current system, is that you end up with these people who profess to be off the charts at everything, which is clearly a nonsense. And that idea of not just saying we value failure in words, but genuinely building into the system a way that there is a value placed on reflective or reflection on your own development.

Guy Claxton and I have a little game that we play when we visit schools and we're talking to young people who are clearly, you know, flying, doing brilliant stuff. And we'll say to them, so that's great. What could you do to make this harder for yourself? You know, what would it look like if you took it to the next level? So this is a largely set of formative processes, much better understood by young people and their teachers and their teaching assistants and all those working with them. It can only help to generate more reflective learners, to use your language, Tim.

Yeah, brilliant. Thank you so much. This is a rich conversation. I so appreciate it. And all the work you're doing is just hugely valuable. Pushing this forward. So yeah, thank you so much.

Thank you, Matthew.

My name is Mike Flanagan. I live in the suburbs of Boston, Massachusetts, and I lead the team here at MTC in my current role. But when I joined back in 2017, I was the Chief Product Officer, meaning I, my primary role was developing the platforms and tools that we use today to build and deliver Mastery Transcripts and MTC Learning Records from high schools to colleges. And I'll hand off to Trish.

Thanks.

I'm Patricia Russell, and I was an administrator at Phillips Academy in Andover, Massachusetts. I was the academic dean and heard about this project before it launched from some folks who were on the founding board, ended up joining that board, the original founding board. There was conversation about needing to have somebody who didn't have another full-time job. Working on launching. And so I helped in the interim before there was a CEO and have been really honored to work on this project ever since and see it grow and get to this point that we're discussing today.

Fantastic. Well, it's a great pleasure to meet both of you. Thank you so much for taking the time. And just bearing in mind that the audience of the podcast is very global, and so perhaps there is a small chance that they haven't heard of Mastery Transcript Consortium already, but so perhaps you could just give a brief kind of 101 elevator pitch just to contextualize it for people, and then we'll dive into some of the details. Obviously, that's where I'd like to spend most of the time.

I will describe the kind of overall mission, vision, and work of MTC, but then I'll ask Trish to follow up and talk more specifically about the current composition of our school membership because it is, to your question, a global membership.

And I think that's—

so, you know, MTC started— Mastery Transcript Consortium, it's quite a mouthful, we have a long name, but it's easy to find us. We're at mastery an organization started with a pretty basic but big idea, which is that we think that there's a better model of schooling, specifically high school. Like, that's where we started. Uh, it's more interdisciplinary, interactive. It's place-based and experiential and project-based. It focuses on development of skills, not just memorization of academic content. And what we observed is that there are thousands of amazing test sites and pilots of this kind of student-centered competency-based learning model. Both in the States and globally. But for some reason, they aren't scaling despite the fact that they are incredibly popular. Educators enjoy teaching in that environment. Students are engaged. They actually enjoy going to school. They're proud of what they're doing. They're eager to talk about their work. And parents and caregivers are delighted to see that their kids are engaged and doing meaningful, deep learning. These models should be everywhere. And there isn't yet. And what we realized is that one blocking issue— and I want to be clear, there are many blocking issues— but one blocking issue is the absence of suitable infrastructure for the credentialing and crediting of that kind of learning model. Specifically here in the States, our model is based almost entirely on the concept of the credit hour or the Carnegie unit, the amount of time that a student spends in a classroom exposed to learning as opposed to how much they actually learn. So we have grading systems that, in which it's perfectly permissible to move on with, you know, a C- or, you know, having mastered 60% of the material, you know, which is not a benchmark we would want for any important task, right? We wouldn't want to put out 60% of a burning building if it were on fire. So we have built with this consortium of schools, we are a membership organization and a nonprofit one, we've built what we call the Mastery Transcript, and it is a digital interactive skills-based transcript that is used by our schools, and it allows young people to drive and own the credentialing and the building of their summative records, but still they are institutional records. They are official, they're validated, they're secure.. And we have worked, invested really, I would say very, very deeply in advocacy and engagement with higher education here in the States. We'll talk about this later in the discussion because it's a nice counterpoint to what's going on with Bill's work in the UK and Sandra's in Australia. But in the US, everything is decentralized. I dare say decentralized to a fault. And so we can talk about the unique challenges that poses us as we try to engage quote unquote higher education. So that's MTC. You have to sum it up. We build innovative school records for innovative schooling models. And we're working to make those just as widely known and accepted at scale as the traditional models, with the idea being that once that happens, you can have one of these amazing schools and you're not forcing anyone to make a choice. Because right now the concern is that if I'm a parent and I want to enroll my young person in a school that is too innovative, they start to look non-compliant with the higher education admissions process, right? It's like, oh, that seems like a really interesting learning was happening, but we don't know how to make that legible. We don't know how to how to convert that into the units and numbers we care about in higher education. That's the chasm for us.

So yeah, no, amazing.

Talk about the, the schools we serve. Well, you know, we're really proud of the composition of those schools.

Yeah, the, of the current membership, uh, it's about half public schools, um, here in the U.S. and half private schools. Public meaning are, you know, in the U.S. that's funded locally by tax dollars, of course. And about 10% of the schools are outside the US at this point, all over the world.

No, that's great. And I mean, obviously we're, we're what, 10 years on since Hawken School and Scott Looney and the, the origins, right? A lot of what you've spoken about there, Mike, is very resonant with educators. They're kind of, they're ready, right? There are so many people all over the world who are so ready for some of those blocks to be removed, right? And so I think it's a really interesting time. And it's also, I would like to give people a kind of a bit of a landscape overview globally of where this is at. And obviously Sandra and Bill have talked about that from their different contexts in the UK and Australia. But for you, what would you say was the current state of this? I think this is quite an interesting time for you. So this is probably a good time to be having this conversation, but where are you at in terms of where you feel like you're making the most headway and, you know, where is it moving?

Yeah, well, I think when we started the project and, you know, MTC really got formally structured as an organization with a team and resources and starting to work directly with schools in 2017. So, you know, coming up now on 7 years of that work, you know, we spent several years really just talking to schools and higher education, trying to understand, hey, what would need to be true, you know, in terms of a design perspective or the user experience for a different kind of, you know, credentialing system to meet the needs of both our K-12 educators and learners and also our higher ed end users. And to jump ahead to where we are now, I think there are 3 things 3 things that most significantly have shifted that are really making us excited about the next phase for us. One is the shift in the national landscape here in the States around K-12 education. The second is the shift in higher education, both in the tension between the admissions function and then also the perception of the value of the degree. And the last is sort of what you might call the sector at large, right? And how we're thinking broadly across the education to workforce pipeline and, and, you know, how we're thinking about the use of skills in those arenas. So in terms of K-12, you know, there's a partner organization of ours that I would really point your listeners to, which is called the Aurora Institute, because we think they are the undisputed leaders in tracking the adoption implementation of personalized competency-based education here in the States. They literally have a map that they update, you know, on an annual basis showing sort of how things are going in each of the 50 States. It's worth pausing there. There are 50 States. If there's one big difference between what's going on with Bill's work and Sandra's work, it's this. Decentralization we have. We, you know, the US really embraces, I dare say fetishizes local control. And so it means that there is no way of working on a national model for competencies. Everything has to be done locally, sometimes by consensus. Sometimes this is great. It means that there are testbeds for innovation. But then when you get to sort of essential questions of how do we make sense of this all centrally, you know, how do we make sure that new kinds of credentials are accepted across state boundaries or inside and comply with state policies? It's certainly a big lift. But the movement towards this kind of competency-based education is only accelerating. Now, there are different theories of why this is. You know, one, it, it could be, you know, the experience here in the US of the pandemic is that it gave many US parents literally a front row seat, you know, via remote school to exactly what education was looking like for their child. It was probably an unfair glimpse. I don't think any school was prepared to pivot gracefully to online learning, you know, with the drop of a hat.

That.

But it essentially left many families really, you know, feeling a little cold about sort of the value of the experience, the engagement that they were seeing. So that idea of like, oh, is there a better way of doing this, right? It's very hard to drive change if everybody's delighted with the current state.

Exactly.

But when people are unhappy, they're like, okay, yeah, maybe there's a better way to do this. Maybe they're, you know, if my child's going to return to school, right? We can have a side discussion around if we have huge attendance and truancy problems happening now in the States. But if you want children to return to school, you have to create an environment that is engaging, that draws them there, as opposed to just mandates that they be there. We're also losing a lot of educators from the profession right now in the States. It's— we're seeing a real teacher talent shortage or talent flight. So an opportunity to create learning models that are more engaging for educators that actually get them closer to why they were drawn to the craft in the first place, right? Which, as our founder Scott Looney has said many times, no educator, no teacher gets into We don't get into the profession to sort children, right?

Exactly.

We get into it to help young people grow. So I think all those are sort of, you know, winds that are shifting in our favor in terms of K-12. On the higher ed side, we've had some particularly kind of landmark cases happen in our Supreme Court around admissions practices, specifically how we reckon with our country's racial past and sort of the injustices that we visited on Africans that we took forcefully from their homes and brought here in chains.. And we had basically by consensus created what, you know, some would call affirmative action, what formally you would call race-conscious admissions practices to try and combat that legacy. And for a while, I think it was working pretty well, but we have a conservative Supreme Court and they upended it and said, no, you can't do that anymore. It's illegal. Yeah. The goal of our admissions leaders is unchanged. They want to build incoming first-year classes that reflect the amazing diversity of talent and background that the country has to offer. One of their biggest tools for doing so just got taken away from them. So they're under the same pressure, but they're not allowed to do things the way they were. So what previously might have sounded to them like a very odd idea— hey, there's this organization that's working with schools to generate different kinds of skills-based summative records with different kinds of data about what young people know and can do and what they're good at. Suddenly that seems much more interesting to them.

Yeah.

They're like, oh, now we, there's a problem they want to solve and we may have a solution. So that has shifted quite substantially. The other thing that is starting to shift is we're having a much more active dialogue about the value of a 4-year college degree. I get annoyed by sort of false binaries and these happen quite a lot in the journalism around this. Everybody has to go to college. Nobody's going to college anymore. Well, the truth is somewhere in between. What I hope we could all agree on is that there are some degrees that are incredibly valuable for the young people that are earning them and will help lead them to better outcomes. There are also some degrees, an alarming number of which, where a young person might enroll in college, take 5 or even 6 years to earn the degree, leave with a tremendous amount of debt, staggering amounts of debt. And ultimately have a credential that doesn't actually do much to help them get a job that's much better than the job they could have gotten without that degree. I am very much against that kind of degree. That is not a good outcome for anybody. We're not alone in that. There's a lot of other people who are like, that doesn't seem right. And so there are now interesting conversations happening and projects being launched and initiatives around saying, what would it look like if we adopt apprenticeship models closer to what, you know, one of my wife's close friends from elementary school, she lived abroad as a child. She lived in in Istanbul. Her best friend now lives in Sydney and her husband is a civil engineer and he doesn't have a 4-year degree. He has an apprenticeship. He, you know, he learned how to be a civil engineer by studying engineering with engineers and dare say he's very good at it, you know. And so we know those models can work and, but we don't use them here in the States at scale. So that's another issue. Like what's the nature of the degree? And if you think in a world where high school to college to job is a very simple linear track, And then it's just like, get good grades, get more good grades, get a good job. When that no longer works, it widens the aperture. Like, well, what are we measuring? And who— what are we preparing young people for? And what kinds of choices are they going to have to make? So that's, I think, the two biggest things that are happening on the K-12 to higher ed side. And then if you take a step back and look at the sector overall, there is a broader national conversation happening around what are the skills that young people need to have to be equipped for life, right? That's really the game here. How do we make sure they lead healthy lives? They're engaged citizens. They find some kind of profession that allows them to be both intrinsically rewarded, ideally, but also secure economically. All of these different things. And there is now a sort of an emerging national consensus that there is a set of skills that are not actually captured in pure academics. And this is where things come full circle. And we couldn't agree more with the conversation you were having with Bill and Sandra about whether you call them the 3 Cs or the 6 Cs. We have the privilege of looking at literally hundreds, if not thousands, of so-called portraits of a graduate that have been built by local school districts and schools. I alluded before to this kind of idea of decentralization. One of the strengths of that decentralization is that every community that is thinking about competencies is going through a locally driven backwards design exercise to make choices about what are the skills they care about. And it turns out that in a country where we agree on almost nothing right now, the US is extremely polarized. Polarized, everyone agrees that these skills are the same. The language varies sometimes. Yeah, you know, what is the exact description of mastery of communication at a certain grade level? What are the, you know, individual sort of detailed, you know, attributes of collaboration that— but the core, the essential skeleton or DNA of these things is incredibly consistent across school districts. And that's why, you know, we— this is a nice opportunity for us to announce that a year ago, two of the biggest leaders in the field, the Carnegie Foundation Foundation for Teaching and Learning, and Educational Testing Service in Princeton, New Jersey. Very storied, you know, organizations with very long histories. We could go into it. It's kind of fascinating. The Carnegie Foundation actually created Educational Testing Service, just as they also created the Carnegie Unit, which is the credit hour. So they made a lot of contributions to the field. But both organizations now have new leaders. And Tim Knowles, who is the head of the Carnegie Foundation, has said publicly, We created the Carnegie unit, the credit hour, and we're sorry. It was useful in its day, but it's now a constraint, not a tool. And we need to figure out how to move beyond it. And he sort of staked his tenure, you know, as leader of that foundation on coming up with a new model. In turn, the leadership, Amit Sevak is the new CEO at Educational Testing Service, has now gotten on stage and said, we have to move beyond what we think of as standardized testing. Testing. It's very strange, you know, where Sandra, when you were talking to her, reacted very powerfully to the idea of standardization versus standards and said, no, it's not standardization.

She did.

Yeah.

I would never have imagined in my lifetime that the CEO of Educational Testing Service could have been talking to her and would have agreed. And it's like, yeah. So those two organizations have developed an initiative that they call Skills for the Future. And the goal is to widen the aperture to take what we used to call soft skills, but are now calling durable skills. To recenter those in the national and international conversation around what counts. Because these organizations are uniquely well positioned for research, they've already done an amazing amount of heavy lifting and review of the literature. And they can show, you know, statistically and defensively that these skills are predictive of success in life, right? It's not just us saying so. It's not just educators having good vibes around these things. We can show that that if young people are equipped with these as a toolkit, they will do demonstrably better as adults and everybody benefits. So this Skills for the Future initiative is refocusing skills. It's rethinking how we assess, moving beyond high-stakes, single-sitting standardized testing to something that looks much more like authentic performance assessment. And we can talk about sort of Sandra's take on that and ours.

Yeah.

But still doing it in research-backed valid ways so that the essential question that frankly we get asked all the time at MTC, how do you know? How do you know that these assertions by an individual individual school or district or educator? How do we know that they are accurate, that the rubrics here are actually being followed? And then what they're also talking about is building. And this is where we'll come full circle. It's a lot of buildup for the reveal. Both of these folks have said quite publicly, every child in America, every adult in America actually should have a digital skills transcript.

Yeah.

We should be moving to a world where every learner has a, what do you call it, a skills passport, a learner wallet a learning and employment record, it is essentially a Mastery Transcript. So we are really proud to announce that actually effective early May, MTC will be part of this initiative. We are going to become a wholly owned subsidiary of ETS, Educational Testing Service. We will remain independent. We are still a 501. We have our own board of directors. So that's a very long-winded— thanks for your patience, but it's big news for us. And, you know, not to state the obvious, we are very proud of the project. We're very proud of the schools that we serve. Trish, more than anyone, works very closely with our sending schools and can talk about that experience. But there's no model in which our 9-person team could reach 2 million students by 2026 or 2027.

Fantastic.

Well, huge congratulations. I mean, this is big news, really big news. I mean, it's next level, right? You're— I mean, it's really— you've taken a proof of concept, built it over the last decade, shown, you know, just how impactful it can be for learners and for schools. And then, you know, that's being recognized happily. And as you've said, I think that's really useful, that laying out how different winds of change have kind of converged in some ways to allow for some of that. But, you know, it wouldn't be happening if it wasn't the proof of concept that you'd built. So, yeah. Huge congratulations. That's amazing.

Thank you so much. I'm not going to lie. It's very validating for the work we've been doing over these past years.

Yeah. And we need it, right? Globally, that's why it's so important because I think we need these proofs of concept to then be able to get to scale in different ways. So yeah. Trish, I'd love to invite you. How will this impact the schools and like the way that the schools interact with this? I mean, obviously it will emerge and evolve and we know quite know at this point, but how would you foresee or how would you like it to scale?

Thanks, Tim. I think one of the remarkable things— line is right about Scott Looney's original idea about a decade ago, but this is year 5 of the Mastery Transcript and the MTC Learning Record being in existence. We are in the 5th year of University of Michigan cycles. So this is actually remarkably fast in our experience for something to have this traction. And Mike really described how the— what's going on in the world and the work that we were doing were are really well aligned. I think in forecasting how we'd like to see this unfold over time, as Mike described, right, every learner having access to this. And I think what we've learned from the schools who've been doing this in this first wave and have been doing it very well is instructive. If you look at sort of original theories of backward design, right, that what you say are the ultimate goals of learning, And that goes way back many years and is very popular still in terms of designing curriculum, right? Those are the competencies. If those are transparent in a community about what those are, the backwards design now has moved to this wonderful phrasing of the portrait of the learner, the profile of the graduate. And folks are really thinking about that. What are the fundamental skills, the Cs that we were talking about, right? The new metrics, complex competencies. As we've watched schools define those, there is a lot of commonality and they make them very public and then they start to build courses, lessons, programs to ensure that students are taught, have opportunities to practice, mm-hmm, are really using those skills and get ongoing feedback on how they're doing and have evidence of their learning in all of those contexts. There's a lot of misunderstanding out there about there's these durable skills and then there's content, right? And that is a complete misunderstanding. You can't have teaching of these competencies, these durable skills, without deep, rich, rigorous content.

Yeah.

So the way we see this playing out in our sending schools is that there is a higher floor everybody must get to. Minimum competency, both in their content knowledge and their academic skills and in these other skills that are integrally related, as anyone successful professionally knows. And at the same time, there's a clear vision of how we're going to get there.

Yeah.

So the transparency, the other thing we know about learning is you need to know what you're going for. You need to care about what you're learning in order to learn effectively. So that means learners have much more engagement and much more say in how they do the project if they know what the things are that they're learning.

Yeah.

So we have more engagement, we have higher floor. There is no ceiling, right? There's no limit to how deep or how far. You wanna do graduate level work? Fine. So you raise the floor, you raise the ceiling, everyone's engaged. And what we're seeing in these schools is incredible rallying. Around individuals in the communities.

Yeah.

And it's working. And so the pathways then for those students as they graduate have also expanded.

Fantastic.

We're seeing kids very successfully being admitted to higher ed. I think today we're announcing 500 different colleges and universities have admitted students with Mastery Transcripts and MTC learning records. And many kids have gone on to apprenticeships and to sustainable employment.

Fantastic.

So we're having reports of all three. So I think that there's lots of good stories out there that we're eager to tell about these early schools and then to see those start to scale. Now, moving from a small set of schools who are doing this to large systems, that's also to Mike's point where we are thrilled to be part of organizations that have much more experience working at scale.

Yeah, absolutely. I mean, there's a huge amount of systems and capacity building kind of competencies, right, in terms of the way that that's going to need to rapidly scale across many different contexts, looking slightly different in different contexts. I mean, it's not a straightforward shift, but extremely exciting. Yeah, Mike, do you want to come back in?

Well, I think there was a moment where Trish was talking about that level of engagement, and it dovetailed directly, I think, with what Bill had touched on in your conversation with him. The way we would paraphrase is, or reflect back what I heard Bill and Sandra talk about, is that if you actually empower young people to make authentic choices about what they're learning, where they're learning, how they're going to demonstrate that gained proficiency, it not only creates psychological space for them to be open to and receptive to more and better formative feedback. They're invested in these projects intrinsically. They want the projects to get better. So this isn't just, oh, I'm doing something because you made me do it. I didn't want to do it in the first place. Then you told me it wasn't quite right. I don't really care. So I get my grade. I'm like, OK, whatever. I move on to the next thing. No, it's something I care about.

I chose this.

This is a project. I'm invested. I want it to be as good as it can be. It creates these conditions where you're open to feedback, but more importantly, you're making genuine, real choices. And that does two things. First of all, I think we undervalue the importance of making informed, difficult choices before we launch young people into higher education. And so you get this very strange paradox where you can have young people who are exquisitely prepared to apply to college, but are underprepared to thrive in college. Because high school, as we've set it up, is too often simply a matter of showing up and doing what you're told in a prompt, timely, error-free way, which, you know, is a kind of a virtue, right? But it's insufficient in and of itself. You know, when you're, you know, given a course catalog at a major research university where you can major in almost anything, right? There's so dozens of different disciplines. You've never actually had to make an authentic choice about what you're studying at all in high school. So it's a real kind of mind shift. Shift. The other thing too is that whether you're higher ed bound or whether you're going to, you know, take some time in an apprenticeship or some kind of like purpose year or, you know, where you gain some experience. If you allow young people to make choices about what they're learning and how they're demonstrating proficiency and to incorporate feedback and then do that again and again and again, you're also building out an amazing array of metacognitive skills, right? You are taking these things that were previously tacit, right? Some people are good team players. Some people are not, and we'll know it when we see it. Some people are leaders. Some people are not. And it's kind of this ineffable quality. No, these are skills. They can be taught. And if you let young people know this is where the bar is, and we're going to help you get there, every young person has the potential to demonstrate leadership, to be a good collaborator. Then you are building this metacognitive awareness. I will say, you know, Trish alluded to higher ed outcomes. We're incredibly proud of all these kids who frankly were very brave. Brave, right? It takes a lot of bravery to put your hand up and be like, yeah, I'll try this weird credential and apply to college with it. Why not? The reason they've been wildly successful is nothing really to do with the transcript. It has everything to do with the learning model and the kids in the learning model.

Yeah.

The first time you sit across the table from a 16 or 17-year-old and they tell you really clearly and articulately and specifically, these are the things I'm good at, these are the things I'm still working on, here's a piece of work I'm proud of, but here's what I'll do with it next. They're just so, it's several standard deviations better than your typical high schooler teenager where it's, what'd you do in school today? And I say this as a parent of two young people, right? So I've had plenty of those dinner table conversations.

Exactly.

That's the reason they're thriving, right? They've built this, this metacognitive muscle and this fluency about their own learning, mental learning models.

I don't know if we've described that the learning record and the transcript both have exemplary pieces of evidence attached, okay, of the students' actual work. So going back to what deeper learning is and the identity of the learners and their engagement, this public exhibition— we all know that public exhibition of our work is hugely important.

Absolutely.

And so this in and of itself also acts as a way for students to publicly exhibit their work as they share their mastery of these. And that's a huge piece, both inspiring the learning model on the ground in the schools, then in the applying for a job or applying to be admitted to college, all of that.

Fantastic. No, this is, I mean, this is amazing. There's so much I wanna pick up on in, in everything you've said. But I mean, if I could just go back a few steps just to the question of the diversity of the landscape that's out there, because I think Mike, you kind of described that in a few different ways. And I think that's for me is one of the things that's so interesting about this is that it, it allows for a kind of thriving of different ways of showing up, right? Different ways of showing things that you care about, the things that you're good at, things that you want to develop. And then to your point about apprenticeships, it allows a different way, perhaps then later on as the sector develops to allow different pathways to continue some of those things. So I think I find it really, yeah, really important that there's validation of the fact that this diversity is important in the landscape of the way that young people thrive. Because we, I mean, we're all different and we're all gonna thrive in slightly different ways and in different contexts and doing different things. And that's been obviously the, the criticism of the standardized industrial model. So maybe if you could speak to that a little bit in terms of how do you see perhaps the partnership with ETS and, and with Carnegie, how might that manifest in terms of increasing that opportunity for diversity across different contexts.

Yeah, I love that. One of the— I want to give a public shout out to our friends at the Big Picture Learning Network because I think one of the things that they did with their project about building the IB PLC, the International Big Picture Learning Credential, is they, they chose that— they made a design choice to say, hey, these are the skills that we're going to focus on, and then we're going to create a partnership partnership, you know, with Sandra's team at Melbourne to have a way of centrally validating and authenticating those skills, but without requiring students to take a standardized inventory, right? And that's why I think the difference and distinction she was leaning into very clearly, right? We can have standards for these skills that allow students to show up in a multitude of ways with a lot of different work product and experience that are acceptable inputs for that. And I think it was a very graceful implementation of that. We approached the work from a different way, which was we chose not to define 6 set skills, but rather embrace the sort of messiness of where each of these districts and schools were in their, their own kind of bespoke or artisanal competency models. But the advantage we have now, and this is true of both their work and our work, is that all of this is being powered by technology now. So whereas, yeah, as recently as 5 years ago, you might have said, okay, we've got Here are 4 different state competency models, state portrait of a graduate. Here's one from Iowa. Here's one from— I'm making this up, but here's Texas. How do we find the commonalities and differences? And so what you would then do is probably hire a graduate student and put them on a snake pen and say, here, spend a semester crosswalking these in some kind of spreadsheet. I would submit to you that that exercise is uniquely well suited to emerging technologies, specifically large language models, right? A competency model is nothing if not a highly structured and meta-tagged corpus of text. And that is something that's really well suited for AI and LLMs to look at. So I don't know if you're a fan of science fiction, but the Hitchhiker's Guide to the Galaxy books by Douglas Adams have this concept of a babblefish, right? This little thing you drop in your ear and it translates for you in real time. I think you can, we will see certainly before the decade is up, possibly in the next 2 to 3 years, the AI equivalent of a babblefish for competencies. Where you as a reader, maybe you're hiring for a particular apprenticeship program, Tim, or selecting for university, can say, look, this is the language we care about for these skills. Whereas right now what we would ask you to do is look at someone else's credential and the way we write and define those skills and ask you in your head to make an evaluation. Very time-consuming, very taxing, candidly pretty frustrating. And instead we'll have an electronic babble fish in between that will say, oh, you care about this? This is how it translates. Translates into what you're actually seeing here. But the other thing too is not just understanding the skill models, it's also verifying or validating the work. Paul Leather is a researcher and practitioner here in the States who had done some really interesting work in New Hampshire around creating what I would describe, and Trish can probably describe more accurately than me, but essentially describing an authentic performance assessment model at scale, right? So you might imagine, what is this going to look like? We're going to ask young people to submit an artifact of their work. We're going to ask them to defend it. We're going to have rubrics. You know, it was a really nicely done model. I believe it was called PACE. And the only problem with it, and it wasn't the quality of what was being done, it was very high quality, was the cost to train educators, make sure that you had inter-rater reliability and using and applying the rubrics, you know, some kind of cross-checking mechanism. As you scale those models, they got more and more expensive. I think I think in the near future, we're going to see models where you can have an authentic performance assessment paradigm where a young person shows up with a, a deliverable, a piece of work that is not a response to a fixed set of prompts. It really can just be something they're working on. And we will have, for lack of a better word, an AI copilot that can help an educator look at that and accurately and quickly assess which skills are being displayed, which skills are opportunities for further growth and development, and to give them almost kind of a very quick kind of MRI or CT scan of, of what they're seeing in that. Yeah. And, and do it just as reliably and accurately and really reduce that overhead and increase the validity. And then the beauty of those models is that every time you do that, what you've actually done also is collected a piece of evidence inside this growing database. And so what used to be inverse models where the bigger the network got, the more costly it was and harder to maintain quality. Now the relationship has been completely flipped. The more data you collect, the more accurate your models can be.

Oh, gross.

Yeah.

And you can train and get feedback on them. And then the more diverse and flexible they can become in the kinds of skills that we're looking for and the kinds of artifacts that we can accept on the inbound side. So I would say authentic performance assessment at scale for us is the opportunity area.

But that's fine. Fascinating. And yeah, because it allows all the different ways that people show up to be looked at and validated, but it kind of goes back to one of your very first comments about almost the kind of the need of convenience to, to make the choice to, to go do the university thing or do the standardized test because we can't quite cope with the translation issue of, well, I'm not quite sure how to compare that thing that's being done in this context. So if you've got something like technologically that's supporting that translation work in a really robust way, bearing in mind the biases that could be built into the algorithms in that. And, you know, we need to acknowledge that and work on that. But really, really interesting.

Yeah, it's an imperfect analogy, but, you know, I think we're all familiar with CAPTCHA, right? You know, that sort of model that's designed to, you know, check and see if you're a robot or not. And the way it worked in the early days really was, was essentially a Mechanical Turk, right? It was taking images or bits of fuzzy text and it was breaking them out for humans to validate, right? And the purpose explicitly was to train, you know, the engines so that someday they could look at a picture and identify all the motorcycles without needing a human in the loop, right? And I think we're getting to the point now where we can see that the large language models and the other AI generative AI models now are very good at it, but it requires a lot of human input. I think over the next couple of years, you're gonna, there's an analogous thing where we still need educators. We absolutely, you know, Education is relational, right? This is not about sort of putting, you know, young people in front of a learning bot, like, and having learning happen. It's people in classrooms, in the field, you know, in apprenticeship and internship settings, getting rich feedback about real work. But then using that and collecting that and storing it so that the next wave of evaluations can be that much easier and require that much less overhead.

Yeah, absolutely. But I suppose the thing that excites me is that it's removing some of the blocks because, you know, historically the assessment has been, I mean, we, Sandra and I joked about it on our very first conversation. She calls it the evil tail that wags the learning dog, right? The assessment. And it's always been the reason why teachers will, this is unfair, an unfair characterization, but the reason for not changing is because of the assessments, right? So they're funneling through the assessments. So things have to be like this. And I always say that, you know, we talk about inertia to change in education, but if you change the content of the exam, right? You know, the exam assessment criteria or whatever. One year to the next, and teachers will shift in an instant because suddenly the assessment has shifted. So they know that that's what they're gearing towards. What you are doing is removing a block there to allow them a much more generative open space, which is really exciting, but also puts the onus back onto the learning model that you were talking about, Trish. I think that you actually need the educators even more now, not less, I think.

Yeah.

Because you, it's so much about out what are the conditions in which this learning is happening? What's the learning model in the context of the school in which these opportunities are being given to young people? This is not about giving kids free rein to do whatever they like. And then as you say, putting them in front of some kind of assessment regime at the end of it. This is a very carefully structured and guided process from, by, and with the adults in the room, right? And so I don't know, Trish, maybe you could speak to that because I see that as being one of the interesting challenges of scaling this work is because it needs quite a significant and important nuanced set of competencies in the educators who are going to hold that space to allow those things to develop.

Absolutely. And I think that's again where we have some really good data from our early adopters of this. And I think this is good news. If there's a situation that needs to be proves the recognition of the professionalism and the creativity and the intelligence of every single teacher and the need for ongoing professional learning and feedback and validation. So the role of being an educator should be very analogous to the role of being a learner, that there's a set of competencies that you are valued for those, that there's ongoing feedback because none of us are ever done at being a perfect teacher, just because we're never done at being a perfect communicator. It's constant learning. That then lifts the recognition of the work that teachers are doing every day, the work that they're doing designing learning experiences, and it focuses on that need for the relationships with the learners and knowing them personally. We also know that every learner known personally in school is how things work best. That's about identity. That's about support. That's about love. If we can make more time for that professionalism and that relationship building, we have dramatically improved the learning and I think the model of education.

100%.

And we are seeing that in the schools that have adopted these, these truly competency-based systems.

Amazing. And to Mike's point earlier, that's going to help with the recruitment crisis, right?

Because who comes into the profession, who stays in the profession longer, and how do we support that around our children? That is an ideal that I think also is within this model that we haven't talked about as much.

Yeah, no, that's, I, I think that's really exciting. A much more enriching profession for the professionals involved, right? And there's an interesting, I shared with you the link to the Rick Hess article about mastery and it was, that was an interesting piece because I mean, he's talking about the different ways that this needs to be part of the learning model and it's, it's a challenge for some educators to understand and shift towards this mastery. Based approach, but being clear about what students need to learn, fine. Having valid and reliable ways to measure, fine. We're like, we've kind of talked about those things and those are being dealt with in, in a lot of what you are doing and, and what Sandra and Bill and others are doing. But his third point I found was, found really interesting that sometimes there's a bias towards the negative of like the young people who are not meeting the standards that we set for them. Right. And I think this is a, it's an interesting tendency that we have, oh, you know, the, the young people given this opportunity in space won't meet our minimum requirements in reading or in maths or in whichever. And I think there's something there as well about the baggage that we're carrying with us, because Trish, back to your point about raising the floor, not, you know, this is not about holding young people back if they don't meet the standards that we're setting for them. There's something else going on that I think it requires a bit of relearning on the part of the policymakers and the educators and leaders involved as well. I don't know if you had any thoughts about that.

Our colleague Susie, Susie Bell, who's our Chief Program Officer, is very clear all the time about saying that you have to go into this work with an expectation that every single learner is able to meet the standards.

Right.

That doesn't mean they're all going to meet them at the same time. It doesn't mean they're all going to meet them in this exact same way. It doesn't mean the assessments or the work products are all going are going to be identical, right? And it doesn't mean it's not complicated. Yeah, it takes deep professional work to support learners who are facing different kinds of struggles and learning challenges and developmental challenges and mental health challenges. All of that is the complexity of, of working with kids. But if you don't go into this with this basic understanding that everybody can learn and everybody can meet these minimum standards, then you are not doing well by our society or our children. So I think that's a shift. There are, and going back to your original point about the industrial model, it is not a zero-sum game. There are not winners and losers. There's lifting everyone.

Right.

Yeah, absolutely. Mike?

I just wanna say, I thought Rick's piece was perfect. I mean, I think he actually nailed the challenges quite well. And, and I think one of the things that is kind of interesting is he, his, you know, second paragraph is really about what kinds of assessments are we gonna need for this to work? Work.

Sure.

And, and he's correct. This is a big challenge. I'm loosely paraphrasing, but we live in a world right now where there's two kinds of assessments. There's assessments that are valid and reliable, but they are not repeatable and the feedback is not timely, right? And then you have formative, authentic assessments, which you're getting feedback in real time about the kinds of inputs, you know, that are closer to the actual learning. But we worry that they're not valid and reliable. So the question is, how do you marry those two? How do have something that does both. It's really interesting that because we had— we need validity and reliability, we've had to make sacrifices, right? We— you can't repeat most tests right now. They cease to be valid if you allow a student to retake them, which is very ironic because that's the essence of learning, right? Trying something, realizing that you haven't nailed it, and then working to get feedback. I'll give another shout out. One of our partner organizations is called CogX., and the founder of that, Javier Arguello, it's all about the sort of the actual science, cognitive science of how young people actually learn, how their brains work. And, you know, he says quite correctly that if you're doing this the way brain science tells you to do it, the first thing you should do is administer some kind of assessment, right? Not just for diagnostic purposes, which most educators would agree with, but actually it primes you to learn more effectively when you encounter material you don't know. It creates this response to say, oh, I didn't get this. What's going on? On. There's hardwiring in our brains. We don't do that today because it's expensive to make multiple assessments because, oh, we can't do that. If we use one, we're just throwing it away for diagnostic purposes or to help prime the brain, and then we're gonna need another one, another one. What would it take to build something that's repeatable? I think one of the, the real insights or innovations of Khan Academy and the reason it got deserved praise when it was teaching kind of like math instruction is that it realized like, oh, you can use algorithms to generate an infinite number of varied problems on a particular skill.

Yeah.

And so it's fine if a student needs to take 200 attempts before they can get 10 or 20 in a row, because the number, you know, is infinite. I think that's harder to do with other subjects, but the underlying principle of use the assessments as part of the learning, not as something that is bolted onto or, you know, ancillary to the learning.

Yeah.

I mean, I would just love to finish by thinking about about somebody listening to this, you know, as I say, all over the world listeners are tuning in and they're in a context where this is perhaps not standard practice. What are the different ways that they might engage? Because I hear you, obviously, you know, there's a lot going on in the US, but all around the world, what would be the invitations that you would make just to kind of to explore this a little bit more, to dip their toe in the water if they want to spend a bit of time with this?

I think that starting to think about what a profile of a learner is, within the learning community is a perfect first step. MTC would love to have anyone interested come our way at mastery.org and look forward to working with them. And I think one of the things we've found is that we all have learned so much from the geographic diversity and the context diversity and the type of school and learning organization diversity that really has helped in all of the design and all of our own self-assessment work of this organization and this movement with both at MTC and with an ecosystem of partners and now in the enlarged ETS Carnegie context. So I would say join us.

Fantastic.

I would also say to start with the Aurora Institutes and start to spiral outward to other players in the ecosystem, you know, whether it's Bill's work or Sandra's work, start by looking, there are interesting reference models. There are schools that are actually doing this. And if you look at them, it will help you, I think, do an essential thing, which is to, I would say, find your why. It's not enough to be curious about this to be able to drive change in a system locally or nationally. You have to have a very clear sense of what is motivating you to do the extremely hard work of change. We know what that looks like for ourselves. We know what that looks like for many of our schools.? And the value of it is the answer varies quite a lot. We have some school leaders that come to us because they lead schools that are frankly exquisitely high-performing on all the standardized tests and college outcomes. But the young people are, for lack of a better word, suffering. They're not well. We also have some leaders that come to us that lead systems that have historically been abysmally performing. And they've been given a mandate for change. So you've got two completely different groups of learners, two completely different kinds of leaders, their motivations, their drivers, what they're trying to achieve is very different. The joy for us is that they wind up in the same place. They wind up at our doorstep. But you need to know sort of what that motivation is. So that's what I would say.

Fantastic. Well, thank you so much for taking the time today. Huge congratulations on the massive step you're taking. And yeah, very much look forward to supporting and sharing as much as I possibly can to, you know, get this happening all across the world. Amazing.

Yeah, thank you for taking the time. Thank you, Tim Logan.

Really appreciated the invite.

We hope you've enjoyed this episode. Please feel free to continue the dialogues with our guests, with us on our blog or on social media, or within your own networks.