Assessing Complex Competencies - A Conversation with Prof. Sandra Milligan

Welcome to the Future Learning Design Podcast.

Certainly the mass delusion is breaking down a bit, so universities are increasingly looking at a broader array of things. And this is where the idea of a profile of general capabilities comes in as useful. And as soon as it starts to be useful, people will start trusting it more.

Hi, and thanks for joining me on the Future Learning Design Podcast. I'm your host, Tim Logan, and the podcast is brought to you in partnership with Intrepid Ed News. This week I'm talking assessment and complex competencies with Enterprise Professor Sandra Milligan, who is the Director of the Assessment Research Centre at the University of Melbourne. Sandra specialises in research on assessment, recognition, micro-credentialing, and warranting of hard-to-assess capabilities. And she is the lead on the New Metrics for Success research project. Sandra is also the Director of the Australian Council for Educational Research. She convened the Melbourne University MOOC targeting professional learning for teachers in the area of assessment and teaching of 21st century skills, which has to date enrolled over 30,000 teachers worldwide. Originally a teacher of science and maths, Sandra trained in educational measurement and is a former Director of Curriculum in an Australian state education department and has held senior research, management, and governance positions in a range of educational organisations. Sandra co-authored the Assessment Research Centre report Future Proofing Students in 2020 and is a contributing author to the book Reimagining University Assessment in a Digital World. Hello, good morning. Thank you, it's fantastic to be able to talk to you about this stuff. I, um, have an interest in assessment because I sometimes feel like it gets a rough deal when it comes to talk about teaching and learning. You're always talking about teaching and learning, but never quite getting to assessment. And then often the assessment gets then blamed for being the reason why we can't make rapid changes in education. It's always—

Well, let me just make a comment about that, because actually I agree with you that the vast majority of time assessment is the evil tail that wags the learning dog, right? And as an assessment academic, I get frowned at quite often for those purposes. But actually, you can flip that around because the fact is that assessment is very powerful. So if you've got good assessment, it can actually be the powerful source of good for teachers. And that's where we basically come in. So that's my life. I'm trying to get assessment as the power for good, for learning and for kids.

No, I completely agree. I mean, it's such a powerful lever, isn't it? When we get to it. But what I'd love to do is talk to you a bit as an introduction about some of those general competencies and complex capabilities that you talk about, because I think it feels to me like there's a bit of an arms race almost for everyone around the world looking for how can we nail down these, these really complex competencies that we, we want to if we move away from just pure academic learning and assessment. But I think that's a really big— obviously you have this all the time, a very big and complex conversation to have around what those things are., and therefore how do we begin to measure them? And even if we, if we actually want to go down that route fully. So I'd love to talk to you about all of those things if possible. But firstly, one of the things you, you said in a great keynote I saw you do was the general capabilities and complex competencies are embedded in our curriculum aspiration and are here to stay. Clearly, you know, you are making the statement that these are not going anywhere now, but I think there's a lack of understanding sometimes about what these things mean. So maybe we could start there, particularly the complex competencies. What does that mean? To you, and then how might we begin thinking about that and breaking it down as we then move into a talk of assessment?

Sure. So people often do ask me, you know, what's the definition of a complex competency or a general capability? And I don't ever just jump in and give a definition because the way we came to an understanding of what these things are is working with schools that have picked up curriculum documents and they've said things like, hmm, this curriculum says we've got to teach kids to have agency or to collaborate or to be good citizens or to have ethical understandings or to be good communicators or whatever. And so that's where we started, and we call these things general capabilities or 21st century skills or whatever. Then people said to us, okay, well, we think we teach that anyway. You know, we do teach kids to collaborate, we do want them to be ethical and we demand that they're ethical. But now people are saying we've got to actually set things up so that students definitely learn them and also maybe assess whether they've got them. And that's the point at which people start wondering, what are these things? How do you teach them and how do you assess them? And that's how we've come into it, working with schools. So the first thing we discovered, of course, is that these things don't exist in nature. You can't lift up a rock and find collaboration there. These are sort of human constructs that have been invented to talk about something that we share an understanding of broadly. When they become in the curriculum, you've got to pin them down a bit more. So to pin them down, we've worked out that first of all, they are complex. So if you take something like collaboration or communication, it's actually an amalgam of attitudes, values, beliefs, and skills that a person puts together that enables them in a particular time or place to communicate, to listen to someone, to respond to them. So the first thing that we've done is try and codify what that might mean so that everyone understands it. We've got a set of materials which we call Ruby Resources, with Ruby being derived from rubric.

Okay, yeah.

So Ruby Resources. And for each of the commonly used general capabilities, we've fleshed out what are the attitudes, values, beliefs, and so forth that make it up. So that becomes a definition of a capability. And because it's quite complex, we call it a complex thing. And because we're looking at behaviors, not psychological constructs, we call them competencies. So a complex competency is just a collection of things that we've put a lasso around, if you like, in a way that makes sense to everyone and tried to tease out what it might be so that everyone understands it. In our own work, we work with a set of about 10 of these complex competencies or general capabilities. And you'll usually find around the world that people have got much the same clumps. And so most people, our clump appeals, but we've done very detailed work about what is it comprised of, what attitudes, what values, what beliefs, what behaviors. And we've done it in a way that makes them accessible for teachers to be able to assess and recognise whether kids have got them or not. Interesting.

No, that's great because I think I would completely agree, and I've had this conversation many times, there's a lot of imprecision around the language of these things. Everyone talks about, you know, they've talked about 21st century skills for the last however many decades, but what that exactly means when you really drill down to specific detail of an interaction in a classroom or, you know, a teacher working with a child in a classroom there's a lot of imprecision and fuzziness there about what that actually means. And I think there's different agendas, people have different favourites, or the schools build them into their visions and missions. And, you know, these are our particular outcomes that we want in this school to differentiate themselves. So it becomes marketing as well. So there's a lot of agendas going on, I suppose.

Just our work is based entirely on people or organisations, mainly schools, but school systems and networks coming to us and saying, can you help us with this particular problem and it's usually something like the school's committed to these 6 capabilities and now we want to see if we are doing anything, how do you assess them and so forth. Now we sometimes say no, we can't help. So, you know, it's easy enough to say what general capabilities are but you also need to say what they are not. And this is probably going to be controversial, and I'll probably really annoy some of your listeners when I say this. But, you know, sometimes people mistake personality traits for capabilities. So no, we're— our general capabilities or complex competencies are things that have to be able to be taught, learned, and assessed. Otherwise, We don't touch them. Another thing is that, like, wellbeing is a really good one, or self-esteem. Wellbeing or self-esteem is not a capability. It's a state that you are in. So when people say, can you measure a student's wellbeing? We'll say, yes, someone could measure the student's wellbeing, and that's probably a good thing to do, but What we're talking about is something that's teachable, learnable, accessible. Wellbeing is a status that you have attained as a result of things that will happen. If you want us to assess wellbeing literacy, in other words, the skills a student has that might help them overcome things in their life, we can test wellbeing literacy, but we don't want to be in the situation of assessing a student's well-being. We're a bit picky about what we define them as.

That's brilliant, because I mean, you have to draw those boundaries, don't you? Because I mean, in your work precisely, it's about the boundaries and which things fit into which category, and therefore, because otherwise you can't actually begin to measure them either. One of the things you said that was interesting, particularly I think there's been a blurring of that boundary between what's a capability and what's a personality trait with ideas like mindset. I think where you end up often we're asking young people or, you know, educators or all sorts of people to integrate these kinds of ideas as that were learnable capabilities into who they become. So you become this thing and it, you know, you adopt a growth mindset or a benefit mindset or whatever it is. And I think there's, there's an interesting, again, lack of precision around the language of what exactly that means. Is it something you do? Or is it something you are as a person? And if so, I mean, there's some moral questions about testing those or, and, and that there's also a contextual question as well for me, you know, as you say, wellbeing in one context or even mindset in one context, you may have a particular view of the situation or, or feeling about who you are, you know, in terms of your wellbeing in a different context, it's completely different. So the idea of testing that in some kind of vacuum or laboratory, you know, is also a bit problematic.

Yeah, I think, I think a big part of our work is getting people clear about what's the curriculum core here.

Yeah.

So, you know, we're talking about teaching, learning, and assessment. So really it is bounded by a view of what the community thinks are the teachable, learnable, accessible things, and trying to clarify that in a teacherly sort of way.

Yeah, interesting. And I, I also enjoyed, there was a comment in your report, the Future Proofing report, about the fact that actually a lot of these things in terms of assessing complex capabilities like this are not new to teachers, cuz you've got teachers like sports or performing arts or teachers who have been trying to measure or assess complex interactions and capabilities in a particular context, you know, they've been doing that forever in performances or events. So I think it's, again, it's just trying to distill out explicitly maybe some of the things that some teachers have been doing for a long time.

Yeah, that's right. I'm very fond of saying this. People will have heard me say it before, but, you know, I bet if you were talking to Plato or Vygotsky or Dewey or any of these people, they'd recognise exactly what you mean by these sorts of competencies or capabilities. What's changed is not that they're not those as human characteristics that can be taught and learned, but that schools are now expected to sort of assess and prove and make sure that they're not just developed through osmosis. And so there's an intentionality behind it for everyone. You know, people say to me, oh, you know, our top kids, they already developed that kind of stuff anyway. We don't need to assess it and be intentional about it. But in fact, when you look closely, some students don't. And unless you're intentional about it for all students, you can't be sure. So I think that's what's changed. There's an intentionality about it now that there probably wasn't.

Yeah, interesting. And, but I guess there's, there's positive and, well, whether you put normative values on it, but there's different impacts of that, aren't there? I mean, there's an accountability agenda because clearly we want schools to be doing what they say they do. And if they're spending public money or et cetera, you know, that's, that's a good thing. But then there's a, as I said, a kind of an arms race of who can measure these things most effectively to then commercialize them in some form, and maybe, maybe as an impact as well. But, um, if we then go into the assessment piece specifically, just because I think as we say we're going to assess these things, and I think we, you as teachers, we all use that word all the time, and it takes many different forms. And one of the things I really enjoyed was your poem about assessment, which I, I actually, I'm going to be honest, I preferred the colourful version in your keynote that I saw versus the one in the report, because there's something really powerful in that. And one thing in particular that I really liked about it was the idea of trust at the end. But in general, I thought it was a really clear description of the process of assessment. You know, what are we looking for?

Yeah, it's funny where that poem came from, because, you know, I've been talking for years and saying, look, most assessment is based on the philosophy that you should be testing the degree to which a student has mastered the cognitive skills that you have taught them in your class. And saying, look, there's more to assessment than that. We should be trying to pick up in a holistic sort of way whether they have learned the things that will enable them to thrive as workers and citizens later on. And that's, I think, what the general capabilities are doing to the curriculum. They're broadening it out so that we're looking at the full things that thrive. But when you're talking about assessment, and I'd go through all the definition of things that the new kinds of assessment would have, I could see people's faces glazing over because I'd be talking about things like standards-based, referenced, and developmental and performative-based, and oh, everyone was sort of going gray. So I did the poem and It's, I think it's about 15 lines, center-justified, colorful, that has got the characteristics of what assessment is, which is basically collecting evidence to make a judgment of where on a position, a scale of competence from novice to expert, a student is so that you know what they know, what they've still got to learn, You can place them there in a way that someone trusts that the positioning is good for the purpose that you want it for. It's that simple, but it took me years to get it like that. So my poem, I love my poem.

It's the wisdom, isn't it, that comes from years of playing with more, maybe more technical ideas, but actually you distill it down into the pure essence of what assessment is. I thought it was brilliant.

Thank you. I'm very pleased to hear that.

But particularly if we can just talk a little bit about that idea of trust, because I think that is one of, I mean, we've referenced it a little bit maybe with accountability and these things, but there is a role there for assessment, which is about can we, whether we means parents or governors or government or teachers themselves, can we trust the outputs of these assessments? And therefore that's, you know, because again, particularly with these ideas of complex competencies, if there's a bit of imprecision in the language and then you're doing context-based assessment and performance-based assessment and it can be quite subjective, etc., etc., there's any number of caveats as to why the trust in the outcome of that assessment may be somewhat compromised. So I don't know if there's anything there about that idea of how much trust do we need, how much confidence, because we're never going to get 100% validity and 100% confidence, right?

That's exactly right. Look, it's the two words I use together are usually trust and utility, because what you trust something to be fit for a purpose. Now, if you take in Australia at senior secondary level, we trust that we can put every student in Australia on one single ranking queue from 0 to 100 at 0.05 intervals, right? We trust that we can do that. It's rubbish. We cannot do it. But the reason it's trusted is because no one's sort of cheating anyone. And it is very useful. It has high levels of utility for the institutions. So it is faux precision. You cannot make that sort of judgment as precisely as everyone pretends we can. So you've got a highly trusted, falsely set of numbers that are quite useful to a certain proportion of people that's trusted. Now, when it comes to general capabilities, the question of precision is important. So I think if you like, on the Australian Ranking, there's probably 1,000 points and we're saying we're dividing kids accurately into 1,000 points. With the general capabilities, you only need 4 or 5 levels. You know, you can tell the difference between a good team player between a novice team player and an expert team player by reference to 4 or 5 levels. You don't need huge amounts of precision. So what we're looking for in the assessment of general capabilities is that you can accurately position a person at one of these 5 to 8 levels, depending on how many you want, and that people can believe that if a person is at level 3, they're a better team player than the person at Level 1, and a person at Level 5 is much better, and that those levels represent some meaningful interpretation of teamwork as it's found in the real world. Now, that's what we want, and that you can use those levels for some purpose or another. Now, in a school, the purposes are usually to scaffold kids with their understanding of how they're going, perfect, that's what you need. Often now those sorts of things are being used by employers and tertiary institutions for selection, but the tertiary institutions are not saying, are you good at teamwork at the 54th percentile? They're saying, is teamwork one of your things? Or are you a person who works on, on your own? And a 4 or 5 level thing is perfect for people to get a match between what they want and what they have. So what we're finding is that increasingly, as people understand what these levels are for, that they have some relationship to how that person behaves in the real world. And that they can trust that if they want a really good team worker, they can trust that Level 5 will give it to them. That's what trust and utility is, and that's what we're sort of shooting for. We're not shooting for faux precision. We're not shooting to be able to rank people from 1 to 1,000 on teamwork. We're seeking to characterize a person on a range of these things so that you can see by and large what they're like. So you'll get students with very bumpy profiles. Some students will be brilliant critical and creative thinking, not so hot on collaboration, not so hot on citizenship, but hugely into, you know, innovation and agency. And you can see the profile and you can see people saying, oh, I get what sort of person this is. That's great. That's what we're shooting for.

Yeah, brilliant. I mean, that elephant in the room of the faux precision in education is just— I mean, I live in France and here it is absolutely rife. I mean, down to, as you say, down to 0.25 of a mark out of 20. I mean, no human being, in my view, can mark anything to that degree of accuracy, and to believe that that's possible is a phenomenal delusion, I would say, not to put it too strongly, but, but it, I don't know whether it's because it's quantitatively done or we will much more readily criticise or question the subjectivity of an assessment of, as you say, someone's collaboration or someone's teamwork based on a 5-scale rubric than we will someone who's marking a particular piece of work out of 100 or down to 0.25 of a percentage point. And there's a deep bias in there, I think, which is challenging to deal with.

I tell you, I'm a bit confident about this. You talked about the delusion. It is a delusion, but I come back to this utility. It's a utilitarian delusion. It's useful. But what I think is happening now is that the utility is fraying. Certainly in universities in Australia, for instance, the mass delusion is breaking down a bit. 'Cause people are finding, for instance, that those marks, while they do predict who will do well in marks in first year, they do not predict who will do well in the course overall and who will succeed in the profession. And while high marks predict high success, low marks do not predict failure. So that people are starting to realize that if you get a lowish academic rank, That does not mean you're going to fail. It just means we can't be certain you're going to succeed, which is different. So universities are increasingly looking at a broader array of things, and this is where the idea of a profile of general capabilities comes in as useful. And as soon as it starts to be useful, people will start trusting it more, I think.

Yeah, no, I think you're absolutely right. That's a really useful addition to the conversation around trust as utility. But as you're talking about those developmental learner profiles, one thing I would also like to ask you is, do you see the spiky profile that you've just talked about being valued? Because obviously there's a million different contexts where this happens, but I think there's a really interesting thing in the, in academic learning, tertiary institutions are looking for the straight-A student, let's say, just to simplify it. The idea of a spiky profile is that different people have different, as you say, strengths and weaknesses and areas for development and whatever language you give them. And that's fine because that's who we are. We're all unique individuals. And the idea, as soon as we get to a point where the selection criteria, et cetera, comes from are not a spiky profile anymore, but some fictional idea that's, that the top student is perfect in all of these attributes. That's when I think it becomes really problematic. And that's where the gaming of the system happens. Because actually, if we can just retain this idea that every human being has by definition strengths and weaknesses, and the idea of average in all of them or top in all of them is somewhat of a nonsense. What would your views be on that?

I guess I'm still working out my views. Like, if you want to get into Harvard— I don't know about Oxford— but if you want to get into Harvard, you more or less have to be an Olympian orchestral conductor who can do physics. And, you know, you've got to have the lot really to get in. And I think unfortunately that that sort of super duper up there thing has really warped our view of what really happens in the real world. I was talking to, you know, Melbourne University where I'm from is, well, we say the top university in Australia. My colleagues in other universities might not agree, but we say the top. We've got a very prestigious med school, so people are keen to get into it. Now I've talked to the selectors there. They say, look, most people who get in there at the 99th percentile of cognitive achievement. And I say, that is not necessary. You only need to be probably up to the 80th percentile to be able to handle the cognitive load for medicine. What you need is to be extremely persistent, to be prepared to work your heart out, to not faint at the sight of blood, and to have some sort of human empathy running through your veins. If you've got all of that, you'll probably make a brilliant doctor. But at the moment, it's the 99th percentile that gets it. So they get a lot of unhappy people who just don't fit. They don't want to waste their marks. So even in a prestigious area like that, I think there's— it's loosening things up. And there are already, I think, 30% of school leavers, for instance, in Australia, who get into university on bases other than their academic rank. And universities are looking for clues as to what they should use. That's why I think that spiky profiles are fine, because people are spiky, so there's a truth in it. And if we can find a way to capture that truth in useful mechanisms, we'll be fine.

Yeah, no, that's good. And I think the rise of things like Developmental Learner Profiles and Mastery Transcripts is beginning But I think we also sometimes even just evident in the amount of time we've spent talking about it, we need to be sometimes careful not to let that narrative define everything else that happens in school or anything else that happens with learning generally in whatever context. It's not just all about competing for those scarce places in these apparently top universities. You know, there's a lot of other things going on about human development and learning, et cetera. But yet maybe then if we can just move on Slightly broader then, because I think one of the things you are very involved in and really, and I'm really interested in this, the rise of the kind of credential ecosystem. So that, and actually that's challenging the dominant narrative of the universities with just prestigious bachelor's or master's degrees because things are becoming unbundled. You know, you might go to one institution for one particular credential or some of them online, some of them face-to-face. Because obviously that's connected to this idea of assessing different competencies, work-based or learning-based competencies. Where do you see it is right now? Maybe you know Australia more than internationally, but what's your sense of— because it's still in a way, it's still early days in that world, right?

Yeah, I think that it's very early days. One of the things I notice is that many, many countries around the world are really rethinking their credential frameworks. Because of the microcredentials or short-form credentials, as they're sometimes called. And by and large, they're trying to loosen up credit arrangements and recognition arrangements to give people more flexibility to mix and match and things like that. And I think that's happening, and you either like it or you don't. I think if you run a big long course, you don't like it. If you run a short course, you like it, you know. And I think students and learners are finding the flexibility great, and certainly it seems to be driving down the cost of developing skills and so forth. I think it's also turning up in schools. We've noticed that schools are often getting in experts, like in computer teachers and firms that do these kinds of things, who issue badges or micro-credentials or whatever you like to call it. And they increasingly want to include these in, say, the senior secondary qualifications. So I think it's having a big impact, but I think it's too early to say how. I would say, however, just sort of to revert a little bit, Tim, to what you said before, everyone talks about credentialing and assessment, as you said, often, you know, the recruitment and the employer view and the tertiary institution view, overwhelms everything. But whenever we do a project like this in a school, it is always, always the case that the teachers and the kids say, what we really liked about this is that we got the developmental progressions and we could see what the learning was supposed to be and we could talk about it. And the students would say, I knew suddenly what I was supposed to be doing and I could talk to my teacher about the evidence that I I had. And so it's at that level where the real power is, although I agree with you, we're often left talking about tertiary selection and employability and so forth, when in some respects, I think they're derivative and second order. But that's where we are. And microcredentials are part of that. I sort of think it's, in some senses, derivative and second order, but they do have utility. And so we will need to take them on board. And we're increasingly trying to recognise them in credentials that we issue.

Yeah, because I mean, I completely agree with that. I think the, the idea of those rubrics and the openness of the assessment, you're opening up a black box that's normally just in the teacher's head, right? I'm going to give this mark to this and this, you know, actually there's a conversation that can happen because it's out in the open, it's visible, and there's language and common language around the progression. Just on the— so for example, you might be having running a project And then one of the outcomes of that project might be a digital badge or micro credential on a particular competency. Do you see that there's a— I hesitate to use the word, but rigor, because you sometimes get this idea that, and again, I mean, it's a little bit of a caricature, but the big IGCSEs, A-levels, you know, ATAR in Australia, right?

Yeah.

Exams. Those are the ones with the real kind of clout and rigor and weight whereas these microcredentials, you know, they're nice to have, but they're still a bit frivolous around the edges, padding other things out. Obviously there's a conversation going on around shifting that balance, but do you see any examples of that actually happening where, for example, a digital wallet full of microcredentials will carry the same weight academically and all, you know, in all those different categories as those previous big qualifications that they might be looking to replace?

Yeah, that's a really good question, and I've got a position on that that I'd have to say is not evidence-based at the moment because it's too early. But, you know, portfolios and backpacks are pretty soon going to silt up with all sorts of things, and no one will know where is the truth. No one will know the truth about what they're saying about the student. We're seeing that already. So there are a lot of people who are issuing badges, credentials, certificates. Kids love them, they put them in their backpack, you know, got a lovely long list. What does it mean about what that student knows and can do and who they are? Nobody knows. Alright, so I'm suspicious about that. However, you can tell the difference that it is possible to make a rigorous micro credential. It's not rigorous in the same way that an examination is rigorous, but the assessment criteria— if you go back to my poem, the poem has the essence in it of what rigor is in this sort of thing, and it is having very rich performance evidence, so have the students performing complex tasks that get them to display it. It's based on expert judgment and teachers' expertise can be trusted here, but it probably should be aggregated with other forms of judgment, so the aggregation of judgment. Sadly, you can also use statistical methods to tell all sorts of things about the reliability and validity of various judgments, if the quality of the progression and the things you're asking your judges to look at are carefully developed, it makes it good quality. So we've got about 10 or 12 criteria for rigor in developmental assessment that you can look at, and that's why we like profiles rather than portfolios, because if someone's curating a profile you can trust that they've tested to see whether the assessments are rigorous and you can, back to our trust and utility question, you can trust that this profile does represent what the student knows and can do and is like, it will therefore be useful and therefore you can go there, so I think the backpacks and so forth are going to get less trusted but a curated portfolio based on rigorous assessment will get more and more trusted. And my prediction is that in 20 years' time, we'll look at GCE examinations and say things like, gosh, did we rely on those things at that time? Goodness me, how amazing. So, you know, in retrospect, we'll look as though we were mad.

Yeah, interesting. Well, I think there's a lot of educators out there hope that that becomes true. Sooner rather than later. But yeah, it's interesting. And as you say, the pandemic's kind of accelerated some of those questions because it's— we've had to figure out what life is like without them because that's been forced upon us, right? I just— perhaps if we could just finish maybe on, on the optimistic note that we, we tried to start with, in a way, on the fact that assessment is overlooked often, but as you said at the beginning, it's that it can be a really useful lever for change, or I think you said tail wagging the dog, but in a useful and strategic way to bring about positive change. Is there anything there you'd like to say about where, you know, perhaps if you see positive examples of that happening or where you think we could focus our attentions a bit more on this assessment question just to perhaps speed up that positive process?

All of my views really, my understandings are all secondhand from my wonderful research partners, that is schools, school systems, and networks who work with, have worked with us, and many of them still working with us even after 10 years to work this through. So I think all of us would think this is a work in progress. In our experience, there are some cases that are miles ahead. There are some people in our research partnership who are really going very fast and far. There are some who are, you know, partway there and some who are just starting. So there's this sort of trajectory of change that each individual school or school system goes through, and it takes years to really get the learning design and the assessment design and the new learning ambitions and the credentialing and reporting all aligned It's that troika alignment again. My optimism comes from the fact that all of our partners, and it's sort of hundreds of organizations, first of all, they're all sort of heading in much the same direction. So it's like there's a validation of the core ideas. Secondly, they're getting faster and faster, and some are already there, and they can act as sort of light, lighthouses or beacons or evidence, not for people to copy them, but to see that it is possible. And every year we're learning more and more, and there are more and more people who are getting on board. So that's why I'm optimistic, because all the good things are heading in the same direction. If I had to say I was happy because 'x percent of kids in the world are now benefiting from these sort of thinking,' I'd probably be less happy because we're at early days. So I'm looking at trajectory and movement at the moment, not scale and scope, but I think they will follow.

Yeah, that's great. And I think you're right, I like your phrase 'silting up.' I mean, I think one of the things I see from your role and that, you know, the Assessment Research Centre and people like you doing this work is to help people have that precision in their language so that it doesn't get all silted up with people just doing relatively, I mean, well-intentioned, but probably slightly ill-thought-out work in this area because actually there is some great stuff happening, but we really need to be very careful about more reputable and more trusted and more, more useful assessment. But it's great to hear that there's good things happening. And I— in the report, the Future Proofing report, you've got lots of examples of people around the world actually doing some interesting things, like Big Picture Learning in Australia.

Yes, Big Picture Learning. Tim, I'll send you another couple of reports that we've done. One of them is a series of 6 case studies from around the world, which includes Big Picture, but includes another 5 organisations that have each chosen their own path. So I'll send that to you.

I'd love that. Brilliant, thank you. I'll share it with the show notes as well.

Yeah. Okay. And I've really enjoyed speaking with you. I didn't know you before this. It sounds like you've got a terrific thing going.

Oh, I know. I mean, thank you. I just so appreciate people giving their time and I feel part of my job is bringing people together and enabling people to hear that there's some really amazing thinking going on all around the world with these things, but sometimes it can get lost in all the busyness of classrooms and schools. Every day. So no, I really appreciate your time. Thank you, Sandra. It's amazing and a fascinating conversation for me as well.

Thank you.

Thank you so much. Thanks, Sandra. We hope you've enjoyed this episode. Please feel free to continue the dialogues with us on social media with the hashtag #futurelearningdesign and on the Intrepid Ed News website, intrepidednews.com.