Mindframes and Impact - A Conversation with John Hattie

Welcome to the Future Learning Design Podcast.

A lot of the work I'm doing at the moment in schools is getting them to ask the kids what it means to be a learner in the school, asking the teachers what it means to be a learner. Our teachers are usually quite rich, kids are quite barren. For most kids, learning is coming to school, sitting up straight, paying attention, getting the work in on time, which has got nothing to do with learning.

Hi everybody. Thanks for joining me for another episode of the Future Learning Design Podcast. It's not every day that you get to speak to one of your personal heroes, so it's a huge honor to bring you this conversation I had with Professor John Hattie. John Hattie is an award-winning education researcher and bestselling author with nearly 30 years of experience examining what works best in student learning and achievement. His influential 2008 book, Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement, is believed to be the world's largest evidence-based study into the factors that improve student learning. Since 2011, John has been the Director of the Melbourne Education Research Institute at the University of Melbourne. He's also the Chair of the Australian Institute for Teaching and School Leadership, as well as past President of the International Test Commission and Associate Editor of the British Journal of Educational Psychology and American Educational Research Journal. John was awarded the New Zealand Order of Merit in the 2011 Queen's Birthday Honours and is a Fellow of the Australian Council for Educational Leaders and American Psychological Association and has published and presented over 500 papers and supervised over 200 thesis students. John's notable publications include Visible Learning, Visible Learning for Teachers, 10 Mindframes for Visible Learning, and most recently, 10 Mindframes for Leaders, published last year by Corwin Press. You can follow John on Twitter @john_hattie.

Hey, John, how are you doing?

All right, yourself?

Yeah, I'm very well, thank you. Good. Well, let's dive in and then, you know, make the most of it while we can. But thank you, I really appreciate you taking the time. I'm sure you're a very wanted and busy man.

I always like to talk about these things.

Yeah, well, me too. So, I mean, obviously you've achieved rather a cult celebrity status over the last kind of 15, 20 years, and some phenomenal contributions that you've made, clearly. For me, I can kind of see a bit of an arc of the way that things might have gone, because it clearly was received in different ways and then adored and quotes like, "Research reveals teaching's holy grail," you know, pretty big statements about the contribution that your work was making. And then of course there was criticism about meta-analyses and all these kinds of things. So I just wanted to invite you with a big opener, just if you had some reflections about the way that arc had gone over the last couple of decades.

Well, the irony, Tim, is that the visible learning work has not been my career. I was trained and did all my teaching as a methodologist, a psychometrician. I taught those research design courses and statistics courses that I'm sure you love to do too. And it was partly because the fact that I wasn't in the mainstream of teacher education or those kinds of things that I was able to look in. And it beguiled me right from the very, very start when I was welcomed into my first university job. And it was like, we like you here because you can do numbers. But, and the big but was everybody told me that if I was really going to make a difference, I had to study what they were doing. And there was that notion of, you know, how come every one of them, they all have evidence. They all have evidence that what they do makes a difference and they all compete. And then I went into schools and it's the same thing. Every teacher wants you to go and watch them teach. And they do have evidence. They can produce kids and stories and whatever. And it didn't make sense to me that everything worked.. And so at the same time I started my career in 1976, Gene Glass invented meta-analysis. And as a methodologist, it was intriguing to me what this new method was, and obviously the best way to find out was to do one. So we actually ended up doing some meta-analyses. And then the pivotal moment was, what if I did a meta-analysis of meta-analyses? Now, it took me about 20 years not to collect the data— that was easy— but to make sense of it to look at all the moderators, all the things that made the difference to try and build a story. And then in 2009, I published that book and it was my 10th book, Tim. What happened to the other 9?

No one had heard of you before that. Yeah.

It was like, and part of it is when I finished doing it, I asked my wife to read it and she read it and said, which 2 people in the world are you writing this for? And it was trenchant full of numbers. And so I ditched that version completely and utterly expunged it and wrote the version that was produced, which wasn't as detailed. So I tried to say at the beginning, yeah, Meta-analysis is but one methodology. It's got positives and cons, et cetera. And the big thing here is about the overlap. And she read it again and said, yeah, you got it up to about 10 people. So she invented the little dial that was in the book that tried to give it a sense of purpose and unity. And I think that helped quite dramatically.

Yeah.

But at the very last minute, I added all the 130, I think it was at that time, 150 effect sizes in order in the appendix. And that got a lot of attention. And in one sense I don't regret it because it got a lot of attention. On the other hand, it led to probably the biggest misinterpretation. About 2 years ago, a colleague and I tried to locate every criticism that we could find of Visible Learning, and we came up with about 100 of them, and 70+ percent of them were about that league table. People saying, well, you can't treat them separately. Well, the whole book is about the overlap. And then you've got to look at the moderators. Well, every set It's about the moderators. So most of the critics have never opened the book, and that's the way it goes. But the thing that stuns me is, in the 10 years since the book was produced, no one has come up with an alternative theory. And last year, I actually put all the data online for free. Anyone wants it, it's up there free in MetaX. And the challenge is for someone to come up with a better explanation, and I'll be the first to say, well done. And so most of the criticism is snipes on the sides, things about meta-analysis. They've not looked at my own writings about this. Now I'm not saying I've got everything perfect, that's not what I'm saying. What I'm saying is that the nature of the criticism hasn't been that trenchant. And the other thing that happened is, as a measurement person, my team at the time, we were working on developing the assessment scheme for New Zealand schools, and it was coming to an end, and of course they're worried about their futures. And so they said, could they have a go at implementing Visible Learning in schools? And I said, yes, provided I'm not involved, because I'm not a bad talker, I can tell a good story, But I also know that when I go into schools, within half an hour of going, it's gone. Nothing happens. Nothing changes. As I said, I don't want it to be about me. I don't want it to be tips and tricks. And it's grown like topsy. We now have about 200 or 300 people in the world, 10,000 schools. And so I've learned a lot about implementation and that's affected how I've written since. Yeah. So that's the story about where it came from and where it's at.

Interesting. And why do you feel that it has gained that level of traction in schools? Because it, I mean, yes, it has the danger of becoming a box of tips and tricks, of course, but clearly people are crying out for something like that in schools. So what's your reflection on that, why it did gain such traction?

Well, I think you're right. The early days where they took the top 10 influences and said, tick, tick, I'm doing that, I'm not doing the bottom, therefore missed the point completely, which I take some responsibility for. I think one of the big things is I never ever say this is how you should teach. And if you look at most professional learning that you go to, someone wants to tell you how to teach like they do. How to use the resource like they use it, how to use the latest whatever app or whatever like they do. And like, I take a very strong line that I don't care less how anyone teaches. I care about the impact of their teaching. And that switch, I think, has been one of the grabs. But hey, here's someone who wants to come along and tell me how to maximize my impact. He's not going to come in and say that your teaching's wrong and you have to do it this way. In fact, there is no one correct way of teaching, and I learned that working for the National Board in the States when I was living in North Carolina. And I had to review the videos of the top 1,000 teachers, English language language arts, as it turned out, and there was no commonality.

Yeah.

Interesting. Yeah. And so not telling someone how to teach, but also that switch from what works to what works best, because virtually everything works, and that switch from let's talk about our impact. That I think has been a grab. I don't know a single teacher who entered the profession for other than one reason, to have an impact on kids.

Yeah, absolutely.

Going back to the core being of all teachers.

Yeah.

But the other big thing I realized as I heard people talking about my work, using my work, and they weren't getting it right, which means I wasn't getting it right. And so that's why I've spent a lot of time in the last 5 years talking about mindsets, how you think.

Absolutely. Yeah. And I mean, I'd love to come onto that, but I just, just one thing I was thinking in relation to, I mean, you mentioned that what works and then what works best, but then also I think I've been having this conversation quite a bit about the difference between complicated problems and complex problems and teaching. If you just look at that page and all the lists, clearly teaching is the most complex endeavor, right? And absolutely we're united broadly by the impact we want to have on young people. But actually, if you start trying to break it down into this complicated problem where you can have best practices, you know, try this, do this, do that, it takes away from that idea that actually it is a complex adaptive problem and we have to be able to grab and reach for things when we need them.

Well, that's why my latest work is about the notion of evaluative thinking, how you make that decision to use this method and that method with this kid on that topic for this influence. That is, as you say, quite complex. And COVID was another good example where I think parents learnt in their face that teaching was quite an intellectual activity that required these complex decisions. You know, a lot of parents had troubles with their one or two kids, and if they paused and realised that a high school teacher has 200 a day, 200 days a year. But my frustration is that teachers are unbelievably good at denying that expertise. I said, that's how I teach, it's the kids, and it's killing us as a profession. So I'm really quite strong on looking at mindframes, looking at evaluative thinking, and just acknowledging that incredible expertise to do exactly what you said, take that complexity and make it work.

Yeah, brilliant. That's so interesting. I agree, we're quite self-deprecating as a profession. But then just, I'd love to talk about the mindframes a bit more, but just quickly, because you revisit visited the meta-analysis. So what was it that made you actually go back to the original studies?

Well, that was on feedback. Like, in the very early days, it became clear that feedback was among the more common denominators of the things that had high influence. And I've spent a lot of time over the years looking at this thing called feedback, and one of the things you learn very quickly is that it's incredibly variable. The same feedback I give you today, Tim Logan, works. I give it to you tomorrow and it doesn't. That variability is key. Yeah, but then what I noticed, and I remember going to England once and I met the Minister of Education, and he said, oh, you'll be so pleased, John, that next year I'm declaring that England's going to be the year of feedback. And I lost a lot of brownie points with him. In fact, the meeting was almost over after I made the next comment. I said, Minister, what do you do the year after? No feedback? Brilliant. And that sense of they were using it, and because we're going to increase the amount of feedback teachers give and the power of it, things were going to change. Well, it doesn't look work that way. The logic we use, we call it the Kenny Rogers model. You've got to know when to hold 'em, you've got to know when to play 'em. There's a right time for feedback, there's a wrong time for feedback. There's different kinds of feedback in terms of feedback about where you're going, how you're going, where to next. But the thing that many, many forgot, and again, and I was guilty of this in the early days, is we look at it through the wrong lens. We look at it in terms of how teachers can increase feedback as opposed to how students receive it. And so our mantra now is that, was the feedback heard, understood, and actioned? Actually, you guys give a lot of feedback, but not much of it's received. Kids learn very quickly that feedback has a cost. You might have to do it again. And I'm brilliant at this. I'm the world's best selective listener, and so are kids. And so one of the tasks we give people to do in our workshops is we say, spend Sunday, write a page of comments, give it back to Tim Logan, wait a day so it's not short-term memory, Then asked him to make some notes, bullet points about what are the feedback I gave you yesterday. It's so sobering that most kids have a blank sheet of paper. If there's nowhere to next feedback, they will stand with that page in front of you and say, I got no feedback. So what happened is that Benedikt, who was Claus's PhD student, said, we started talking about this feedback notion. I said, look, I think that if you look at the more recent meta-analysis on feedback, the effect size is going down. Because people are misinterpreting it. Interesting. And so, and then the other question, a more methodological question, let's say you have, as there was, about 25 meta-analyses on feedback. He said, well, what if they're all based on the same articles? Doesn't that mean that you actually haven't got 25? And I said, well, no, that's not how it works. In most areas, 30% overlaps about the norm, but it varies. And so he said, well, I'm going to go back and get every meta-analysis, find all the original articles. Can you believe this guy? I just thought this is wonderful. So he did all the hard work, and as you saw, the effect size dropped considerably, not because feedback's not important, but we misinterpreted it so badly. And I think this is something we need to be— but this is the beauty of new knowledge. We add new things to the world and we have to go back and reinterpret.

No, it's good. I mean, it's a good modeling on one level, but it's also sobering in the fact that, yeah, again, we just have to be more nuanced, right? We are always looking for these, these black and white, these quick fixes and easy answers, and actually it's just You know, the world's more nuanced than that, right? So then I'd love, let's get into the mindframe stuff, cuz that's clearly, I completely agree, it's such an important area. One of the phrases in the book was practices trump labels and beliefs trump practices. And so come back to this idea that beliefs, the underlying belief about why you teach and the impact you're trying to have is more important than how. And as you already quoted yourself, I couldn't care less how you teach. I think for a lot of people in 2017 was a bit of a a wake-up because some people, as we've said, were using Visible Learning as a recipe book for how to teach. So yeah, perhaps if we could focus on the teaching side first, and then your latest book is Mindframes for Leaders.

Look, if you look at the various teaching methods and particularly all the inquiry ones, the problem-based learning, discovery, individualization, personalization, the effects are incredibly low. And believe it or not, people get very angry when they hear that. I get some incredible emails sometimes and Most of the hate mail is actually ironically written on letters and sent to me in the mail, which is quite weird. But anyway, the evidence is that they're low. But the interesting question is, why are they so low? And it turns out the answer is because they're introduced too early. Kids don't have the knowledge to make relationships or do problem-solving about it, problem-based learning. And so what I'm doing at the moment is I'm looking at a more of a sequential model of learning. And when you look at the model of learning, that sort of goes from first exposing you to something, then consolidating it, then going deeper in relating it and transferring it, and let me be clear, learning doesn't always follow that linear model, but it's not a bad model. Then it turns out if you want to relate different teaching methods to the various phases of learning, of the 400 teaching methods I can discover, there's only about 15 that actually do that. Most teaching methods are all over the place. So what chance has the kid got of processing information at the level that you're actually teaching? So I think we need to be smarter at differentiating teaching, and then it's going to what you said then is The intellectual skill to make those moment-by-moment decisions about particular teaching methods, about how you give feedback, about how you prompt kids, where you move them to next— how do you capture that? And that's where we went to this notion of mind frames. And as you know, we decided 10 was enough, and so we picked up what we thought were the, the top 10 for teachers. We've now done it for leaders. My son and I have just finished the book on Visible Learning for Parents and the 10 mind frames for parents.

Brilliant.

Yeah, we're working on the student one as well, but it was trying to get at that thinking that made the difference. Like, the most important one, in fact the one that's repeated 10 times if I'm being honest, is that when you walk into a class, Tim, I want you to say, my job here today is to evaluate my impact. Now that begs the question about what do you mean by impact? About what? Yeah, how many kids are going to benefit that and how big is the magnitude? Then I want you to be, as great teachers are, there are many, many of them out there, I want you to be incredibly sensitive to whether the student is actually learning or not, and if not, you are the one that takes responsibility for alternative ways, clearly not repeating the same method that you did the first time when they didn't get it. And so that notion of that, if you think that my job is to evaluate your impact, the kids are unbelievably the beneficiaries. And so each of the mindframes are trying to get at a different part of how we think and process and make decisions in the classroom, and also in terms of the leadership. Yeah, so for a leader, it's the same notion when they walk into a staff room. I want them to say, my job today is to evaluate my impact. If they— and I'm probably the worst culprit in the world here— if I walk into my staff room and I speak 100% of the time, there's no way I'm going to hear my impact. And some of us don't like to hear our impact, so we make it such that it never happens. And that is a travesty.

Yeah, there's a safety in keeping talking because then you just imagine you're having an impact and you walk out of the classroom and think, ah, that was a great lesson.

Or even better, and I'm good at this, you ask a teacher for a particular reply knowing you're going to get the right answer. So you assume everybody got that answer.

Absolutely.

Yeah.

I don't know. And that's Dylan Williams stuff, you know, that, that's critical, isn't it? Definitely. But one, just one small point. I love the idea that you've called it mind frames, cuz I think language is quite important and obviously everyone's talking about mindsets and I just think there's an irony in talking about setting something because it clearly needs to be adaptive and flexible. Right. So the frame that we look through at what we're doing is so interesting and, and important, but also The impact, I still think there's so much disagreement in our profession about the type of impacts we're having. And that, that's actually really a core part of where I think some of the internal kind of discussions come from. Actually, what are we preparing these young people for? What impacts are we trying to have? Or are we just at the level of the cognitive, or are we, are we at a level of something deeper and more to do with thriving and flourishing?

I do this quite often when I work with schools. I'm just going to shadow a kid for a day as they go to each high school class. And you go into the first class and the teacher talks about the impact statement, here today we're going to get you through the assessments, or we're going to cover the curriculum, or we're going to turn you on to the power of learning. Those kids hear so incredible multiple interpretations during one day. One of their skills is ignoring the lot. One of the major mind frames of a principal, of a school leader, is to create some common understanding and some common language. Again, like a A lot of the work I'm doing at the moment in schools, not very far from where I'm sitting right now, is getting them to ask the kids what it means to be a learner in the school, asking the teachers what it means to be a learner. Now teachers are usually quite rich, kids are quite barren. For most kids, learning is coming to school, sitting up straight, paying attention, getting the work in on time, which has got nothing to do with learning.

Yeah.

And so that's a really sobering way of showing the teachers what their impact is, as opposed to them sitting down and saying, this is my impact. It's more than a one-way street. And so I think it's very important for principals to have a sense of what it is, and I think COVID amplified the issues of impact, it amplified the issues of equity in our schools, and I think we've got a really good opportunity right now to say, well, what is it we value in the school? Now, my argument is it's a basket of goods. Your country has been obsessed with just achievement in that basket of goods. If we don't know from COVID we certainly know that developing students' strategies of learning, developing classes and inviting places for kids to come to, developing their social and emotional attributes about learning is all part of our outcomes. In my jargon, we talk about the skill, the will, and the thrill. There are skills. I make no bones about looking at assessments, maths and English and whatever. Then there's the will, those dispositions that often called the 21st century skills or whatever they are, where we want that resilience about learning. We want them to take on new challenges, and then the great thrill is every one of us want our kids to be masters of some areas and to go down deep. And I see those as all part of the 3 outcomes and inputs, but the 3 outcomes in the basket of goods. I don't think you need more than 7 or 8 in the basket of goods, but a school should have a common understanding of what they are. It's beyond a vision statement. It's looking at this as what we value. The one that I'm probably most obsessed about is the sense of progress. Like, at the moment With test scores being so dominant, people wrongly think— and people being kids, teachers, and parents— wrongly think that high-achieving schools are good schools. Well, if you start above average and you go to a school where you just cruise, that's not a good school. And so switch the discussion and say every kid deserves— every kid deserves at least a year's growth for a year's input. But it's not either/or, it's not progress or achievement, it's through progress to achievement. And I think there are many kids in our schools, there are many schools with many teachers where they are making incredible progress, but they may not be high-achieving schools, and we need to esteem those.

Yeah, no, wow, absolutely. I mean, one of the things there is that I think that's one of the reasons, coming back to where we started, that people liked the dial that your wife cleverly came up with, because it was that idea of 0.4 being the average, right? So anything above anything above that was more than a year's progress for a year's input. And so we want some kind of— I completely agree with it. You know, we've got to be focusing on growth rather than achievement, but trying to give that some kind of metric or measurement is also a hugely challenging thing in our world. And we like to try and put numbers on things because again, it gives us some security blanket to know that we're doing the right thing.

Yeah. But my frustration is that in most parts of your country and mine, you've got access to tens if not thousands or tens of thousands of measures of achievement. Like, I've never met a teacher who couldn't make an achievement assessment within 5 minutes. But how many measures have you got of progress?

That's true.

And I think that's— like, we developed the New Zealand system. They have between 200 to 300 million assessments at their fingertips related to their curriculum that measures progress. Hardly any other country— there's some in the US, the Northwest Labs people have got good measures of progress. I know they tried to do it in some of the other systems. But it's, I hardly ever see it and it's probably one of our most critical things.

Yeah. It's so interesting, isn't it? Why not? Yeah. And then just one last question related to, I've also recently also heard you talking about the moment that we are in, or we, we still are in unfortunately with the pandemic and you've taken the golden ticket, bit of a Charlie and the Chocolate Factory reference, but, but that this is a once, once in a generation possibility to build back better. And there's a lot of people. Using that kind of language and rhetoric often about it. But clearly, you know, if I have the opportunity to ask anyone, it would be you to think about what would be your recommendations there. Not again, not for how to teach, but what would be your approaches or the processes or the systems you'd put in place to rebuild back better, or maybe unbuild some things that we'd already built.

Tim, I got into trouble for that comment because people assumed that I was not acknowledging the sadness, the sickness, the death. Yeah. But I want to make 3 points. Firstly is I cannot find in any country in the world during COVID where a government department, a district, came up with policies that help you teachers teach during COVID times other than whether a school should open or closed. Yeah, I think the first thing is what happened in COVID was truly an educator-led revolution that I've never seen in my life. It came from the bottom. Then I look at the research coming out, like here in Victoria where I live, COVID happened at the start of our school year and we had 4 months out and so we've built up quite a reasonable amount of research evidence about what's happened, and the thing is, if you compare the results of last year with the previous 10 years, if anything, last year was better. Certainly in the— where there was high-stakes exams, there was no question it was better, and that's been replicated in many other disasters, the Christchurch earthquake and other things, and if anybody uses the phrase that last year was unprecedented, they have no knowledge of history. Because this happened in 1953 in your country where there were many more deaths than there were in COVID. Obviously the Spanish flu, which did in that case did start in your country, wide around the world. This was truly an educator-led revolution. The second thing is when you look at the teachers who survived the most in COVID is those that already knew how to have that gradual release of responsibility. They weren't dictators, they didn't talk 100% of the time because you can't do that in Zoom land. So that's the second one, the release of responsibility. Related to that, the third one is students who had the skills to self-regulate, to know how to work in groups, to know how they're going, how to get help. They were the beneficiaries. And so when you look at those things, and I could go on to many more, you've got to ask, why is it we don't see that in the regular classroom? Yeah, I heard last year for the first time, besides the words of how effective was I as a learner, 'How efficient was I?' And some students were saying things like, 'Oh my gosh, I can do this work at home in 2 hours. What do I spend 5 hours in the classroom doing?' We don't look at that efficiency because of our structures. Now, I'm not advocating that we should go to homeschooling. What I am advocating is that we can replicate some of those homeschooling conditions in the regular classroom. We can look and say we don't have to control all our learning. Our job is to get kids going down the right highway, making sure they don't go off into cul-de-sacs, but letting them drive the car occasionally so they learn how and sometimes we need to pull them back, and that's that mindframe skill. So I think that at least those things we learned from COVID But one of the things I also know from the many extreme crises that we've been through in the world over the last 100 years since regular schooling happened is we rush back to the old normal so fast and we lose the opportunity. So I'm begging every school leader out there to spend some time with their staff saying, what really did go well? During our COVID teaching, how can we introduce it, and then the thing you raised, and what do we stop? What do we stop? What do we de-implement? Maybe we should do some walkthroughs to look at how many questions do kids ask, just simple as that. Normally in the usual day, kids will ask 1, maybe 2 if that, questions about their work they don't know the answer to. You know, they'll ask about process, what page am I on, can I go to the toilet, but let's just start there. So I think this is an incredible opportunity. Now, on the other hand, my visible learning work is based on the regular old grammar of schooling, and it works for about 80% of the kids, about 40% of the kids really well, another 40% get sent, they just give up and say, well, I'm going to go along with this guy. That's not good enough. And if anything, COVID amplified the equity issues in our school, and too many kids are being left behind. So what did we learn in COVID? Even with some of those kids that normally were left behind, like some of the naughtiest kids in your classroom did quite well in COVID. Because they didn't have the usual distractions. So you could hear I could get quite obsessed about this. The biggest travesty of COVID is that we learned nothing.

Yeah, I completely agree. But it is, there is clear, I mean, you can already feel it, right? A tendency to just pull back because again, we're safe in the old way.

But as you said, the usual winners won and the usual losers, well, we know how to handle with them. And that's just horrible.

Yeah. Well, thank you. I really appreciate it. We managed to make it before your granddaughter went to sleep.

I've got to go and wake her up now.

Thanks a lot, Tim Logan. Thank you, Jonathan. Love to talk about this stuff.

Brilliant. All the best.

We hope you've enjoyed this episode. Please feel free to continue the dialogues with our guests, with us on our blog or on social media, or within your own networks.