Beyond Competency: Colin Chapman on Deeper Learning with Technology

Hello, everyone. Welcome back once again. I'm Noah Shartov, this is John McNally. I said, gesturing, possibly not to wear the wiggler. We're both here from Wolfram Research and today we're talking to Colin Chapman, who is sufficiently far from enough away from us that you are literally talking to us. From tomorrow, I believe.

Indeed.

What better place to get a perspective on the educational ticks of tomorrow than tomorrow.

That's right. Exactly right. It's 8 o' clock in the morning here. That's still a civilized hour.

Colin is master teacher at the Victorian Academy of Teaching and Leadership, and he works on building educational programs that go deeper than just recall or recognition, but into more essential learning goals. And today we're going to be talking about that, about the role, teacher training, about educational technology, lots of topics on which he is an expert. So, Colin, thank you very much for being with us here today.

I'm always up for a bit of a chat and I enjoyed our time together when we were in the States, when I was in the States recently at the Wolfram Technology Conference. I'm keen to continue the conversation.

As did we.

Yeah. It was a pleasure meeting you there, Colin. So thanks again for joining us.

No worries.

Well, since we're going to discuss education, let's begin with your own. You have degrees in education and cosmology and astrophysics, I understand.

That's right. So my postgraduate is in astrophysics, cosmology. Nothing too exciting. But what's interesting about studying physics, of course, is you end up being a bit of a Swiss army knife with regard to problem solving. But what's interesting with the work that we do at the Academy here is that we work across the disciplines and can sometimes get a bit trapped into thinking that physics is the Swiss army knife. And we've got this wonderful way of thinking about things and conceptualizing problems, modeling ideas, moving those models from idealized models towards more realistic models. And we do that, of course, by making the models less perfect as we go along. But English does the same thing through their process of hermeneutics. And mathematics, of course, does similar processes through heuristics. And all the fields engage in this sort of activity. So it's really refreshing to be at the academy and to broaden an understanding of all the idioms that inform so many different disciplines and the connections across them. And what's interesting is that the connections tend to be this modeling type sensibility that is our best guess that something is probably approximately correct and iterating and iterating until we have something that's interesting and compelling for us to be able to interrogate.

Now, are you talking about physics from an educational perspective or is that more from a research perspective?

We'll say that's interesting because one of the sensibilities we have here at the academy is that we're very keen for teachers to work in the discipline rather than being describers of a discipline. So moving away from a textbook, they have their uses as being a map, but we really want teachers to bring in problems that are, I won't say authentic because they can sometimes be fake. Right. But interesting questions, the sorts of questions that Conrad Wolfram talks about in the maths fix. Questions that are compelling, that use the tools that are available for people to actually be practitioners of the field as they're learning the field. And this is a very important topic. Some people are saying right now, I would agree with Conrad Wolfram that it was an important topic and Stephen Wolfram 20 plus years ago in that if we have the tools for UW practitioner in the field, you ought to use the tools. And the moral panic that people are experiencing now about Chat gtp. So GPT is really something that in mathematics and science education we dealt with over a decade ago and the answer was ask better questions, develop better stimulus, use the tools, be a practitioner of the discipline, not somebody who simply describes it. And that goes back to the comment you made off air before, which is this idea that recall questions aren't where we're at. The machines can do that. Yeah, very well. Although it's interesting to consider and contrast. We had a discussion yesterday about Chat GPT and it's interesting that the algorithm has got a mixture of common sense understandings of ideas and trying to mesh it together with what it sees in the text that it's examining in order to come up with some interesting piece. And it's a great exercise for learners in schools to actually go and interrogate that text for those common sense understandings and so on and compare them with what an academic approach might be or a scientific approach. I sort of transgressed a bit there, haven't I?

Well, actually, so we were actually going to ask you exactly about ChatGPT. So shall we go down this path right now or would you like to return to the slightly more general background of what it is that you do and what it is that you teach teachers how to do and then return to ChatGPT? I leave it to you because both of these paths are quite interesting.

Well, look, education's a maze and mazes for being lost In. So let's go for the getting lost in.

Right, so let's just jump down to the ChatGPT.

Let me, before we go to the specifics of ChatGPT, I did want to ask, you mentioned 10 years ago, these questions of how do we deal with a tool that's redefining these things came up. What was the impetus for that conversation 10 years ago?

Well, I was a little frustrated with robotics programs that were just simply implementing scripts. Really just, you know, go and pick up this block, put it somewhere else, nothing really compelling. And then we move towards more of a story based approach through say, Lego Robotics, their competitions and so on. But I believe that students had the tools with robotics to develop interesting responses to questions and instrumentation. So an example of that was when I was teaching in Switzerland. The students were interested in forest because in Switzerland we have little pockets of forest, not like in Australia where it's for acres and acres and acres and they often community owned. And one of the issues over there is that roads are being cut through forests and it fragments it. And one of the students wanted to ask the question because they were concerned about this fragmentation. Where is the forest? And that was the question. And we developed instrumentation to be able to measure certain proxies. Right. And so one student developed a system for just measuring and there are two approaches for this one, light penetration. And so they, one of them did a voltage divider network with an LDR measuring light levels. And then they calibrated the system. Another one just had a solar panel collecting and then, you know, calibrating that. Another student was interested in soil ph and how that changed as they went through the forest. Another one temperature and another one soil temperature. So they had all these different measures. So they, they were actually able to implement that measure without me giving a script because they had to work all out themselves. And what was really curious was a wonderful tool out there called Google Maps. And the students overlaid their data without me saying anything, they just did it. They overlaid their data on the local map for the area and developed these contour maps. And they didn't expect that to happen, but contour maps happen. They did transects and they got contour maps of light level, they got contour maps of soil ph and temperature and all the rest. And they discovered that there was a plateau and they defined that plateau as being where the forest is. And it was interesting because every time a new road was made into the forest, they could work out that the damage was not just the road itself, it was to the Integrity of the forest. And the more paths you have, of course, those little islands become smaller and smaller and smaller.

Amazing.

So, yeah, but this is the sort of thing you can do and should be doing because it's a real project and it came from them, not from me. Yeah, sorry.

Oh, no, I also had a follow up question about that. So these students were in one class collaborating.

That one was a pair of classes.

Okay.

And collaborating.

So I want to dig down on that pair of classes then, because if one thing that I think is really innovative about some of the techniques that you advocate is the way that students can collaborate outside of what we consider like a class. Right. Because you could imagine doing some version of that technique in, you know, all the students are in one class with one instructor. But that's not the most interesting case. You really want to push things further than that, like what's done in science. So could you elaborate on the logistics, a little bit of how can an educator facilitate that sort of emergent behavior among students?

Well, why shouldn't a student work across the globe? Why shouldn't a student work on a problem and come to an understanding that. Well, it's interesting in a classroom. So in one classroom there are different perspectives as to what an effective measure of where the forest is. Right. Or where the forest might be. And a bit of risk taking in there. And we'll talk about risk later because I know that's something we were interested in when we were in the Wolfram Technology Conference, but. So what's interesting is a student will listen to another student and think, oh, that's curious, that's interesting. Or your measure is a critique of my measure. What you've done to implement your measure is a critique of mine and mine is a critique of yours in different ways. Well, if a student is working beyond their classroom with a student in Germany or a student in Indonesia or whatever it might be, they're getting other perspectives. And let's say the student is working with somebody from Indonesia. Well, instrumentation in Indonesia for a local school may not be as accessible as for a student in, say, Australia. So these sorts of questions about equity of access to technology become important as well. And it's a part of the science project, really, in the end, that's important. One thing that I'm interested in challenging is, you know, the deadly battle for the acronym. So we've got stem, we've got steam, we've got STEAM C, STEAM mc, all these acronyms throughout the place. To me it's systems thinking. And what's really interesting about systems thinking is that the system is beyond the experiment because the system is placed in a societal context, an economic context, an environmental context, and so on. And these contexts become. Well, they become provoked when students are working with other people from different contexts and they start to realize the system is a little bit bigger than I thought it might be. And then they have to make decisions. Should I constrain the system so I can work more effectively on the problem that I need and then perhaps later on expand it to incorporate these other concerns and ideas, which is something that we do in the systems engineering study design here in Australia? Or do I go the whole hog the gestalt and then hope to get some sort of insight? Overall, the interactions between students that you're.

Describing are what I think designers of basically every large online course the world over design discovered they wanted to create. Many of. Many of the designers of these courses already knew they wanted to create that and have failed in various interesting ways. So how would you say, what would you say is the secret ingredient to having students genuinely interact in the way that you described, rather than sort of, you know, requiring a certain number of forum posts that nowadays can be written by ChatGPT? So how do you actually get the students to do that in an effective way?

Well, compelling problems, not just scripts that you follow. Genuine responses to stimulus. And I suppose that's something that's interesting because we talk about this problem solving paradigm and to me it's more a response to a stimulus. And when we look at a response, then those responses can be quite diverse. And not that this, they're solving the problem without solving the problem. They're solving aspects or they're solving perceptions of the stimulus and so on. You know, an example of that might be to. And this is with a teacher group that I had last year where the challenge was to prevent falls in the elderly. So falls in the elderly are one of the larger causes of fatality in Australia. Break neck of femur, we have hemorrhage and so on. And we, we took an interesting approach to that, which was to develop the most ridiculous solution to the problem. And so we have these large, I like to get scrolls of, we call it butcher's paper over here, but, you know, wide scrolls, probably about a meter across. And we unwind a part on the table and each group will come up with ridiculous solutions. And some of the solutions are quite unreasonable. You know, lock people in their houses, for example, and so on. But then some are more creatively unreasonable and we ring a bell and then each group has to go and switch tables and go around in a circle and the next thing they're challenged to do is to whatever's in front of them make that solution work. And so we have these really interesting dynamics and we keep ringing that bell and getting people to circulate and they keep unrolling these papers. And what we find is interesting is that there's so much excitement about making something that was a crazy, deliberately crazy solution to a problem, something that is well doable and you get these really emergent ideas. So one key idea we had that just came out of the group was trust, right? And how do you build trust into a technological response to a problem such as falls in the elderly? And it is a big issue. There have been interventions in the past where people have been asked where large foam rollers around themselves so if they fall, you know, gets absorbed. And then there are people who have got a distrust of technology. It's got hard edges and all these sorts of things. And so from a technology response point of view we really wanted to understand, well, how do you design trust in a system and then how do you deliberately design a system that is trustworthy but also a little bit jarring when it has to be to warn you about dangers and so on. But you know, this is part of the way to get people involved in, actively involved without saying you must do so many hundred words here and do this post here and so on. It's going from an absurd process, absurd solutions to or responses to a stimulus towards something that's interesting. And the reason that I thought that might be an interesting way to go forward is there was a cartoonist from Mad magazine back in the day and he had, I think he had stupid solutions to everyday problems or I can't remember what, what. But he developed these car parking solutions for the United States which in New York which had a Ferris wheel, one of them was the cars would go in there and that's now something that's happened, right? It was an absurdity, but it was actually operationalized. And so I find that a really interesting way to engage people of all levels in a problem without being the, without compelling them to produce, well, empty words. I think especially in the learning phase and with respect to encouraging risk taking behaviours we can document later, which is part of the agile response to design. So this idea of approximately probably correct and then keep iterating.

Let me ask you about this deliberately absurd beginning. Is the goal that you will discover things like this Ferris wheel which initially seem absurd but are actually good ideas or Is this more about developing skill to make other ideas that might be more attainable actually turn into realities?

Well, let's think about risk taking because we have a, we have a rhetoric of risk taking in education, but, and we've had this discussion before, but we don't reward it. It's, it's really just rhetoric. Take risks. Take risks on the part of the.

Educators or the students. Are you talking?

That's interesting. Yeah, because we all, we often say, no, that's your job, student, you the risks. I'm going to sit here in my deck chair and watch. Right. So for here at the academy, because we're keen on people working in the discipline, we know that people in the discipline make multiple errors on the way towards a response which is perhaps more satisfactory. That's how it works as a practitioner. So a teacher, as a practitioner is doing that, taking those risks. And what's interesting, and I call them train wrecks, a teacher getting themselves out of a train wreck is teaching some really important dispositions. And also, you know, what does one do in this real circumstance to work themselves out of the train wreck and retrieve something interesting from it. It's a curious thing to me that we have this language also of modeling in education. And look, there are some things you model, behaviours and so on because you want students to learn social, more traits and so on. But when it comes to being a practitioner discipline, being a mathematician, being a physicist, being a systems engineer, you don't model, you do. And in the process of doing, you do all those things that one has to do when things don't go right. It's countless. The number of times when I've thought on the way riding my bicycle into, into work and I thought of a problem and I stopped short of a fully worked response, just enough that I can start and then I do it, implement it in front of teachers, students, whoever might be there, and it becomes a train wreck. And then we all get involved. And when you talk to the students afterwards, the thing that you get from them is I noticed you did this when this happened. Right? And they even pick up those moments of panic because sometimes it happens that you get a bit of a panic because it's a bit frustrating. And then they say, but I noticed that you did this, you had this, this process that you go through and they can give you also relations to things they do in other areas of their life where they go through similar processes. And again, this is part of the system, part of a systems thinking approach to working in a discipline is not Just can I do this quadratic equation and reproduce it, which I'm never going to do in real life because I'm always going to have the machine do it for me because I've got to do the thinking part. It's really those. The system includes the dispositions of somebody who works in the discipline. And even in the discipline there are multiple dispositions, which is another reason why we need to have multiple people involved in these experiences so that people can see engineer A, B and C, same problem, exhibit at different dispositions, perhaps different outcomes.

Two questions about risk actually, before really jumping into systems thinking and modeling systems. So in terms of making it, making students and instructors feel that it's safe to take risks for them, do you think that the initial absurdity of some of the problems you're describing in that exercise and sort of framing the initial attempt as explicitly absurd, do you think that sort of helps psychologically with students, you know, not worry that I'm going to be penalized for trying to work on an absurd example because I know it's absurd from the beginning? Do you think that's part of it?

Oh, yeah. And that's, that was my train of thought before I went in another direction for a moment there. But that's that absurdity is it gives such a safety because in the end, if it doesn't work, it probably wasn't going to work. But, oh, I've made it sort of work in some interesting ways. And now I can take that aspect. No, well, more than that, there's not just a physical response that they can take and say, well, now I'm going to put it into this other thing. But these dispositions with regard to being a system engineer or a physicist or a writer in English or whatever it might be, I can take those dispositions to just have a area. And one of the interesting things I have in, in school land is having students encouraging them to, well, identifying thinking. Because often we just say, okay, the correct response to that question, tick, tick, tick, tick. I like to have small groups working. Look, and I'll disclose as well, years and years ago I started off in primary education. I'm not sure what that translates to in the States, but you know, year 012-34-2014, you know, the early years of education, I started off there because I thought that was my, my calling and it was interesting. But I've taken those sensibilities of small group work and inquiry from primary and I use it for secondary also use it for teacher education. And this idea that people can work together on Small, small tables. They can be more attentive to other people and how they're solving problems rather than a whole class. It's hard to be attentive to everybody else because you're a social Adam. But on a table there's interaction and it's not just about people sharing, you know, a way that they've done a problem. It's the way that they've been as a mathematician doing the problem. I had one interesting student who was high performing student and he would always, when he thought he had come to a response to a question that was reasonable, he would ask people, have you finished that question? And eventually say yes. And he'd say, fabulous, what did you get? They all compare and find they all got the same answer right, whatever it might be or same response or similar response. His next question would be, I got it this way, how did you get your answer? And he started collecting multiple approaches to problem solving and that was his key to success. It wasn't memorizing a curriculum to reproduce in the exams, it was memorizing, no, experiencing ways of thinking that he could utilize in the examination. Because in the examinations here for matriculation in Victoria, the questions are written in such a way that we have according to Bloom's revised taxonomy. So we have some easy. We have three outcomes for mathematics, for example. So first outcome is, can you do things mathematically in the standard formats with the conventions that are used in the field, build certain manipulations, whatever they may be, the sorts of things that a machine can do. And then outcome two is, can you do mathematics in unfamiliar circumstances? And more importantly, outcome two asks can you generalize your result? Something that machines not so good at doing, Getting better of course. Mathematicas in the lead there. And then outcome three is, can you use technology and computational thinking to extend and explore the ideas? Right? So those are the three outcomes and we write the questions to have students respond in those ways in the exam. Now if they just go the recall route, okay, they get the outcome one low order questions out of the way. That's maybe 20% of the exam. The higher order thinking, being able to think mathematically or in a physics way, they get the top marks. It's not because they know more, it's because they think in diverse ways.

Interesting.

I think this sort of metacognitive ability is a rare thing that you see happen spontaneously, right? Because people need that modeled and need to be able to explore that process for themselves and talk to others, as you critically pointed out, to be able to get the Broader perspectives on how to problem solve.

Can I ask what may seem an obvious question, but from your assessment, why is it better to teach at these higher levels of the taxonomy? This image, by the way, is from the Wikipedia article on Bloom's Taxonomy. I've seen it crop up a few places, wanted to have it up on the screen for anyone wanting to review.

So Bloom's Taxonomy and Bloom's Revised taxonomy is an interesting document in our matriculation level here in Victoria. Our physics study design actually has Blooms Textilen on the, on the fifth page. So when the students read it, they see this and the idea is that they should be mapping their response to the stimulus against Bloom's Taxonomy. And if they find themselves hanging around that remember area, then the response is, well, it's just really partial. And we now know with ChatGPT and using Mathematica and other offerings, the machines do the remember very much so. Look, there are useful things to remember if students are doing mental arithmetic and these sorts of things. But again, they can remember things from experiences. If a student is learning how to make 10, for example, in year one two, do that through experience and then they can remember that particular fact through the experience, rather than just being told how to make 10, you know, 4 and 6, 3 and 7, that sort of thing. We're interested in students going towards the create aspect because the tools. There's a. A democratization of digital tools and it's been around for a while and it's, it's increasing and it's pervasive. It's interesting to observe that people who go into electrical apprenticeships, so people who wish to become apprentice electricians in Victoria and Australia, they're doing the things in their apprenticeship that I was doing in physics back in the 80s, right. And they're using the tools, digital tools, to be able to do really interesting things and develop responses to people's needs. From an electrician's point of view, right? So technology is pervading all sorts of industries and careers.

So what's becoming the industries standard was.

Cutting edge not too long ago, is an everyday. So an electrician now isn't just putting in a PowerPoint, they're thinking, well, home automation. And if I'm looking at home automation, I've got to take into account what am I going to measure in order to be able to have the house respond to the needs of the user or the environment or whatever it might be. But these are essentially creative activities. And Bloom's taxonomy, it's given as a pyramid, right? But really, it's more of an Euler diagram or a Venn diagram because create, incorporates, evaluate, analyze, apply, understand, remember, evaluate, incorporates, analyze, apply, understand, remember, analyze, incorporates everything. Right. So the idea that somebody is just creating things and it's a free for all and they're not really doing the intellectual work is not true in my view, because in the creative process, especially when we incorporate reward for risk taking, evaluation is in a risk taking activity. When you're developing an intention with respect to your response to a stimulus, evaluation is pretty well the second thing that you will design. And you're doing that because you're taking a risk and developing your intention. The intention is determining or intention is broad. So intention isn't just falls in the elderly. My intention is to prevent those falls in the elderly. It's more than that. It's my intention is to understand the causes of falls in the elderly. Right. And part of that understanding would be to understand the psychology of falls. It's to understand the physiology of falls. It's to look at the latency of signal processing in a person, how gait changes when we walk over obstacles. The intention includes how am I going to plan my response to this? So how am I going to look at the machines I need to use to be able to develop broad solutions? How am I going to manage time in a workshop which I'm sharing other people? How am I going, what do I need to learn? How am I going to manage that learning process? The intention is very, very broad. It's simply, it's not just simply saying I'm going to build this, it's saying I'm going to respond to this particular stimulus. And to respond to the stimulus, there are all sorts of things I need to do as a practitioner systems engineer, which is to. Which is to plan my work, plan interactions. And what's interesting about this planning and this intention building and it's part of the risk, and this is what we've built into our study design in systems engineering is that as in life, with respect to somebody working as a physicist or an engineer or a mathematician, the plan seldom goes to plan. In fact, it'll be unusual for the plan to go to plan. And one of the things that we, one of the tools we use to see whether a student developed an intention effectively is at the end of their project when they submit their work. If the plan has got no corrections or changes or additions or different headings, then we expect that the plan was written after the project was finished. But what we've built into the study design for Systems engineering. And I'll just see if I can pop that up on the screen.

Yeah. So while you're bringing this up, you have spoken about using systems thinking and modeling not just as a practice, but also critiquing the practice of modeling as part of what you want students to be thinking about. So can you speak a little about. So how do you present that to students? Right, because modeling is something that we spend a lot of time talking about, as you mentioned. So you've also got techniques for bringing quantitative specific models into given situations that are accessible for students. So can you show for our audience what sorts of things do you, what techniques do you recommend if they want to do that sort of thing in their own teaching?

Well, there are some good books out there that I think I use sort of think about. But as a general idea, overarching modeling applies to statistics, that applies to map making and so on. Is that models, statistics, maps must necessarily lie. They have to tell lies in order to be able to communicate certain ideas. So in map making, for example, or map presentations, those borders between countries aren't the thickness of the line that's dividing them because that line is actually a significant surface area doesn't exist. But of course we have to do that lie in order to do that. From a physics point of view, cows aren't spheres. But physicists are famous for using a sphere. And we like to use a sphere because it's a best case scenario for a lot of physical phenomena like surface area to volume ratios and so on. And if that best case scenario falls short, then all scenarios are probably going to fall short. Right? But the fact that things aren't actually spheres is a shortcut way to actually explore an idea. But in the end we want to make that model based on spheres, which is perfect, idealized, you know, almost a Platonic idea. Right. Of the forms. Right. Even though of course sphere isn't a Platonic volume. But we know, but we make that sphere less perfect in order to be more realistic, which is something people don't think about. People say to you, I'm a perfectionist, you know, and in the end, and it's like, no, no, you need to be a non perfectionist in order to be more realistic.

It's so true that you can absolutely go too far in either direction with this. I love the quote by Neil Gaiman in his book American Gods, which uses the classic quote, the map is not the territory. Excuse me, I'm hearing my own voice twice, so I'm messing up some syllables. But he points out that the most accurate map possible would be exactly accurate and therefore the the size of the territory because you need that much size to fill in all of the details about it, which would make it 100% accurate and 100% useless for what a person wants a map for.

There is a wonderful short story by Borges called On Exactitude in Science and for me it's my lodestone. It is. I thoroughly recommend that anybody listening go and get that. It's one of the shortest short stories around it's butter page and perhaps we can put it up as a resource at the end because it's now I think publicly available.

I want to see what you were about to share with us, but before we do, I wanted to ask a question about the taxonomy and the tools that we've been discussing. Goodness knows we like educational technology here. We're a big fan of it. We're a company that creates a lot of educational technology and there have been some recent writings by Conrad and Stephen Wolfram that are significant to that topic. Conrad wrote recently about how the addition to Chat GDP to the world might mean that education needs to adapt to the reality that this could be a useful tool in a lot of industries and a lot of professional use of language. And at the same time Steven recently wrote Stephen Wolfram recently wrote about how ChatGPT is extremely powerful but that it doesn't really understand the answers that it's giving. It doesn't fact check anything against any sort of reliable external checker and for the most part and as such it can give very convincing information about something that is not actually true. So with all these thoughts and details rolling around about this technology and I'll stop sharing this because it's to truncated, I'll use a better version of it. I want to ask you mentioned and I agree that Bloom's taxonomy is so useful because each level sort of implies and includes the previous ones that you can't really analyze something without also recognizing it and also understanding it and all those levels before.

I also like your way of thinking of it as a Venn diagram more so than a pyramid. But.

But it's interesting that article from Stephen and or Conrad, I can't remember which author but that piece to me was great because using WolframAlpha of course WolframAlpha doesn't just state something, it will give you the providence of whatever data it's.

Using.

And or explanations of the equations that use to figure these things out.

Yeah. And also allows you to clarify whether your meaning was understood right so there are some really interesting aspects to Wolfram Alpha that go beyond what Chat GPT does. But what's interesting is we should simply be using the thing. Tycho Bray was famous for saying, I will not look through a telescope, I will use my unmediated eye. And he claimed to be able to do all sorts of things, measure to small, small angles, that the difference between star positions and so on. And you know, that was a, well, a romantic view about the unmediated human and how it should be able to perceive the cosmos. We know now that that's really not the way to go because there are things that we can't see with the human eye that machines can. And our job is to just collect all that together, sense of what the world might be or what the universe might be. Try not to be too teleological about it. But just using machines and instrumentation is important for us to be able to go forward and do the creative work. It's clear that Chat GPT doesn't do creative work. And I think we had this discussion before. I can't remember if this is off air or at the very beginning, but.

No, make some interesting poetry. But that's a separate. Sorry, continue.

But it's, it's an interesting activity for students to use ChatGPT and then to interrogate it for what it says. So I had it recently write a 250 piece, 250 word piece on a particular text. And what I noticed immediately was it was taking common sense understandings of words and then trying to put it together with the sequence of words in the text and then state something. And it was wrong, profoundly wrong. Now what's interesting about that, that drives us to be able to create alternative solutions to getting a machine to write something that's compelling, right? It feeds back into itself and improves itself. It's the ultimate hermeneutic process, isn't it? And to have students use the tools in a critical way is interesting. So they're involved in creation. But we remember that when we're creating something, our task has to incorporate, evaluation, analyze, apply, understand, and remember. So just saying create and your rubric just has create on it is certainly insufficient because you can fake create if you're not engaged in that evaluation. Analyze, apply, look. And evaluation is broad, right? Evaluation isn't just saying, oh, this source is reliable, and so on. It's also asking, you know, questions of efficiency, right? Was this the most efficient way to get to this particular response to this stimulus? Is there another way? Now ChatGPT is on the surface of it efficient, but I would argue not efficacious. Right. So because it's, it's, it's removed from a whole lot of types of engagement that this person really needs to get through in order to be authentically creative. They need to be able to analyze, apply, understand and evaluate and so on. And that's where our brains come in.

Speaking of having this list of action verbs that go along with features in the taxonomy that you're aiming for as learning goals, one of the things I was most impressed with in your assessment criteria were the very intentional choices of action verbs that you had settled on. So one example progression that I can give for audience members is there's one assessment criteria where if students are listing things, okay, that's good, that's where you want to start. But really you want to be describing them. But actually you don't just want to be describing them, you want to explain them. But actually, no, you don't just want to explain them. You want to justify the decisions that you're making in terms of your design and in terms of how you're going to respond to what's happening in your design. And you want to justify those things in terms of the way that you explained it, in terms of the way that you would just describe it without so much explanation in terms of the list. And so that progression of action verbs, that's tied in with your assessment criteria, that's also tied in with the conceptual framework that you have thought was super impressive. So the question that I have that's in that is, is it just thinking about this taxonomy of learning goals or is it also an interplay between the learning goals and the tools that you use, or how do you come up with such convincing, you know, progression of figuring out where the student has gotten in their learning in terms of being sufficiently open ended so that you can have a useful, you know, keyword that teachers can think about, but also precise enough so that there really is a difference between I explained versus I justified versus I only described.

Well, what I'll do is I'll show you a document and we can share that with the audience. So, and this document informs education in Victoria from prep, so very first year of schooling right through to year 12 and pre university. So let me say share here and let me see if I can find that document. I think it's that one. So you should be able to see glossary of command terms. Is that what you see?

Yep.

Okay, so this is, you can see here, it's Victorian curriculum F10 VCE study designs and VCE examination. So this informs our total education offering in Victoria. And here are all the command terms. And this is accessible to the students. So part of the education journey is that they learn the command terms or better, they map what they've done to the command terms. And we talk about this as being part of the moderation process. And we should talk about moderation because that's an important process that we have in Victoria. It's not unique to us, but it's something that we're quite proud of. But you can see it's got here a definition of what is meant by explain in the context of our curriculum documentation, identify, have a look, investigate on the bottom there and investigate, observe, study or carry out an examination in order to establish facts and reach new conclusions. Well, that's interesting because that's saying don't follow a recipe book experiment and just do it for the sake of. It is actually saying, now there's a purpose to any investigation and that purpose is to reach new conclusions. It's not to say, here's your demonstration laboratory, we're going to turn this solution a different color. Good luck. It's saying, no, the purpose is. And so teachers have to write their assessments in order. So if they're assessing investigation, there has to be this sensibility of what an investigation is. And I love that.

The example you gave about people just studying, studying what is a forest. When some people were looking at ph and light, they weren't just doing that to say, I want to know what the ph and forests were. They were doing it to see when areas stop being a forest due to intervention from human action.

And isn't it interesting because there was not one objective measure. There were multiple measures that could be used to draw a conclusion. And none of those measures was the forest. They were proxies for this idea of forest. And that meaning didn't become obvious until they actually had displayed the information, the data in interesting ways. And of course we know that Wolfram Mathematica is excellent at being able to display all sorts of data and information in different ways so that our brain can then try to draw conclusions. And this ability to represent information data allows people to truly investigate, draw new conclusions. And you can see here we've got justify, so show, prove or defend with reasoning and evidence an argument, decision and or point of view given data and or other information. That's justifi so again, a teacher has to map their assessment to this verb to show that they're demonstrating a justification has happened and what's Interesting is that it's got reasoning in there. So a chain of thought. Now that chain of thought, it could end up being an erroneous conclusion, but the student can be rewarded for a chain of thought. That makes sense. An argument of some sort of. And it's interesting to me as well that there are the word decisions in there and to me, a sign of a good curriculum document if it's got the word decision in it. And in fact, the mark of a great assessment is if decisions have to be made. And the decisions. If a student is factorizing a quadratic equation just for the sake of factorizing it again, something you just don't do in real life. Because first of all, natural phenomena doesn't end up with integer coefficients. Right? So you get a machine to do it, and then your job is to just display it and then work out what you know. You might be able to think about the circumstances you're looking at.

What does it mean?

Yeah, a decision making process is what we went through with that forest example. The students had to make a decision about what they might measure. They had to make decisions about how they might represent that information. They had to make decisions about how they might map the information. All sorts of decisions were in there. Now, if the students are making decisions, they're taking risks. And if there's an opportunity to take a broad range of decisions all the way from planning the investigation to managing resources, to thinking about planning, activating learning, all these things are decisions that can be made throughout their engagement with the stimulus and developing a response. And if they're making a decision, there's an opportunity for them to justify that decision. And there are some important elements here, because if we. Justification works really well when there's an opportunity at the very beginning of the project for the students to develop an intention with regard to their response to a particular stimulus or range of stimulus stimuli, lock that away, carry out that intention as they're going through the plan. Doesn't go to plan, little train wrecks happen as they're going along, have to make new decisions, new plans. Sounds like the polya process for mathematical problem solving in a way, doesn't it? Right. Not surprising. They keep doing that, that iteration. And it's not just iteration of the design, it's iteration of the plan to be able to respond to the stimulus. They get to the end of the study, they compare what happened to what was intended. This is how our study works. We actually have three criteria, which is about developing the intention locked away, and then the other half of the Study is, well, the middle part is carrying out the plan and the last part is being rewarded for the plan. Not going to plan. 25% of the score for their level of achievement is given for the plan. Not going to plan. The whole project can be a train wreck. 25% of the mark is there for them to go back and interrogate the intention. Were there things that I should have anticipated? Were there things I couldn't anticipate? Let's say Covid, for example, you wanted to use the machines, couldn't access them for access reasons and so on. There are all these things that they can be rewarded for. But what we can make this document for command terms available. Now, these command terms aren't just unique here. I had a scan of some documents for mathematics in the States and there's evidence of those command terms in the curriculum in the States too. Now, what's important about command terms is they're written into the curriculum documents. So you'll see these command terms sprinkled throughout. And one nice example is in year eight, year nine, mathematics students are asked to investigate similar triangles. That's the statement. It's not students will calculate with similar triangles or students will recall similar triangles. So investigate.

This is a small difference.

Oh, it's, it's more than subtle, isn't it?

It's a huge. Yeah. The effect is, it's like massive because.

The students have to actually design the investigation. And if a teacher does not develop a situation where they can do that. Now, one of my favorite activities for that is of course to measure the shadows of various objects in a at a particular time of day. Go and put those into a table, generate a curve for it. It's typically a pretty straight line, depending on how slow people are at measuring these heights. And then using that to predict heights of trees that you can't measure the height of because they're too tall and you can use the shadows and so on, well, they can then draw conclusions about similar triangles. What are the key properties? What are the relationships that are important? That's teaching the curriculum. If a teacher just gives an exercise from a textbook where all those relationships are just laid out, they're not teaching the curriculum. And the assumption is in matriculation that they do have those skills to be able to demonstrate later on.

I'm loving all of these conversations, want.

To keep them going.

But we're about at the one hour mark, so I wanted to make sure that we addressed a question that came up in the chat from a viewer earlier, which was do you Think AI will make educators and possibly our current idea of education obsolete. The viewer was talking about how well AI can assess the individual needs of learners at different levels of their development, different ages. And does that mean that given enough proficiency with these things and AI will be able to be the teacher someday?

Well, it's hard to say where AI will end up, but let's sort of place ourselves in the here and now. One of the things that we have talked about with educators across the state is if you. Well, teachers were teaching remotely in Victoria for a significant period of time. That was part of our response to the COVID outbreak. And one of the things I noticed when people came back, I warned them, I said, don't teach as if you're remote. And we had some teachers, and it was because of stress and burnout and so on. They were projecting textbooks onto a whiteboard and then pointing at it. And it looked. It looked like observing the classroom as if they were still teaching remotely. Right. And so AI can get back to your question. I can assess some discrete skills. Can you factorize this quadratic? Can you fit a sinusoid curve to this data, whatever it might be? But what I can't do at the moment is ask interesting questions by itself because it's not in the mind of the people who are learning the subject in the classroom. It can't, at the moment assess dispositions very well. It may well be able to come to a view about assistance, say, and ability to stay on a problem and continue grinding away. But I might be able to give you a number for that. But it can't teach you how to be. Have the disposition of a mathematician with regards to heuristic processes, whatever it might be. So if teachers teach, there are interesting aspects of the curriculum, both in the States as well as Australia, if teachers look at the verbs and use that to inform how they're going to design learning experiences and effective assessments. AI can't do that. Not yet. Right. And if teachers take that route, then they're showing their expertise, right? Which is. And in fact, using those verbs means that you're sort of AI cheat proof. Right? Because in order to be able to fulfill those criteria, you can use any and all tools. Because in the end, if you're creating a response to a stimulus, as we've discussed, that implies that you must have used all those other skills. So your assessment needs to say, well, where was the analysis, where was the understanding, where was explanation and so on, and justification, all these sorts of things. So I believe that, as we are now AI is not going to replace good teaching that's informed by teaching in the discipline itself, which has been a feature of our work here at the academy. Teaching in the discipline means that you're using the discipline and its idiom of engagement with the world to be able to inform creative, critical, analytical, justified, efficient responses to stimulus in the world. And if we design our work and learning experiences and assessments around those discipline dispositions, then we're on our way to having a compelling education that's, that can be informed and helped by AI, but not taken over by AI. Now, I'm not, I'm not suggesting that there will never be a robot in the future that can do all these things and good luck to it, because in the end, if I can sit in the deck chair and think, or the robot does that grunt work, that's fine with me. But at the moment, to get away from that moral panic of technology use and AI, we really should pay attention to the verbs. And let me point out those verbs that John was talking about before. And I know we're over time, but I'll just.

Oh, no, let's continue.

Yeah, and you can see. So I'll share.

As you're sharing that. So I have to say I think that this is a good solution for the question that people have. Right. Because one of the reasons, I think, why a lot of the public discourse is capticated by the idea that, oh, these technological tools are going to either totally replace the need for teaching or are going to revolutionize teaching, it's because we know from our own experience, our educational history, that not all of the activities that we did in our school was necessarily reaching those higher levels of the verbs and more importantly, the doing and the practices and dispositions that you're talking about. We remember times when we did things that were more or less tedious and a little trivial for various reasons, trying to build toward other things. But so what do you think? As actually. So please first continue what you were going to share about these specific verbs and the progressions that they're in them. And then my next question is, okay, so by using these higher skills that build on, you know, simple recall, we don't just want to do simple recall. We want to use these more involved skills. How do you use tools then to be able to achieve them?

I'll take that question second.

Yeah, there's two questions in there, but.

There'S, there's an interesting paper on white box, black box thinking that I'll mention in that regard. So I think you should be able to see the VCE system Engineering school assessed task.

You can, but can you zoom in a little so it more fills the screen side to side.

Great, thank you.

Much better.

Thank you. Great. So here we have level of performance, we have an assessment criteria and we've got some indicators. So this is criteria two. So the first three criteria really are for the students to develop their intention with regard to their response. And these responses are quite broad. So we've had students who wanted to measure pollution levels in the middle of lakes that they couldn't get to easily. And so, and they're all individual, they don't work in groups, it's an individual project. So this particular student developed a boat that would make its way to the middle of the water. The student had developed some sensors. They read some research about detecting salinity and turbidity in water. And what was interesting is that they looked at the papers on salinity probes and using conductivity, really simple circuit and Arduino and so on. And they improved on it because they, they would reverse the current through flow through the probe to actually periodically clean it. So they did some really innovative add ons to what they read in the research. They weren't simply following somebody's research and doing it blindly, then sending the information back to a base and being able to use it for assessment and students making vertical gardens or model farms. And so a broad range of responses. They developed this intention in the first three criteria. And we lock that in because when we do that then we've got a base from which to assess whether the students or how the student made decisions as they were going along. Because when the plan doesn't go to plan, must make a decision anyway. Here you can see we've got at the high end, very high. They have to justify feasible options. So they're expected to come up with design and model options for a response to a stimulus. And typically we go for three, but no more than five things. But they. And the modeling is a combination of mathematical modeling as well as computational modeling. I'll do modeling through Wolfram Systems Modeler, which is one of my favorite bits of kit out there because you can pull Wolfram system models into Mathematica and manipulate them and demonstrate the model. So the students present not this static document about their model. There's something you can actually play with the model and you could see how.

It works, what would change if you change some of the parameters.

Just beautiful. I really enjoy it.

You've also created, you know, training videos for teachers to be able to do this. Would you mind if we put you on the spot slightly? Could you give us a brief just tease of what sorts of things you would recommend that people dive in and do with system modeling?

Well, there's, there are a few things that are really interesting to do in Systems modeler, and it's the cave with Systems modeler is to start off with some very basic models. So a bread and butter circuit for system engineering students, and I should say in systems engineering, the students are actually expected as part of their study to develop a integrated controlled electromechanical response to a stimulus. So it has to have electronic elements, it's got to have mechanical elements, it has to have some sort of feedback so that it can respond to its environment and, and self corrections and so on. And this is really neat because in Systems Modeler you can use Model Plug and you can incorporate Arduino and you can actually model how Arduino responds. So if it's a voltage divider network, you can build that in Model Plug and you can show the Arduino live responding to you waving your hand over the ldr. And you can see the digital response to the light level. So you can set up some limits. And when the light is between two limits, you get a digital high. Right. And it's just wonderful for a student to show that's what's happening. The other thing is just modeling a voltage divider network itself and then pulling that up in in Mathematica is neat because if you use manipulate command, you can change the resistances and the voltage to wider network and you can show that the output changes proportionally. And if you use a ramp voltage, you can see that. I can show that in a moment.

If you'd like to. Please do.

Yeah, so, but to go over these verbs here, you can see it very high. We've got to justify the, the feasible design options. At the high level it's explained. At the medium level, it's describe. At the low level it's identify feasible options. And then the very low is also identified. So you can see the progression of verbs. Yeah, and this also emphasizes that if you can do identify, great. Because what we do as part of moderation process is that we also work with the students to use the rubric to see whether what their response is doing. And if they map themselves to identify, what's interesting is they can go and have a look and say, oh, okay. The next one along is describe. What do I need to do to move from identify to describe? They can go to the command terms and then they can go through that process. When they think they've got to describe feasible options, they can go and have a look and say, well, explain. So what's interesting about a rubric rather than just a score, because you can have four students get 67% in something and they've got 33% differently. Wrong. Doesn't give much information. Right. Whereas a rubric gives information because they can see where they are and where they. What they might need to do to go on. And this study design is full of this. And of course, we'll put a link to this rubric in the resources later on. Let me just.

I also especially have to say that, you know, this, I think, is an excellent example of using rubric as a tool for students to learn. Because I've seen this tried to be implemented in various contexts to a greater or lesser degree. If you're working on a project like this, trying to rethink, okay, are my assessment criteria really what I want my students to be using for their learning? This is a fantastic example, I think.

And what it does too is it provokes the teacher to keep thinking is what I'm doing allowing a response, a full response to this, to the whole range of criteria. If a teacher gives students a kit to do something, well, they don't get anywhere on any of this, at best is very low. Right. Because if it's a kit, there's no design response possible. And certainly they can't justify something that somebody else designed that's just not possible. So it puts a bit of pressure on the teachers to think about what they're doing and to determine a more effective or useful assessment. Now I'm trying to find the button to unshare and for some reason it's disappeared on my machine. Oh, here it is. It's on my other screen. I've got three screens going, so I'll stop that one.

There we go.

Now I'm going to, I'm going to share just a small model. That's that one there. So I'm not going to show the systems model for this, but you should be able to see a manipulate plot there. So this one is for an RC network, so resistor capacitor network. And what it does is it relies on capacitor charging at a certain rate and we use a time constant. And that acts as a proxy for. Well, you can, you can measure capacitance in a way with this if you calibrate the. The system. So if you've developed a sensor which is based on Capacitance. And we have, we have students who work on systems for horticulture and so they want to work, measure soil moisture. And so you have resistors, sensors as well as capacitive sensors. And there's a good argument to say that a capacitive sensor is much more reliable. All right, so this, so this is in a notebook, of course. And what's cool about notebooks is that you can have, you can go and throw your diagrams in there and all sorts of bits and pieces of information, but you can also have a model that works. So this model pulls out a, an RC network that I built in systems modeler. It wasn't too difficult, but of course we can change values here and the model updates itself and changes the curve. You can see I've put a line here for, what is it, 63% charge. And you can see how the time constant changes for the capacitor. I should also have one for motor in here. So let's have a look at that.

Awesome.

Yes, great. Where's that one gone? I think what I have to do is stop that and up here. So one of the things that we're interested in, of course, in system engineering is noise. One of the, again, one of the reasons why we need to practice the idea of being a practitioner of the discipline, because noise is no simple matter. Now I'm sharing that.

Are you referring to. Oh, you mean literal noise? When I hear noise, I think random changes in data.

Well, that's of course another thing that's of interest to us all. But this is motors. So I'm putting capacitors across. Motors is pretty important to be able to reduce noise, especially if people are using microcontrollers and so on. So this model here, I think you're one enabled dynamics there. There we go. So here is some static models, of course, let's pull the model and I'll just show you how simple the model is itself and we'll just let it, give it a bit of time to go and evaluate itself. It might be. So, of course we can change the gear ratio and we can have a look at the two inertias in the motor. And what students find fascinating about this, of course, is that, oh, these lines are curved.

Right.

It doesn't just automatically go up to bang level. There's, there's a bit of latency. Right. And then of course, this model is not pulled in. But let me just share the model itself just to show how easy it is for a teacher to build a model, but also for students to build a model.

Sure. And while you're pulling that in, I guess I'll comment briefly that, you know, being able to actually model something makes all the difference in seeing that. Oh, that's actually not going to be a linear thing. That just goes from A to B. Right. There is some more complicated dynamics here because one of the issues that a lot of teachers run themselves into is that, okay, I want to teach about some actual phenomena. I want students to answer questions that they're having. And yet if I'm having to sit there and calculate things by hand doing, you know, complicated non linear dynamics is probably not what you want to do by hand.

No. And a student doesn't have to be able to do that to actually see it. Right, right. Because they've used the tool. So that means a student who is in, you know, from year 10, 11, 12 can play with this idea and understand what's going on without having to do the calculation. Now I think I can share that now. Let me see. Should be able to see, I think this model here, this is the RC model and you can see how simple it is to build it. It takes nothing to play with the idea.

This is not a graphic you created to explain the concept to students. This is the model itself.

That's exactly right. And also with the improvements to SIPs of modeler and the simulate lab as well, where you can actually go and manipulate the model in the simulation is a pretty useful thing. But I prefer to always to pull it into Mathematica because it means that the student has got a narrative in the notebook including all the things that went wrong. So don't delay anything. All those models that weren't where didn't do what you expected, they should be there too, because that's part of the narrative and that means I can reward you according to the rubric.

I want to throw two things into the mix here. First of all, I love what you're talking about, about failure being crucial, that it's one of my least favorite things about the song called Sage on the stage method of teaching where a teacher comes up with basically the message that they know everything about this topic and will give the information to the students. It gives the impression that all the work they're doing is correct all the time because of their skill. As though at every level people aren't making mistakes constantly, which of course they are. They are either wrong about something or they discover something completely different than they were expecting. And they need to be able to know how to adapt to that. And I love that you are making it part of education to show A the experts are doing this and B learning what to do in that situation is a big part of becoming educated. And I also wanted to throw into the mix. I wonder if that's one of the things that will keep a human element in teaching no matter how good AI gets. I mean, it's weird to think about, but in a big way. Humans are very good at making mistakes.

That is brilliant, Noah.

That do that as little as possible, but it keeps happening when far as we're concerned. And that that may be a crucial part of education, that a machine designed to do things as correctly as possible, even though that will keep making mistakes too, is something that a machine would have trouble teaching. Mainly because when a machine has an instruction on what to do if something isn't working, the instruction is very often stop.

Or the blue wheel of death sometimes. That comment you made is just fabulous. It's given me real joy, Noah, because that's the ultimate response to the earlier question about AI in education. AI will never make mistakes as well as humans. Can you just. That is wonderful. In fact, we should get together and write a paper on that.

So that's good insight.

Yeah, yeah.

Because we're not just talking about being wrong. ChatGPT is just fine at being wrong. What we're talking about is noting your mistakes and learning from them in some way actually reacting to it.

Indeed. Right. And that's that, that true understanding what an agile process is, which is, you know, this idea of being approximately probably correct. And, and just keep shaping, shaping, shaping. And the shaping isn't just working on the problem itself. It's about, well, is there some aspect of the system I didn' about? Is there a bit of learning I need to design into my experience here that I can better understand this and so on. It's. Yeah. Really? That's a compelling statement there. I love it.

I want to tie this back in with the brisk that you were talking about before. So one piece that we didn't talk about explicitly too much yet in this conversation is so the teacher, you said really has to model that process of how to get yourself out of a train wreck. And the part that we didn't talk about is how do we make that something that the teacher feels comfortable doing? Right. Because I think there's a part of literature that's about how teachers are generally not inclined to do that for various reasons that putting yourself into that train wreck situation live in front of your students is not always the place that they want to be. So how do you make it okay to be there?

Well, one of the things that the Academy is doing through its fellowships program, which is beginning this year, and one of the fellowships is industry engagement with teachers. Teachers will get a fellowship in order to work in industry and actually see what it means to work in a discipline, whatever that discipline may be. And again, we're multidisciplinary across here. So that's one thing. The other thing is for teachers to reach out into their communities and find those places where the discipline can be made to work. If it means going to your local council and the local council identifies an accident spot in traffic in the local area, then working with council to work out some way of measuring some data to get an understanding of what that might be. What's nice about that is everybody can take that risk not knowing there's one way to collect the data collected multiple ways and play with it and to have this playful aspect to the subject itself. I think also if a teacher is attentive to the verbs in the curriculum document, you know, when it says things like investigate, it's not, it's, that's completely not being a sage on the stage, right? It's, it's saying that you need to work with the students so they can design an approach towards this look, it can be daunting for people. And you know, one of the things that we try to do, and one of the reasons I made those videos and I've really got to get on to making some more, is to avoid a trap that can happen with support for various software where it starts off with some sort of elementary easy things you can make or build or play with and then suddenly goes to level 11. We've got to have all those little progressions in between, but also make some sensible steps as to what's a useful thing to model. So voltage divided networks are really useful things to model because it's a bread and butter circuit. You can measure pretty anything with a resistive sensor. RC networks are similar. And also realizing that you can't break Mathematica, you can't break systems modeler, right? It's. You can play with things and you can version what you're doing and you can actually see your progression and you can see your growth. And sometimes not being aware of where we began or remembering where we began to where we are is, well, that's really important for people to be able to do, right? So yeah, because we, we lose sight, we keep thinking, oh, we're not, we're not expert, we're Not Stephen Wolfram yet. So we've failed. It's like, no, no, look at where you were. Look at what where you are now. Same with learners as well. And that's why using a rubric is useful because a learner can go and plot where they were at the beginning on the rubric, see where they are at the end. And if I. Let me just share too, please. We also encourage teachers, and some teachers do this to work on a project at the same time as the students are if they're doing system engineering. And what's interesting about this is they can. Well, they're exhibiting what it means to work on a project and to find something to do really challenging. And I should note as well that a student's response to a stimulus for their project could in the end be a bust. But because we've got this agile sensibility where they're iterating designs and so on, that allows us to have data with respect to the concerns of the study itself. So even though ultimately the project might, you know, truly fail, an example of that was I had a student making a, a short throw wheelchair lift for somebody in their street for a particular situation. And they did some beautiful modeling. I did some virtual modeling. They calculated the vectors and then they made some physical prototypes and beautiful, beautiful, beautiful. Did all that modeling, produced the device. Device failed because the thing they didn't anticipate was you can model the dimensions of sheet metal and you can make a physical scale model of that, but some of the properties of that sheet metal don't scale and you get warping the real thing. So the project ended up being a bust. But because of. And I'll just share this because of this. So this is our project management to realize preferred option page. They can talk about. Well, what, you know, was this thing that failed. Something I could have anticipated at the beginning? Well, no, right. I didn't understand that the scale material properties was something they ought to have done, but they know it now. So you can see independently demonstrate skills in time management and organisation to produce a preferred option independently and describes evidence of progress and risk assessments for production work and diagnostic testing and justifies decision making and modifications if required. So the thing they've made decisions along the way, it's been a bust, but they've still made those decisions. We reward that decision making if that decision making is justified in a meaningful way, you know, and that's criteria seven and you know, criteria eight actually asks them to evaluate their use of the system engineering process and looking at diagnostic testing and Predetermined criteria and evaluation. As we said before, you design your evaluation at the very beginning when you're conceiving your problem. And even the evaluation criteria itself is up for grabs. Right. As far as critique is concerned and explain recommendation to the design and production of the system. What would they tell their past self if they had another go? Each of those two criteria, 12.5% of the whole project, it's 25% altogether. And the project itself accounts for 50% of the score in the study at the matriculation level.

Let me echo what John was saying and saying. I love the idea of rewarding intelligent failure, rewarding seeing how people learn from it, how they react to it, how they agilely respond. But I want to play devil's advocate for a moment here and ask how would you respond to someone saying, well, in the real world, in the industry, there's consequences for failure, there's consequences for your thing not working. Shouldn't students experience those consequences in class? Shouldn't they have negative repercussions of their project not working?

Well, that's a really interesting question. Yeah, because in older design paradigms where it was things were just a one shot, you designed it one shot, you know, and you can think of, as much as we all love Lewis Mumford, there are a whole lot of things that he designed that were really a bust and they were one shot projects. The Mexican car and so on. The agile processes which we've written into our study design say, well, have a look at whatever it might be, talk with the needs knowers, think about the system and conceive of it in the way that you can at the moment. Because part of the criteria 7 and 8 is, well, the system I realize is bigger than I imagined at the beginning because the person who I'm designing for, responding for, they're part of the system. I neglected them, I just thought it was me. But the agile processes, say, would argue that they are more efficient at getting us a response to market. They would say in their language because they keep checking in. And part of that check in is an appreciation that perhaps the dimensions of the system are going to become more extensive or perhaps I can constrain them more. Whatever the response is, I can test things before I've gone full blown. And with the addition of additive printing and CNC machining and all these other technologies which allow rapid prototyping, people's design responses are seldom now one shot responses. They're more agile responses. And again, this fits in with sensibilities that are not unique to engineering or physics because of course mathematics has its heuristics, English has its hermeneutic sensibilities and so on. So all the fields, you know, science has fossil fuel, falsification is a key element of its design response to experimentation and so on. I would argue that there are consequences for things going wrong, but the consequences I think are incremental and instructive or agile responses. For one shot responses, they're more punitive.

And a lot of what we hoped would be no longer implemented, although frequently is is assessment methodology tends to be that more one shot rather than the wrong but instructive incremental assessment. So luckily I think there's been progress in recent years in getting away from that and hopefully with more technological tools, more of that progress will happen.

Well, so that was interesting for us. We had during COVID when students couldn't access workshops, so learning from home for a significant period of time and I had to rewrite the study design to take all that into account. I had to make very few edits to the study design because, well, let's say the student was designing a response to some stimulus and they modeled it in Systems Modeler. Right. And showed it in Mathematica and they iterated that virtual representation that satisfied the criteria without a physical artifact. Right. So we can, we're able to reward using those tools and for students to, as you say, prototype model and so on and to use that as their learning process towards what being agile might mean.

Fascinating. So it's sort of challenge agnostic that whatever is causing people problems is something they'll react to. And that's part of the investigation, that's part of learning.

And those sorts of unstable situations are rich in decision making. Because ultimately did Covid affect a student's ability to make decisions? No, in fact it gave more decisions. Yeah.

It may have affected their options that they could decide between, but they could still make decisions.

And it's part of the system.

Right.

So because it's part of the system in criteria 7 and 8, they can then say, well, when I started the project, the system was this. Now the system is within this broader context of a pandemic and our pandemic response and responsibilities for that. So therefore I've made these decisions, these particular decisions panned out, these ones didn't. So I made a new decision and here my justification and so on for them.

One thing I might suggest, based off of our previous conversation about tools like ChatGPT and also having this rather than just using recall and the first step in a Larger process being what we're assessing. In the We Were Wolfram Technology conference talk, you talked about having these multimedia technological artifacts helps avoid the burden of writing prose for every single step. Right. Prose can be helpful, but sometimes the story can be told by other things. So using a large language model to sort of jumpstart the writing process and, you know, skip to the editing where you're using your, you know, you're a little bit later into the process of what you're trying to accomplish. But rather than saying, oh, my first draft is wrong and I need to revise it, you can use a lot of the same skills to say, oh, ChatGPT's first draft is wrong, I need to revise it. Right. You know, sort of. In fact, peer editing is an activity that gets used very frequently and is very beneficial. So you can sort of use a large language model as the peer in some sense that you're editing. At least that seems to be. What do you think about that?

Well, it's interesting to me because I have never explicitly taught students how to write a lab report. I've crowdsourced it. So I've set up forum and in those fora students, students could only see other students submissions once they had submitted themselves. And I'd. And I first did this back in 2009, I suppose, using Moodle platform because it allows you to have that sort of forum. And the students were really struggling if writing a lab report. In fact, there was a whole collection that just some were getting into sort of minutiae and others were broad. And it wasn't really satisfying my desire to have the students communicate clearly the narrative of what the experiment is. There was an interesting article in the New Yorker where a mathematics professor dropped out of academia for a significant time because the refereed journals wouldn't allow him to publish the mess and all the wrong turns on his research. But we'll leave that aside. So good. It's a good article to read in New Yorker. I'll see if I can pull the reference up at some point. So I set up this fora and just said, look, the idea is we need to communicate this information about this experiment. Off you go. And your job is to write your first idea about what the lab report could look like. And then you must critique six others. And that was the instruction. At the end of the week, they were all writing lab reports. And it was really interesting for me to look at the discussion threads that were going through because students started off being. Some students became really defensive about because I thought, oh, People were attacking me. But then the students managed the conversation and relationships themselves such that the students, even though defensive, they still felt I might try out what they're saying. And when they did it and then looked at the rubric, they realized, oh, now I'm doing that thing that the rubric is asking for, whatever that thing might be, right? And at the end of it they'll. The lab reports were what I wanted. Not only that, but we had some innovations. One student put an animation of the experiment as part of their lab report rather than just describing it. Which brings us to, you know why a Mathematica notebook is so useful? Because that narrative is super important. And I think we always have always crowdsourced our work. I think in general, people don't go and write a paper to be published and they're the only eyes that have seen it. At minimum, the editor is going to see it as well, right? And the reviewers will see it and send the thing back. So we've always crowdsourced it. So let's be honest about it. And then, you know, get the machines in, do all that, that sort of thing, but incorporate that part of what you're doing into your Mathematica notebook. Because the Mathematica notebook can record everything. That's what's beautiful about it. So, you know, it's the rule really with Mathematica and that books is you don't get rid of anything. You just. You leave it all there so that you could see where you were.

Visuals, the parts that don't work because.

You wrote it wrong, it redefines show your work.

Yes, exactly. And WolframAlpha inquiries that came out with things that you didn't really expect and you had to refine while you just keep them all in there, including the refinements, because that's important for you to learn how you might make a query that's more satisfactory and so on. But this narrative that you talk about, John, is. Is super important because it's the narrative which expresses this person being a practitioner of the discipline. A bit of working out in a corner doesn't express much as a technical achievement, but it's not actually talking about engagement with the discipline as being a practitioner. And our study designs and our curricula in Victoria in general reward that narrative of exploration. Look, and we all have our debates, right? Nothing is either or. So we know that at a state level there is a need to have some assessment which is asking for discrete skill demonstration, whatever it is. But ultimately that should be a small section. There's Got to be a section in deriving a score which is involved with creation and so on. And look, let's revisit that black box, white box thing. So, yes, there's, there was a paper which I keep losing every time, and I don't know why I keep losing it, but it was an Austrian mathematics professor who talks about a white box, black box approach. So there is some argument that it is useful for a student to be able to know, well, you know, here's my quadratic, here are some ways that I can factorize it. And we know that there are forms of the quadratic which are useful in different ways. So, you know, you'll do a turning point form, you'll do a polynomial form, you'll do a difference of two squares, you'll do the focus on directrix form, right? There are multiple representations and it's useful to go through that calculation. But once you've done that process and you know that it's not magic thinking, then it becomes a tool. So then let the machine do that for you. You know, there are different representations, you know that you could get to it yourself, get the machine to do it now, because once you do that, then you can discover new generalizations, new representations, and then they become tools that you can then automate for the next thing you need to discover. And so the idea is that you start off white box. Well, standard deviation calculations, for example. You go through an activity of the students and ask, well, here's some data. It's all over the place. It tends to sort of, if you look at the data, it tends to represent a line that's doing whatever it's doing, right? What would, but not quite because our line is here and the data is here. What might, what might be useful to know about this data and this line that's purporting to represent it. And you can give different ones, right? Some more precise than others, right? So you give that range. What, what could be useful to know in order for you to have to trust this, this line of best fit, whatever that line might be, Maybe have.

Someone come to some sort of realization through the equations and say, oh, it's kind of like, but not exactly the same as the average, that this thing varies from the mean because it's the small squares and then square rooted and like.

Well, that's right. How far is the data away from that line? Which, which one is it? Which one of these displays do you trust the most? Well, this one data points very close to the line. I trust that one more than the one where they're all over the place. We know. Well, maybe it'd be nice to know how far away those data points are from that line on average, right, the point that you're making. So, oh, there we go. We've got, we've got this idea of, you know, variation, spread of data and so on. And, and we can, we can average information that's sort of an average of something else, right. And from that you can actually then derive, you know, some sensibility about what the formula might look like. Right? You do that. You know how it works. But in the end, in the world of big data, you are never going to get a data set from CERN and do those calculations by hand, because that's intractable, right? Even the machines struggle with that one. So you get a machine, no statistician does those calculations by hand. They always get the machine to do it. But they have gone through the experience, that white box experience of not being told what the formula is, but asking questions that might inform what the formula might look like. And then playing with that, and then once that's understood, black box, it machine does it for you. Because the next thing you need to do is to start asking questions about. Well, now that I've done that and I can see. Well, it's a deeper question, right? This, this line of best fit is better than this one here for this data. It's pointing towards noise in the experiments and so on, whatever the concerns may be. You're asking deeper questions because as we talked about before, experiments don't come out with integer values for everything. And in fact they don't come up with the same values for repeated experiment. They come up with a range of values and that range points towards some things we might have to consider when we design the experiment.

Again, this progression that you're describing in terms of thinking about, okay, line of best fit, how do I describe the closeness of data to some model having experiences just with that question and then putting it in a box so that then that box can be just a small piece of working memory so that then I can go and ask more intense questions about whatever problem I originally cared about with this little tiny piece as a box in my working memory, and then I have the rest of my working memory. I can think about these higher order, more, in some sense more complex, but really just different hierarchy of question I'm trying to answer. Rather than worrying about some algorithm that now is a tool in my toolkit, I can worry about what am I going to use it for?

And it Allows you to have a deeper dive into what's important about what, how the world is. Right. The world isn't about counting people and their preference for pets. Right. It's, it's about real data that's quite messy. And it's one of the interesting things that we have, as I discussed before in our mathematics study designs and, and those study designs are all publicly available. If people type in VCAA study designs, Mathematics Australia, they'll, they'll get it up. But I'll, I'll work with Noah in the next couple of days and I'll, we'll put some links up for the other study designs, not just for system engineering. But one of the hard things for the students is to get them away from the idea that being technically proficient with this calculation is what is ultimately important. Oh, my goodness, mathematics at this thing.

Quartet up on the screen.

What you can say about what the calculation exposes is what's important and what we reward.

I mean, there's all sorts of technical data about this. You know, the standard deviation of X and Y, their correlation, the means of them. And that's useful information. But I just love this example, which is up on Wolfram's demonstrations, by the way, for anyone looking for it, that there can be a lot of other information that this misses. Like, here's four sets of data where all of those measurements that I mentioned before are identical. And someone who thought the only important thing was calculating these things might think, oh, all of these sets of data are the same. They're really not.

You need to be able to. And what a great question. A great question for a student is how many ways can you trick this machine to give you the same answer? And you know, this is beautiful because these are absurdities, right? That they, they clearly commit. Well, that one is outrageous. But having. Having a student trying to trick an algorithm into a response is a really nice way for them to get an insight into how things work. Right? Yeah, it's a really nice activity. That's something that should be highlighted in the resources for this. That's a good one, Noah.

It also shows, in fact, I believe that that phenomena has been used to emphasize the need for visualization in terms of people studying things. Because if you're just focused on, I'm going to teach the algorithm for doing this calculation by hand. Well, a student isn't going to write a data set of a thousand points by hand that no one would suggest to do this. Right? But if you're able to actually visualize this data set of a thousand points, not by hand, but using a technological tool, then you might start to spot those patterns and see the difference between the one that has some higher order term to it.

Yeah, exactly. It's a nice one. Well, that takes us two hours.

Yeah. Thank you so much for going above and beyond with us.

That was good fun. I really enjoyed that. And it's. I'm keen to have another in person meeting at some point. I'm going to be in Vancouver in March. Excellent.

Very much looking forward to more interaction. However we get it on my end as well. Well, thank you so much for being with us today. And we will be uploading most if not all of the documents we've been discussing here today, so just double checking to make sure there aren't any more questions coming in live. Doesn't look like it. So thank you again.

No worries. And what I'll do is I'll make a list of docs. Probably take me a couple of days to get them sorted.

Sure.

Excellent.

All right. I really enjoy.