The Question Framework: From Assessments to Applications

I'm Bob Sandheinrich, I'm a developer here at Wolfram, and I'm going to be talking about how to use the Question and Assessment framework to create educational technology. I'm Bob Sanheinrich, I'm a development manager here at Wolfram, and I'm talking about Question Framework. This is Question Framework is, like it says, collection of tools for making questions with automated assessment. It's, you know, educational technology, courseware, course tools, whatever you want to call it, whatever buzzwords you want to use, it's all in that category. So why, why did Wolfram decide that we should get into this game? And, you know, automated assessment is a hard problem and it's a computational problem, and we make computational software and we think it's the best. So we think we're well suited to make the software that does automated assessment. But we need to build a framework around that to help people use it because a lot of the people who should be using this aren't going to learn the depths of programming. So we need to build this framework of questions and assessments around the computable engine to make, you know, that engine available. All right, so kind of, if you want, if you want to get a sneak peek. Actually, I'll just go ahead and we'll get a sneak peek at like, what this talk is building to. So, you know, everything here is, you know, a work in progress. The beauty of it will improve, I'm sure, over time. But the idea is that, you know, you can. Type in answers and it'll tell you if you're, if you're right or wrong as you, as you submit them here. If I can click this thing. There we go. All right, so this is sort of the building blocks to make one of these, you know, distributed quizzes with built in assessment. All right, so what are the keyboard. All right, what are the pieces of this thing? So this is kind of the agenda that we're going to go through in the talk. The sort of the building blocks that let us create those and what we have so far and where we're going in the future. So, you know, the first three bullets here are all Wolfram language symbols, you know, functions that are in 13.2, and some of them earlier. And then we're going to talk about deployment of this stuff, and then we're going to talk about question notebooks. All right, so assessment function is the lowest level thing. So we're kind of, kind of build up throughout this talk. We're going to start at the lowest level thing. This is the Computational nugget that determines whether students answer is right or wrong. Right. So that inside the assessment function there's an answer key. And then you apply the assessment function to the student's answer and it tells you whether the student was right or wrong. So it can be super simple. The answer key can just be a value and if it matches it's right. If it doesn't match, it's wrong. So it can be very basic and then it can get more complicated. So here's one where here we have an assessment function and there's an answer key and there's also the second argument that says how what we call the comparison method. So this is an interval method which says if the student's answer is inside one of the intervals in the answer key, it's that is considered to match their answer. So if they. Where'd my mouse go? How did I not see it? All right, all right, so if they give a, give an answer inside between, you know, within 0 to 500, it's going to be incorrect and they're going to get negative five points. And then we have this explanation that will be fed back to the student to explain why they were wrong. And you can see we filled these all out. You know, there's overlaps in here which that's perfectly fine. And you know, intervals can have things like infinity because you know, Wolfram language is good at such things. All right, so we can define this assessment function. We can get an assessment by applying it to an answer. And then what we get back is this assessment result object which is just contains an association. We can look at everything in that association by applying it to all and we can see what's in there. There's a score and max and min possible score and whether it was right or not and what the answer was and the explanation and all that stuff. So this is like I said, the computational nugget part of it. You can also, instead of using one of these built in comparison methods like interval or just having a simple thing like the answer is 100 doesn't match, you can fully customize the assessment. So here I have this function which basically is taking the student's answer and comparing it to all so that the answer queue is full of cities, all the values in the answer key are cities. So this will say which city in the answer key is closest to the student, the city that the student says. And if the one that's closest to St. Louis, Missouri, they'll be counted correct and they'll get five points. So this is Completely customized and we can apply that. And Pittsburgh is close, closer to St. Louis than it is to Chicago, Indianapolis or Des Moines. We also have partial credit assessments. So we have this element wise assessment goes to true property. And then you can, when you do partial credit, you can also customize how both the scores of each answer are combined. That's this score combiner function and how the correctnesses are combined. So if they give 10 answers and seven of them are correct and three of them are incorrect, are they correct? Overall, it's not clear what the right thing to do is. So you can decide that yourself if you want. So let's define that. So this one for each answer, it's looking if it's prime. So pretty simple there. And I had two right and three wrong. And I can look in this. I think I have it down here. The zoom thing's getting in my way. Here we go. I can look at the score. I can look at, in this case, if I look at everything, it has this element information. So it's kind of broken down piece by piece in the assessment result object. So we kind of try to preserve every piece of information that might be used later on. All right, so now we. So that was assessment function. Now we're going one level higher in question objects. So question objects contain assessment functions, but they also contain interface information. So it, you know, tells, it creates an actual interface for the student to answer questions. So here's like the simplest one you can imagine. Now this showing is super wide. This is like a presentation notebook thing. They're not like they, they look better in normal notebooks. So I can, you know, get that one right because I'm really good at math. We're a math company. And you know, I can take, I can create a little, make it a little more complicated by using question interface as the first argument instead of just the text, like what we call the prompt. So the first one, all I put in there was a string and that was used as the prompt. And. But here I'm doing, I'm creating a question interface object which allows me to specify the interface more. And I'm saying it's a numeric range which by default will show up as a slider. And I can answer this. And you'll see the explanation gets fed back. And Albert's retired. He retired last month, so I don't think he's going to get to 742. But we'll update that explanation later. I'm still not ready to say goodbye yet. All right. And then again you can customize it even further. So here's. Here's two that are using the assessment functions that we defined on the previous slide for this one for a city with a population near St. Louis. So here, short answer, which by default shows up as an input field, but I have this interpreter in here, so whatever I type in here, you know, I think let's try champagne. That's where we are. That'll probably be wrong. Yeah, I think in honor of my colleague here in the. If I can type in the seats here, Anchorage. I think that one's close. Yep. All right. All right, so, and then here's one that, you know, we had that partial credit assessment function we defined on the last slide. So you can do a list, what we call list multiple. And that creates a little interface like this, where every time I put in something, I can keep. I don't know, let's have. Let's have some wrong answers here. All right. Remember, if I. If they were mostly right, that assessment function was calling the whole thing right? So I got this. What's kind of neat is if you. This little checkbox, you can click it and it'll put the assessment result object right on the clipboard. So it's kind of a neat little utility so that you can, you can get that out. It's not just kind of hidden inside a blob. All right, so that is question object. One level further up is Question Selector and Question Generator. So let's start with Question Selector. This is basically a tool that serves questions from a question bank. So you have like a predefined set of questions, and you want to be able to serve one to each student. Right. So you kind of make the test, make a quiz so that not every student gets the same one. It makes it a little harder to cheat. So here we have Question bank with the four questions we defined on the last slide. I can put those into a question selector. It just shows up as this object tells you how many are in it, but it just sits there until you apply it like a function to zero arguments. And then it's going to serve up one of the four questions in there. Right? So. And this is. This is a building block. Right? So probably not just like using this in a notebook directly, like I'm showing here, it's going to be. It's all going to come together. You stick with me here. All right. Sort of like question selectors, question generator, but instead of like this, taking a pool of predefined questions, this takes like a parameter space and A function that generates a question on that parameter space. So this is a pretty simple case. A and B are both integers. And we're asking them what's. What's a plus B? And this, the assessment function is just like this test function equal to is an operator. It has an operator form. So this works by, you know, operating on the student's answer. So you can see I have a question generator. It just sits there again until I apply it like a function, just like question selector. And I can keep doing this until I can get one that I know the answer to. And then. Oh, no, can I not read 500 off? Oh, I'm way off. Oh, no. Yeah, I just can't read. I thought it was a 33. Okay. Yeah, Yeah, yeah. It's kind of small. I'm going to blame the presentation notebook. All right, all right. And then. So that's a simple one. We can make a more complicated generator. So this is just like that name a city with population close to St. Louis example. Except this time I'm not just going to focus on St. Louis. It's going to pick a random city from anywhere in our knowledge base, which is like all the cities. So now every time I apply this, I get this. Anybody know about Bogdanita? I don't know. Let's go with Urbana. Urbana is probably the same size as most of these. I'm always going to get this one. So this one wrong values that I have to get closer than. There's 10 of them. So probably. Probably never going to answer it. Right. All right, so there. So that's. That's question generators. So those are sort of the building blocks for defining question content in the Wolfram language now. So why don't we stop and see if there's any questions about that, because we're going to kind of move on to the next. So, Yeah, I mean, the way I implemented it, they would probably be right. Yeah, yeah, yeah. So, but, yeah, yeah, yeah, but you can. But, you know, it's code, so you can, you can do that. Yeah, Yep. That's a good point. All right, so now we're going to talk about deployment. So so far, even that. So one way that a student could get around this. Yeah, go ahead. Oh, that's. That's great. I will. We will circle back to that at the end. Okay, that's. Because that's a great question. Okay, so. Right. But, you know, even if you. Outside of like the, you know, crafting an answer, that is basically a tautology and that there's an easier Much easier way to cheat. Right? I can just, I can just look at the input form, right? There's the. Here's the answer. So, right, so that, you know, is not great. You know, it's completely unsecured. Everything is just in the notebook to be explored. The solution to this is the cloud. The cloud comes in here and it provides solutions to two problems. Not only this security, but also distribution. It two birds with one stone thing. We have this function in the function repository called Question Deploy that is like the, it's the equivalent of cloud Deploy and it is cloud deploy under the hood, but it does some other things that we'll see and it works on like any of the stuff we've seen so far can be put into Question Deploy. So if I take just an assessment function, I can do Question deploy and I get back this deployed assessment function, I can use it just like I did in the notebook. But you can see here, the answer key is not in there anymore. All that's in there is a cloud object that, you know, students can send requests to, but they can't inspect it, they can't import the code. Question object. When I deploy a question object, I get back in association with two things. I get back a cloud object, which is important for like administration. If I want to change the permissions, if I want to move it, if I want to delete it, I want to, I'm going to want to have that cloud object and I get back the question object that has been, you know, cloud deployed. So if I look at inside that thing, there is no answer key. All that's in here is again an assessment function with the cloud object. And of course I can, you know, let's get it wrong. I can use it just like I would use one that isn't deployed, right? So it all just works. And then Question selector, similarly, you get back in association with two things, a cloud object and a question selector. But this kind of has two levels of deployment. So not only do we not want the student to be able to see the answer key, we also don't want them to be able to choose their own question from the question bank, right? So if we look at the, you know, all this in the selector is a cloud object, so they can't like choose their question. And then when we take this guy here, we take the selector, let's just do it this way, makes it more clear and we apply it like a function. Just like when it was running locally, it'll go to the cloud, the cloud will pick one from the bank and return it. And when we look at that one again, the assessment function just has a cloud object. So there's no answer key in there. So it's kind of two levels of deployment in this case. Right? So that, that's customizable. You can, you can choose what all to include in here. Right? So obviously it has things like answer correct and score. It does show the given answer. I think that probably makes sense. You can optionally have it. So in the case with like interval, the interval, except, you know, the student's answer didn't exactly match the values in the answer key, you can optionally choose to have it show which answer in the answer key it matched. So sometimes a teacher might want to do that, sometimes they might not want to do that. So that, that's an option. I think by default it doesn't show it. It's looking like here. And I don't remember the option name off the top of my head, but it's in there. Okay, so question generator is just the same. We deploy this city quiz and we get back the cloud object and question generator. You know, if we look at the input form of the question generator, you know, you can't choose your own parameters. All the generation is done in cloud. And this, you know, has a, has a separate level of complexity because once the compared to the selector, because once the question is generated, like what's happening right here? We need to know, we need to track that, we need to store it on the server side, the generated question and who it was generated for. So if you look, if we look at the input from the generated one, you'll see not only do we have the cloud object, but we also have this, this UUID that can be used to make sure that when you answer the question, it's graded against the right one and all that gets tracked. Right? So, and let me, let me answer this so that the next thing I'll get around. Everything is the same as Urbana. Champagne then. I mean, come on, I want to do it twice because it'll help me show the next step. Did that work? Okay, so that's hitting the server each time because it's a deployed assessment. That's right. So then, so then how do we. So then, you know, we've students have been answering this and getting them wrong. So how do we get the scores back? You know, how does the teacher know what they did? So we have another function, the function repository called Get Question Assessments. And I give it the cloud object that came back from Question deploy and I say, I Think this is which students do I want and how many of their assessments do I want to get back? It can be just the most recent or it can be all here. You can see Bob has answered this twice and gotten it wrong both times. I guess it doesn't say wrong. No, it does correct 0 to 1. That's question deployment, which is an important part of the entire infrastructure. And now this. The next part is the question notebook. So all of this stuff ties in together in question notebooks. And you can create a question notebook. There's a file new something question notebook. Or you can just say create notebook of question notebook. And this is where like a teacher can define a quiz. And this is very much a work in progress right now. It's very code centric, but we're going to work diligently to make it very much low to no code. So, you know, you can say. Which MLB players. It's always baseball with me are in the. Is. Is in the hall of Fame. This is just only one correct answer. And then of course you got to put Pete Rose in. Right. Right. Famously not in the hall of Fame. Right. Everybody knows that. Who else can we do? How about Bob Gibson? Because everybody likes Bob. He's in there. And Roger Clemens. All right. Yeah, yeah. So, yeah, so there's a multiple choice. And you, you know, so we have this drop down. You can put in randomized question is what we call question selector in the dropdown. And this one works a little bit differently. Instead of typing in right here, this pops up like this sub. This sub notebook or palette or whatever. And it's kind of like a mini version of the authoring notebook. I should. And you can fill this up with questions. Oh, let's see. Frank Thomas, one, two MVPs. And that's. That's true. Of course. We have things. Yeah. Like numeric range. Like we saw. I did one of these before. You know, let's say how many World Series have the Mariners one. Which is of course equal to zero. So I don't know, negative one to eight. All right. So yeah, so you can build a question. This is a bug right here. This shouldn't say question generator. It's reported. We're going to take care of that. That should say question selector. You finish this and then it pops it back into the original authoring notebook and you have this also. I should say that in these notebooks you can put any arbitrary stuff between these special cells. You can say, let's do a section questions about. Players. I don't like. Any content you put in here outside of these special cells will just show up in the student's notebook normally. Let's see, let's look at this configuration menu next or settings menu. So this has a bunch of things that you can customize like colors and labels that's sort of self explanatory and don't really need to demo it. But then this stuff down here is the exciting part. So these two choices you should think about for sure. Do you want to assess each question individually? So they fill in one, they push a button, they fill in one or submit them all at once? So they fill out all the answers and then push one button and it assesses them all and then you can assess, you create and generate a notebook where the assessment is local and again that won't be secure. Right, but maybe it's for third graders and they're not going to figure out or maybe it's like a survey where there's no cheating to be done. So you might not care at all. But a lot of times you want to say to do it in cloud Baseball Quiz 7 Probably I probably have six more baseball quizzes, don't you think? So then I say okay and then I can click Generate and oh no. Oh no. This is what happens when you're on the developers computer. Ok, well that's not great. All right, we'll just move on and look at, you know, here's a quiz that I filled out earlier and that's already deployed and if you followed the QR code or the link at the beginning that was the quiz that generated it. So you can, you know, it's here. So here's a, this is what we saw at the beginning. So a completed quiz. So this is, that's the kind of the whole workflow we have in 13.2 so that you know that let's. I'll spend some more time talking about like the roadmap for the future but we can stop for questions about what we have right now. Now if anybody has any. Joel. So generally it seems like every question, right? So you can, yeah, that was one of the options in the setting. You can change it, you can. That's one of those two options that I called out. So if you do them all at once and then if this actually worked it would each, each question object in the resulting one wouldn't have its own submit button. There's just one at the top. Okay, this time it worked. All right, let's, let's see how that, how that did. We'll just say it Was like a network error. Right. So, okay, so this delay we're going to improve, but what's happening here is, is it's generating a custom quiz for me. Right? We had that. Oh yeah, see, okay, this is, Yeah, I know exactly what this is. This is because I was working on this yesterday and forgot to go back to the master branch. Ok, well there's just one submit button at the top. Okay, okay. Right. So let's see, I was, I was in the middle of a train of thought. What was I talking about? Baseball. Right. So, okay, so yeah, it's worth mentioning this index notebook. So when you do the generation, you get this index notebook which, you know, this is a copy that's on my local machine, but there's also a copy of this index notebook. In cloud. When you deploy a quiz, it's really deploying a directory and that directory has a bunch of endpoints. It has one for generating a quiz. That's what I was talking about a second. So when you, when you hit open in cloud, you know this, it's going to show me the same broken notebook again, but it's going to be fast this time. Right. So you saw the first time it took like 10 seconds. We should be able to get that down to two or three. But what it's doing is generating custom notebook for me, saving it, linked it to my account. Assuming I'm cloud connected, when I follow the link again, it's just going to keep giving me the same ones. So students can't keep pinging the API until they get the questions they want when they're selectors and generators. If there's no selectors and generators, it just, it should be fast every time. It doesn't have to do that generation. Right. So you get the index notebook and this is kind of like the teacher's tools for managing the quiz. It has the code for retrieving students answers has like this locked version where the generators and selectors will appear still as generators and selectors. They won't generate this lock. Just we might change that term. It's not a great term. Yeah, so this is kind of all the tools that the teacher will use to manage, manage the quiz. All right, so future stuff. So there's, there's, you know, every part of this project has places to expand and we want to do all of them so that the authoring environment is going to be a big, a big push here soon so that we can really get to that low or no code tool like a GUI type interface for teachers to define questions Right. You know, especially common multiple choice questions. There's no reason to see any code at all. And there's a lot of questions like that. If you want to get more and more sophisticated, at some point someone's got to write some code. You know, that's kind of going to be always be the case if you want something really fancy. Someone has to write code now, not necessarily you. You know, we could envision a world where there's a, there's a repository of questions where, you know, you can distribute and share content and, you know, maybe then you just throw find a question that works and tweak it and you don't really have to think about code, just thinking about values within the code. Most people can do that. We want to add more interface types in general. We have the numeric range and the list and the choose multiple and the multiple choice grid. But there's a bunch more. And some of those are going to be more dynamic and gooey and clickable as opposed to just sort of typing and radio buttons. We want more comparison methods. So this gets back to the question you asked about putting, you know, just putting the question back into the field and letting it evaluate. So we already have like, you know, there's like an expression or what we call an item comparison method where it will evaluate, right? So it's if that's what you're using and it says what's 50 plus 50? And you just type in 50 plus 50, it's going to evaluate to 100 and you're going to be called correct. But we also have like a held expression comparison method and if you use that, then it'll be held and that comes in really handy. We already have like this code equivalence comparison method which uses the same tool that backs the Wolfram Challenges and the EIWL and some Wolfram you quizzes that kind of looks at code and performs transformation rules on code without evaluating any more than it needs to evaluate. And a lot of times can be pretty brilliant about determining if code is equivalent. So that's really good. If you're writing a question about the Wolfram language, right? If you're writing a quiz, quiz people on Wolfram language that code equivalence is a really powerful tool that's already in there and it holds expressions that way. We really want to do something very similar for math expressions because we think that that's a big use case that people want where, you know, we have transformation rules, again, we hold the expression, we form transformation rules, and we really want to make it easy for to have a good set of, like, name transformation rules where the teacher can say, oh, I want fractions simplified but polynomials maintained, or, you know, you know, you know, have numeric tolerance on all real numbers. But all these sorts of sorts of rules. We just, we want, we want to kind of, I imagine a world where the teacher has, you know, 20 checkboxes and they can just choose which ones, which transformations they want to apply and which ones they don't. That's kind of, kind of the long term goal for that sort of math equivalence. So, yeah, some more comparison methods, more interfaces, better tools for teachers to write questions without writing code. This idea of repository or a shared place to trade content. Yeah, those are sort of the roadmap. And that could be a long roadmap, but that's where we're going. That's it.