QuestionObject Updates: Math Assessment, LLMs and More

I'm Bob Stanheinrich, I'm a developer here at Wolfram. And this talk is about the question framework and question Object. It's going to be about. I'm going to recap what is in the framework, what are the main features, then I'm going to show what's new and what's coming. So it's sort of a past, present, future sort of thing. And I've been really inspired by the education talks so far at the conference and the LLM talks and the education and LLM talks. And this is another one. So there's a lot of people that are sharing good ideas and I really think it's the start of something exciting. So let's see. So the question framework is kind of the core Wolfram language, you know, framework for quizzes and assessments and, you know, question interfaces for asking questions and assessing answers and doing automated assessment. So a lot of the other talks that you might have seen hopefully are building on top of what we've been putting in or will build on top of it. So to recap, sort of what's in here. So these are all system, except for the last one. These are all Wolfram language system symbols that are standard built in things. So assessment function has been around the longest. And this is the like low level, purely computational tool for assessing an answer against an answer key and determining if that answer is right. So this isn't a full question. It doesn't contain like the actual text that you're asking. It just contains the answer key. So this is an assessment function where 42 is right and it contains a couple of wrong answers because those can be useful when building questions from an assessment function. And if you ask it about 42, it gives you an assessment result object, which is just what it sounds like. It's a dedicated object for the result of an assessment. You can look inside of that and there's a bunch of information. Answer, correct and score are probably the most important things, but there's other information like the timestamp. So this is basically just a metadata wrapper assessment result object. But every time you assess something, one of these is created. That was a really basic one. Here's an assessment function that is a little more interesting. The second argument is geoposition. It's going to treat all the answers as latitude longitude pairs and then compare them against the values of the answer key. That way, you see, I've also included a tolerance of 200 miles. When you give an answer to this assessment function, it's going to say, okay, what's its position. Is that position within 200 miles of New York? Is it within 200 miles of Baltimore? Is it within 200 miles of Boston? Sort of in that order. And it's going to award the points for whichever one matches. So New haven is within 200 miles in New York. So you are marked correct. And you get 10 points. Here we can look at the metadata. 10 points. If I'd asked it about Washington D.C. which is, you know, closer to Baltimore, it would have given me, you know, it's going to tell me that I'm wrong. And you're at minus two points. So you can sort of create your answer key strategically with whatever values you want and accompanying scores. All right, so that's assessment function. So up one level from that is question object. So question object represents the whole question. So this is sort of the main object of interest for the question framework. That's why it's in the title of my talk, because sort of everything goes through question object. So in this case we have who won the 2023 World Series. This ended last night and the Rangers won for the first time ever. So that's our right answer. So, you know, here we have the actual question that you're answering, asking. And then the second argument is an assessment function. All right, so that's sort of a very basic multiple choice question object. Now you can make things a little more interesting by instead of just putting this string in the first argument, you can use this dedicated symbol question interface. This allows much more interesting interfaces. You can customize it a lot more than just putting in some text here. I'm saying it's a short answer question as opposed to multiple choice. The interpreter is the location. Anything you put in there, it's going to be treated as a physical location again like a latitude, longitude. And then the prompt is basically the same thing as the text that I had in the previous one. But here instead of just text, I have this column where I'm controlling the style of the text and I'm adding an image and the result is this guy. So where I've sort of customized the interface and the assessment function here, this is the same assessment function we defined above. I named it Near City. So this is with New York and Baltimore and whatever other city I put in there and a tolerance, so I can give it the same New Haven and that's full of Yankees fans. But if I say something like, you know, Concord, Massachusetts, that's, that's full of Red Sox fans. So that's going to give me a wrong answer. And I can, if I, if I can. One like hidden feature that we kind of need to make more obvious is you can click on that guy and you can get the assessment result object out. The same assessment function can be used in a different question. So here I'm using that. Still the same assessment function, but I'm going to do a multiple choice question and I'm going to give custom labels. I'm asking where I can find great bagels and dump. This is not to talk badly about Baltimore or Boston's bagels. I'm sure they have great ones. But New York has a reputation to upkeep. So here's, you know, the same assessment function being used in a different interface and different question. So this is all very modular system where assessment functions can, you know, have use on their own without being tied to one specific question. And similarly, a question interface could be used with multiple assessment functions. All right, so then one level higher than question object is question generator. So this does the randomization. So in this, so here we have question generator and this is sort of a simple one. Here the, here are the random parameters. Then the second argument is a question object formatted as a function. Here it's a function with this hash A and hash B. This is very similar to the API function if you're used to that, except the values usually come from within, they're not coming from outside. So the values are coming from inside. I can keep generating new instances of this question and of course it works. I hope you'll trust me at this point. All right, so let's look at a more interesting example. This one requires a bunch of helper functions and I saw a great Pokemon example from Jeremy earlier today, so I wanted to follow that up. And we have this Pokemon evolution test which I need help with who to center it, evolve them to. I added this little cheat code in here just so I would know. Let's put an extra T, see if the interpreter works. Hey, that works. Then one of the newer features we have is better handling of explanations in the interface. They were supported within Assessment Function and Assessment Result Object, but they didn't really show up in the question object interface. Well, so that's gotten a little bit better. In version 14, if I give a wrong answer, it'll give me a different explanation. And here it's going to say incorrect. You can customize these explanations and each value in the answer key can have a custom explanation and they're all fed back to the user and they're all stored in the assessment result object. As well, Everything has a symbolic computable form, right? Because this is a Wolfram language. So even though we have a nice interface for it, underneath everything is computable. All right, and then I want to mention, but I'm not really going to demonstrate question deployment. So this is a topic on its own that I could do a half an hour on probably. And this is the main mechanism we have for taking question content and putting it in the cloud so that the answer key is secured. Right? So right now a clever student could go into this and just basically see the answer key by looking at the symbolic expression. And sometimes you don't really care, they're only cheating themselves, but sometimes you really care about security. The best solution for that is to deploy the answer key to our cloud. I talked about that much more in depth at this conference last year. There's a link to that part of that talk on YouTube right here. This notebook can be downloaded, so feel free to look into that. But question deploys a function in the function repository that will help with that. Oh, I hit the wrong buttons. Hold on. Not and not. Last slide. Next slide. All right, sorry about this. All right, so that was sort of the recap of existing features. So now we're going to move on to new features and the main one to highlight here is dedicated math assessment. So math assessment is really complicated. So here's two questions. What's the derivative of 3x cubed plus 2x squared? And expand the polynomial x times 4 plus 9x. And you know, they both. If you. If we do. If we write it out. Let's write it out. 4x + 9x squared. This is the correct answer to both of these problems, right? So they have the same answer. But if this 4x are 4x. Sorry, x. And now this is in the way. Times 4 plus 9x. There we go. Is a correct answer to the calculus problem, but it's not a correct answer to the polynomial problem. Right? So you can't just compare expressions and see if they're mathematically equivalent or if they're represented exactly the same in Wolfram language syntax. Those are kind of like two naive approaches that we certainly have support for, but don't really work for real world math problem scenarios. What we've introduced is the code that's underlying these examples and that's these three dedicated comparison methods. Here I have assessment function calculus Result is what it says. This will allow most things to evaluate and it will compare the results basically just if they're mathematically equal. So you know it. This As I showed on the previous slide, this will work. You can have, you can do plus one, you know, plus one, minus one, you know it. Yeah, it doesn't matter, right? In the calculus question you just want to make sure that the result is correct and the format of the result typically isn't what you care about. But you can't have things, you can't like have the answer as as or the question as the answer, right? So if I, if I put this in here, right, it will not allow that to evaluate because this most is very important. Anything that it thinks is calculus is like this, the stuff of calculus, which is what is being asked about that will not be allowed to evaluate and it'll result in a wrong answer. Right? So this is what is happening in the calculus result comparison method. The polynomial result comparison method is much more locked down than the calculus result. It allows very little evaluation because typically this, the format of the result is something that the teacher cares about, right? So you know, I can switch the order of the terms. That's no big deal, right? I don't think a teacher is going to care about that. I mean, sometimes they will, we think most of the time they won't. So by default you can switch the order of the terms. We also allow, you know, like you could have if this, if this was square root of two, it could also, it could be represented as square root of two or two to the one half. Right? So those sort of basic common equivalencies are supported, but sort of mat actual like evaluation semantics are mostly not allowed within polynomial result. And then even more locked down is the last of these arithmetic result. So you know this. Oh great, I should have left that in there. All right, now I have to solve this problem. Two apples minus one. Is that an orange or a peach? Okay. Oh no, I got it wrong. Plus plus a peach. All right, all right. So this sort of, you know, this, this arithmetic result again you can reorder terms in like plus and times and some simple things like that. But it's fairly locked down. I forgot to say in polynomial you can also, by default you can also pull out a constant numeric factor. So if you, I guess if I had, I can probably do this. Nine, four. Right. So that's still interpreted as correct because the actual factoring of the polynomial is the same. Right. So we have these sort of easy to use automatic built in methods, comparison methods for these mathings and we're going to add several more in coming releases. So this is just the start of the sort of automated Built in. And these are also built on top of a much more general open math expression comparison that is a little too complicated to document at this point, but we might document that at some point as well if you're looking to customize it. All right, so that's sort of the main new feature since last year, since dedicated math assessments. And then so now we're moving on to future and the coming soon. So LLMs, you can't have a talk at this conference without talking about LLMs. So there's an undocumented comparison method that's in version 14. Feel free to use it, but don't fall in love because it could change, called LLM Question Assess, which just asks the LLM to determine whether the answer here is. Is correct for the question here. And if I say, what is an elephant? A big gray monster with a funny nose, I have no idea if it'll mark this correct or not. That's the beauty of LLMs. Oh, gosh. Maybe it's timing out. Let me make sure. Let me make sure I'm using GPT 3.5 because that. That'll go faster. All right. A large pachyderm. All right, I'll leave the typo and we'll see. All right. It likes that one. Oh, it's telling me that it's. I misspelled, but. Oh, it's incorrect because it's misspelled. Ah, so picky. So picky. LLM. Come on. All right. But you can see that the explanation generation is actually really great. You know, the, the assessment, as far as correctness goes, is usually pretty good, but the explanation is. Is really fantastic. So we'll probably. So there's a lot of ways that LLMs are going to work their way into the system. We'll have something where it's just creating an explanation. We will tell the LLM whether it's right or wrong, but it will automatically generate. Tell. Tell the student why. And it'll probably do pretty well at that. There can be tools for generating questions and all that sort of stuff. We want to get them right before we build them into the core Wolfram language system and document them. Then this is a very. Coming less soon. This is hot off. The prototype build feature is a new question notebook. So we've had these questioning question authoring notebooks for a couple versions, but they've really not been a success because they didn't make writing good questions, like the interesting kinds of questions that people want, very easy. They were fine for making trivial questions, but to make something interesting you had to write code, and so it didn't really work out. So we've overhauled them all. I'm going to create. This is like a quiz. I can put any content in here I want, and I can say simple math questions. Try your best. You can put any content you want in here, and it'll end up in the quiz for the student. But the workflow here, that's completely changed. And, you know, all of this. These icons and stuff are placeholder development stuff. So you guys get a special look into my placeholder icons. So this pops open a new notebook where I can. I said, this is a short answer question. So I can say something like, I'll use the same one as before. What is the derivative? Derivative of X cubed. I'll just keep it short since I'm already at 22 minutes. All right? And I'll say select for formatting because it'll look better. I'll say this math and further, it's calculus. And I'll give it the right answer of 3x squared. And then I can test it in this notebook. Here it is. And I can say 3x squared and it works. And I say finished, and it puts it back into my original notebook. Let's add one more because I want to do one that's a little more interesting. That creates a generator here. I'll say edit in question notebook. Again, this is multiple choice. And I'll say what is A plus B? And. And I'll highlight the A plus B. And I'll say select for formatting. These button names are certainly gonna change, so don't write it down. And I'll say A can be between 1 and 10. And I'll add a second random variable. And I'll say B can be between 11 and 20. And then I'll say that the answer is a number and A plus B is the correct answer. I'll add a couple more about A plus B plus 1, maybe A minus B and B minus A. All right, now I want to. And I can add some metadata. I'll say it's difficulty one and let's test this one. And that adds it down here. I know that I can do it. All right, so that worked. And I say finished and it inserts it back into this notebook. And now from this point on, the workflow is the same as in the existing question notebook for the few of you that might have used this. Where I can open a configuration menu, I hope. Where did that open? Well, this is the hot off, hot off. The prototype branch. Part of the talk. Okay, well, you can configure how this will get generated into the final notebook that the student sees and whether it ends up deployed on the cloud or whether it is local, in which case a clever student could dig into it and find the answer key for cheating. And then you'd hit generate and it would do all of the processing. And in the case where it's cloud deployed, it does a lot of great stuff where like the generator gets deployed to a specific endpoint and then each time a student requests it, the generator runs and it generates a question with the generated question built in to the notebook it's got serves gets served to the student as a new cloud notebook where they can answer the questions and it all. All works out. But fortunately, I think I've broken these buttons in the last day or two, so. But that's good because we're almost out of time anyway, so I will stop talking here. Let's see. Oh, yeah, if you download the slide, there's a few links. I should say thanks to the rest of the team, Christophe Paklaza in particular, who's done a lot of the work that I showed here today. So thanks to him for the work. All right, let's see. One question just came in from Phil, who says we are at. Where are we at? On customized feedback on particular incorrect answers. Gathering and analyzing data on student responses. Right. So for a. Yeah, I can. Let me just open a new notebook. So if you have assessment function of. Let's just say correct is the correct answer and you say score goes to one and then you say explanation goes to. This was right. Right. And then I want to take this and I'm going to make a few wrong answers. So I'll call this wrong one, wrong two, and wrong three. This was wrong for reason one. All right, so you can do this sort of thing and I can wrong two and. Oh, what? Oh, sorry. The scores were all one, so it's treating them all as correct. Let's make negative three. Why not that? So now it's going to say it's correct. And if I look at inside. Inside of that, here's the explanation. I can say question, object and hello. And put the assessment function in here. When I choose these, I should get the explanation. This is specific. One thing we need to do that's not in yet is we should be able to do like blank goes to. Or, you know, like wrong goes to this and then like, you know, f of wrong. Hash wrong. Probably right. I don't know. No, probably like Delayed rule F of wrong. Something like. Or maybe this is the delayed rule. This probably makes more sense, right? We, this is something we need to do but we have not done yet. Where like you can create an explainer function that depend. That works on the student's answer that isn't just choosing from one of some predefined things that's not in yet. It's on the to do list. If it's important to you, let us know and we can prioritize it more highly then. Gathering analysis and analyzing data on student responses with question deploy which I hand waved and moved past to save time earlier. If you deploy a quiz or an assessment function or question object, it records all the answers and you can retrieve them with a. Let me go to the function repository question. Let's just search for question. So we have get question assessments, the first one. So this, if you've deployed it, then this retrieves the data and you can see that it, it can be retrieved by student and it's assessment result objects or multiple assessment result objects. And so you know, you, you can retrieve all of the results including not just the most recent answer, but if they answered it five times, they're all tracked and it's all in computable form. So sort of post processing on this to create statistics or visualizations or whatever. We're not building that in, but you're in the Wolfram language. So you have the most powerful computational tools and statistical tools and visualization tools in the world. So you know, follow your heart, you know, get it done. You know, at that point, hopefully the person doing it, there's somebody in the in the loop doing it that has coding and math skills or you know, create your own resource on function and put in the function repository that does the sort of things that you think teachers want because that would be great too. I'm certainly not the authority on what, on what people want from their data analysis on this world. All right, Amy says thanks, I appreciate it. I should have left some, I should have left all my mistakes in my presentation notebook like you taught me to a few years ago. But I just don't have that level of confidence. All right, I'd like the students answers to be accessible for the purpose of generating feedback. Oh yes, that's what I was just talking about. Yes. Okay, so I asked you to provide that feedback if you wanted it and you just did. That was quick. That was quick feedback. Thank you, Phil. All right, and that is all of our questions. We are one minute over. So go on, go on. My friends and learn more at other talks and have a great day. Thanks for coming.