Live CEOing Ep 782: Design Review of AssessmentFunction Comparison Methods

Hi, everyone. Welcome to another Wolfram Language Design review. This is part of our cycle for version 14.1, and we are looking at assessment functions, part of our question framework. Bob.

Right, so. Right. So we have this question framework that has all this automatic assessment built in to say is an answer to a question, right or wrong. And there are all sorts of ways you can do that. So we have what we call comparison methods, which is defined, you know, set in the second argument to assessment function.

Yep.

And so what we're talking about here is new ones, improvements to existing ones, that sort of thing.

Yep.

And one thing to sort of. I agree, the argument. We don't have a listing guide page for these yet, so we started putting one together. It's not in CVS yet because I never know what to name guide pages, but we can look at this and I'll commit it when we have a name.

Okay.

But, you know, we tried to come up with some categories of these things which.

When you say match arbitrary expressions, you mean literal match an item first.

Yes. Item uses match queue. It's basically just match queue internally.

Well, I mean, is it. So can you put patterns in there or not? Yes, I see.

Yes.

Well, I think you have to describe that. I mean, is it really. Should it be called item?

That's what we came up with as sort of a generic. Well, the student's answer is not a pattern. Right. The student's answer is an item. Right. So it's the students.

Are we naming these argument student?

Yes.

Okay. Why don't we just call it expression? Because it is an arbitrary expression.

It could be expression.

Yeah, yeah, I think that's what we should call it.

Yes. I think we went back and forth on those names at one point and we went. And we.

I would say expression.

So we can add background ability for items. Yeah, yeah, keep working.

Right, right. But then we should say there, Compare by pattern matching expressions. I think you can pattern match. I think you can hyphenate pattern matching. In that case. Sure.

Let me save this.

Okay, so compare based on numeric difference. All right?

Right. So that one isn't pattern based.

It doesn't end the next one. String is also not pattern based.

Right, that's right. It's like edit.

Okay, so compare literal strings. Why are you saying text? Just say literal strings. I mean, this is a programmer type page.

Okay.

Based on color distance typo. Thank you.

And again on the next one.

But so how are you specifying the distance that is allowed?

There's a distance function option, but, you know, these mostly Feed through nearest and nearest has default distance metrics based on the, you know, I mean, do you.

Have to nail it? I mean, it's like, is the Battle of Hastings 1066 or 1067?

Right. So if the ANER key says 1066 and you say tolerance of 10 years, then.

No, but, but, but, but. Assume you don't specify a tolerance. What's the default?

Oh, zero. Yeah, you have to nail it. Machine.

What if, what if I said the date of the Battle of Hastings was January 13, 1066, what happens?

Right, so. Yeah, that's a good question.

I don't know.

Should we try it?

What's that?

I don't know. Let's try it. So like, if the original date object was just 1066.

Yep.

And then.

And type it in or whatever.

Yeah, I can, I can use a string. But so that's marking it. Correct.

Right.

It's. It's going through nearest.

Okay, so if nearest 67 is the second thing. 1067. 1 1.

Okay, so the default default tolerance is zero. Seconds. Seconds.

Yeah, but, but the question is, does it fit a granularity? And then the next question is, you know, I mean, to me, I don't know, I don't understand modern education, but to me, when somebody says, you know, the length of this object is 7.25864 centimeters, it's like that's, it's wrong. But I think in modern education it's considered right. Am I making sense?

Yes, yes. Yeah. I mean, dates have this special granularity issue that comes up all the time.

Well, I know, but so do numbers. I mean, so in other words, there's a numeric tolerance and I don't know, I mean, a default tolerance here. I don't know what the correct behavior is here. And I'm not sure we have anybody here who can opine.

Well, so one of the comparison methods we have later that we have here is what we're calling numerical approximation. And we. So a couple of these, you know, we did some work and proposed them for 14.0 and we decided that we weren't confident enough. We hadn't done quite enough homework on what people want and how people are going to use them. So we went and we did that homework and we spoke to experts within the company and we're bringing them back again now for 14.

Those experts here, they are not.

I don't think so.

No.

Avery, no.

Is that people like Alec in the uk, did you talk?

Well, for some things. For the algebra solution, yes. For numerical approximation. Michael Trot mostly.

Okay. Okay. Well, I, I, I'm not certain I can confirm these things without talking to the experts, so to speak. But let's keep going on what, what your agenda was.

Okay, Right, so, right. So besides this guide page, we can keep going through and editing this if you'd like, but I'd rather dive into the specific pages. I think these links are broken for me. So let me grab them here. Right. So algebra solution in 14 0. We propose calling this Algebra Result. And we decided that was not right. Um, because this is very much checking for a question of like, you know, solve for X in an algebra class and your student comes up with an answer. So in some algebra questions you very much care if like, you know, it's a, you know, proper rational or something. But for these you don't. Right.

You, you just care that, so 46 is the correct answer for this, Is that right?

Sure, but you could just use number for four sticks. But like, you know.

If you have four six A plus B or something, is that considered right or is that considered wrong?

Let me type it out so I know exactly what you're saying. So four six A plus B. Oh.

That'S not going to be useful because, because the 46 will auto simplify in this setup.

Right. So when in, within the world of a question object, typically everything is going to be held right.

So I'm asking, does this, is this a correct answer when the known answer is 2/3?

Ah, yes, yes, it would be in this, in Algebra Solution. Right. We plan to have another comparison method called Algebra Result, but it's not ready for 14.1. But in that case it might not be.

That's a totally obscure thing. Those are totally names.

Yeah, we can talk about names. Yeah, Algebra. Well, Algebra Result is consistent with the existing names we have.

Can we get somebody here who knows about kind of math education stuff like Bradley Janes or somebody? I don't know.

I mean he's one of the people we consulted with on this.

Okay, but fine, but somebody who knows about that because I don't know that stuff. You know, if I look at an elementary algebra book, it's going to have all kinds of names and words in it that I don't know. Right. So I, I can't, I'm sure there is a name for this, I'm sure some, you know, math education theory person came up with a name for this situation of it's right, but it's not in common form or something. I don't know what, I don't Know what it's called? I don't know. I mean, I'm just, I'm just a practical user of mathematical.

I mean, so, so let me. What. So let me just explain what came out of our consultations with Bradley, James and Alec and, you know, people with education backgrounds. So, you know, we thought we could have one algebra comparison method and we decided that that was hopeless and we really need two. And one of them is, you know, tentatively going to be named Algebra Result. Of course we can change that name. But it's the complicated one that's trying to handle the case that you're describing where the form matters in complicated ways. This one is the other one that we think is very commonly used and it's a simpler case and it's, you know, what comes out of solve for X. We're really. What we care about is that the student doesn't.

Why don't we call it Algebra value and algebra form? Algebra value, Algebra form, Algebra value. If it's, if it's equal, you know, if full simplify can, can establish that it's equal, we're happy. Algebra form requires that it be in the right form. Do you see what I'm saying?

I see what you're saying. I'm thinking Chris does that. Chris has done a lot of thinking on this as well. Does seem reasonable to you.

Form and value.

So we are calling solution would be value and what we are calling result would be form.

Yeah. If you see algebra value versus algebra form, you can explain that as. Let's write the description. Algebra value is compare by algebraic value and the next one is compare by algebraic form. See what I'm saying? That seems comprehensible. I don't, I don't know if it is.

Yeah, I like value for sure.

Yeah.

I mean, like I said, form isn't ready. So we can, we can defer a final decision on that for another version.

Yeah, okay, but I think that's the right term. I mean, if, if you are going to insist on the form like is, you know, look, algebra value. Okay, so compare by algebraic value without algebraic transformations. Right? Well, so does it. Okay. Is x squared -1 equal to x +1 times x -1 according to algebra value?

Yes.

It is equal.

It is.

Okay.

All that this cares about is that you're not like in your solution. You're not, you're putting in solve or things like that.

Okay, all right, all right, so, so, okay, compare by algebraic value per N. Allowing.

Allowing all sorts. Yeah, okay.

This is not informal, allowing mathematical transformations or something. Or algebraic transformations.

Well, simplification.

Well, it's not necessarily simplifications. Well, that's just actually compare by algebraic value.

It's allowing operations that we don't consider to be algebraic solving.

Yeah, but I think by algebraic value is fine. You have to document it in more detail just to get rid of the premises. It's just confusing. So algebra form x squared minus 1 and x plus 1 times x minus 1 would not be considered equal?

Probably not. But we'd probably have all. Definitely not method.

That's what we're calling it then. Definitely not. Right. All right, let's move on. Right.

All right, so. So the. This is the actual doc page for what was algebra solution will now be algebra value. Do you want me to magnify this or you can.

No, I can read it just fine. I can read it fine. Okay. Allow what's a solution? Lhs.

Right. So this is. This is the. What, the name that we've implemented right now, and it's not a great name. So we're definitely open to suggestion. So basically, you tell it that X is the independent variable you're interested in. And then you can then if they. Because a lot of times in an interface, a student wants to type, you know, X equals 1 plus 3A instead of just 1 plus 3A.

Oh, Christ.

Right. So you kind of need to know what the independent variable is or. Or the dependent variable, you know, you need to know that there needs to be a special symbol that you know is special because B plus 3A. B equals 3A is not correct. Right. You.

Well, then, I mean, isn't this a template? Isn't it basically, you know, if it's X equals. You should allow. Okay, if they could either type in 3 or they could type in X equals 3.

Right.

That's what you're basically saying. Yes, correct.

Yeah.

So why not give a list of held. You know, a list of functions? Why don't just give. The function is either hash, hold hold of hash ampersand, or hold of X equals hash ampersand or whatever else.

So you're saying. Let me make sure. So you're saying, like it would be in the assessment function. You're saying you would have here.

No, no, no, no, no, no, no. The assessment function is giving you the answer. Right. So it's 13 or something. Type. Type assessment function of 13.

13.

And now put. Now put. You know, you could potentially have. Okay, you could either have 13 or x 13, or you could have something where you say, you know, basically wrappers with a w Arrow no, whatever. And then it's either hold of hash ampersand or it's hold of X equals hash ampersand, whatever else comma, you know, whatever it is. I'm not sure this is right. I mean, I, I don't understand why, I don't understand how people are going to type in X equals whatever. Who's going to look for that? I mean, I don't understand what that is or X or whatever.

So typically like in the question object, you know, you would. Right. So, so here's an example where, you know, you could, the student has to type into an input field. And if, if we have this, if we put this transformation in here, then we have the flexibility to allow any of these. Right. And, and another thing to keep in mind is in the world where people are defining these questions in the question notebook, which is the topic of another design review for another day, no one will ever have to write this. This will be scraped from a definition notebook and created automatically, right?

Yeah.

Okay, so there, there will be a lay. There will be a convenience layer where you don't have to know about method options and names of transformations and exactly how to parameterize them. All that matters is that the ability to do it works.

Okay, so I mean, basically what you're trying to say here is you want the possibility of having a. Yeah, right.

This transformation.

I mean, I think what you should call this a solution name or something or. No, no, it's not solution name. It's. It's.

Like solved variable.

I, I don't know what the variable is because I, I think it could be anything there on the left hand side and it could be, could be some composite thing that says. Let's. I mean, I'm thinking sort of result name or something like that.

Yeah, I'll type it over here.

Template or something.

Well, if it's going to be done with a hash thing, but that's a big nuisance. And if what you really want is just to say.

Form, result form.

No, but it's not what it is because the result name, the thing will just be X for example, here. Right. So you're going to say.

Yeah, 98% of the time it's just going to be a single symbol.

Optional result name. Yeah, I'm thinking, I mean, I think people would understand what that is.

What do we call the, like the second argument and solve in the documentation?

Variable.

Just variable.

Yeah, but look, I don't know whether this is always going to be used for, for solve. I mean, it's just, I Think optional result name is nice and neutral and makes does says the right thing.

Okay, we can rename it that. Do you think it's something we should document now or should we leave it for a bit?

I would document it. Let it marinate it. All right, I would document it. I think it's useful because we also.

Haven'T documented this method goes to transformations, which is something that's used in all of these mathematical comparison methods. And, you know, really the way all these are implemented is they're really all just a list of transformations and a couple other options.

I don't think we need to document it. I think that that's in the weeds of implementation.

But we can't document optional result name without documenting transformations because that's where it lives. I guess we could support it at a higher level.

No, why don't we just support it at a higher level? Like, just support it at the top level of algebra value.

So to be assessment function, key algebra value. And then right here you think optional result name.

I guess so, as a. That was an association of.

Yeah, I mean, so we already have. We also have this. We have this, you know, standard form that these things resolve to where this would be comparison method. And if you, you know, you're already sort of getting into the weeds here.

So I would. I would support it that in that way.

Great, we can do that. I like it.

Okay, now question is, should it be algebra value or algebraic value? Or if we want to go Victorian, algebraic cull value. But let's not do that. No. Okay. Can I look at the other names, please? In the. In the guide page? Yeah.

Right. So these are the other math ones. Calculus result, polynomial result, arithmetic result.

I kind of think it should be algebraic value, an algebraic form.

I like algebraic form for sure. And then you probably want to do them both.

Algebraic value is fine. I mean, it means the right thing, I think. Okay, moving on.

Okay. All right. So then the next new method we're proposing is numerical approximation, which is the name. I don't love this name, but we haven't come up with anything better. Basically, it means that you're applying a tolerance to each numeric value in an expression.

How does it relate to number? That was the previous thing you had.

So number the entire value in the answer, and in the answer key is a single scalar number. This is a mathematical expression that contains numeric values.

Right, I see.

And in this simplest case, there's just an Absolute tolerance of 0.01 applied to each numeric value.

Right.

And this is you know, there's all sorts of reasons why this is strange, but it's also very useful and very much something that is common.

And I don't understand why that isn't part of algebraic value. That's a. You know, if you want an algebraic value with, with numerical tolerances, why not? Just because it is an algebraic value comparison. What happens if we have, you know, if you have tolerance things? I don't quite understand how that works. If you have both a, you know, x minus 1.001 times x plus 1x, you know, plus 0.999 or something. Where does that fit this?

I'm sorry, can you say that again?

I didn't parcel + 1.001. No, in parentheses. In parentheses. X minus 09999. Okay, right. And you're comparing that to x squared minus one.

Right. You know, typically these will be held. So it won't expand or anything.

Yeah, I understand.

Yeah. Chris. What. How would our. How would that work?

So if you have to compare to the squared, so there is no other way that. To expand this because of the way you cannot. So it goes to calculus result. I guess.

I don't understand. I don't understand. I'm trying to do this, trying to figure out, is this equal to x squared -1 up to numerical tolerance. Right. What will it do? I'm presumably.

I don't. I don't think we have a solution that will make that work.

So we are not transforming. We are not rounding the values, so there is no chance to be equal.

I don't understand this. If you just expand that thing. If you have a transformation which does. Yeah, says expand or simplify. Try simplify of that expression. Just try simplify that expression.

Okay.

Try expand.

Yeah, expand.

Well, Okay, so my question is. Ah, so that's a. That's a screwy thing because there's a term there shouldn't be that.

Right. Well, right, exactly.

Well, okay. So. I think this is an algebraic form story with some kind of numerical tolerance. And it seems like that's what it is. Right, right.

Algebraic form with numerical tolerance. Yes, yes. And this is sort of the reason why this is complicated and why we don't have anything ready to review yet.

Okay, well, that's where this should go, in my opinion. I don't think there's any way to put this in. In a way that everybody will understand without having the algebraic form concept. Yeah, I mean, if it's a single number, that's fine. What happens if it's a list of numbers? I Assume that the single number thing handles, you know, If I have 1.07 and you know.

Oh, yeah, yeah. I mean, the number comparison method handles a list of numbers. That's not okay, but. Yeah, but just there's these expressions containing numbers. Right now we don't have a way to have an. To have tolerance on these at all. Okay, so that's. So you think, you think punt on this until algebraic form is ready?

I think we have to. I don't see how we can put it in and have anybody understand it.

Okay. Okay. All right, so then, existing comparison methods, new features. This is, you know, Chris deserves credit for all of this, but I'll do talking. So in calculus result. We've added nice conveniences for constants.

Good.

Which we didn't really have before.

Yep.

So in the. We're basically requiring the teacher to represent the constant with big C, but the student can represent the constant in any way that's reasonable.

What about 13? Is that a constant?

No, I don't think so. Is it, Chris?

Nope. Okay.

No. Right.

I mean, I'm a little confused by that because some indefinite integration methods will give different constants than others. I mean, differing by a constant value.

So, yeah, this is symbolic. For symbolic comparison. So.

Yeah, again, I'm not the expert here, but I think, you know, there are things where there's something with logs or something and something with a. I don't know what, inverse tangents or something like that. And one of them may have a different sort of base, natural base number than the other one.

So if you integrate with. From. With numerical values.

Yeah.

You have a constant which is numeric. But. But we don't support.

No, no, no, no, no, no, no. Two different algebraic forms, both correct. Can differ by a constant. Yep. Can differ by three or something.

Right.

I mean, the integration team would know many examples of that. Where two different expressions, you know, where you do full simplify. I mean, I don't think we do this right now of saying, here's an indefinite integration result. Do a simplification. Oh, by the way, if you subtracted one from the result, you would be able to get a much simpler form.

And within our framework that's fully supported, you just put multiple answers in the answer key in the assessment function for these sort of edge cases. Right. But like the common edge case where you just. Just like first order integration of a polynomial where you can have plus C and if somebody puts plus A, it's okay. Even if they, if they omit it completely. It might be okay depending on what the teacher wants. Right.

I'm eye rolling because I think it's always okay, but that's.

Right. So the default is that it's okay for us. But putting plus three by default is not okay because I think most teachers don't want that. And putting plus A plus B also not okay because most teachers don't want that. Right.

Yeah. Okay.

Right. So. So we've kind of, we've made the default what we think, you know, 9th grade teachers want or whatever 11th grade teachers want. And so that's for like, you know, how it works for like, you know, first order integration where it's just something plus C, which is really the most common thing. And we also looked at differential equations where this, you know, the constants, there's lots of constants and they're all over the place. And what we had to support for there is as long as there's a coherent mapping of your constants to the constants in the teacher's answer, it gets more correct. But you have to have them all.

Yeah, right.

But you don't have to have the same C1, C2, C3. You know, you can have different C values or you can just use A, B, C, D. Right. As long as it's not, you know, X. You can't use X obviously.

Right.

D and D instead of D. That.

Sounds fine to me.

And yeah. And this. Chris came up with a pretty efficient solution for this.

So.

Good.

Yeah. So this, so this is added. And this will all work in calculus result by default now.

Excellent. Great. Okay.

So yeah, so we will untenant of that. And then polynomial result is one that we've added a lot of support for patterns. So, so like blanks. So you can.

Yeah, I understand. Why is that sensible? I mean, with a blank.

This was a request from the CBM folks that they had a specific use case for, you know, they wanted. So here's their use case really.

Right.

Where you know, if you have some coefficients. Right. But not others, you can get awarded some credit.

I see, I see, I see. That's complicated.

And this kind of just completely fell over in version 14.0. And now it works.

Well, good.

Right. It still doesn't let. Right. If it's factorized differently, it works.

And, and handling kind of a weird corner case. And I don't know how far the, you know, how long that tail should be.

Like. And like put it. Yeah. Putting a minus into a blank caused all sorts of problems that I think we've solved. So.

Okay. I mean, it's tricky because we're basically. You're basically, you know, skirting around mathematical correctness here. I mean, it's like. Like the concept of partial credit. It's like, you know, if I say square root instead of cube root, do I get partial credit? I have no idea. As far as I'm concerned, the main thing of interest is whether you get the right answer. And the partial credit thing is a. I mean, is a very weird.

Right. Well, we put the onus on the teacher here, right?

Yeah. Well, in time, LLMs will care about whether they get partial credit.

Sure, sure.

I don't know if that's going to be important, but it might be. I mean, in other words, if you're trying to get an LLM to learn something, I think it's sort of hopeless doing this stuff. I think it's kind of a bad idea. But, you know, like, with humans, saying, well, you were getting closer is more useful than saying, you missed type thing. All right, what's next?

Yeah, yeah, yeah, yeah. And yeah, definitely there's parts of this where, like, this will help with, like, explanation feedback in a tutor.

All right.

No, no. So that's all of the, like, new features. And then the rest of the agenda was purely talking about documentation. So right now for this listing, we only have pages for these three, and we are writing pages for the rest of them.

Good.

For in 14.1, I don't. Do you care to see the first few pages?

I cannot.

Do you care to ever see those or do you trust me?

I wouldn't mind seeing them at some point, but as a drive by, you know, in. When they're pretty close to complete, I know what to do.

I think I know what to do, too, but I, you know, don't know.

I wouldn't mind seeing them, but. But just to admire them, so to speak.

Sure thing. Yeah. So once. Once they're all written, we will. I will have Avery follow up.

Sounds good. All right. All right.

That was the agenda.

Okay, terrific. Let me just. Hold on one second. Let me just. Before you.

Oh, yeah. Did you have a name for this guide page is like, listing of Assessment Comparison methods. Nb.

I would think so. Okay, that sounds reasonable to me.

I will put it in CVS then.

All right, thanks very much, everybody. Good work. Okay, see you soon.

Thank you.

Bye.