Live CEOing Ep 699: Language Design Review for MathExpression

Okay. Hello, everyone. This is a Wolfram Language Design review about how we are setting up our assessment framework for math expressions. And the question of when is x squared minus 1 considered equal to x plus 1 times of x minus 1 and when is it not? Is that a fair summary of what we're talking about here?

Yes. Yes, that is.

Yeah. Right.

So we've had this assessment framework for a couple years, but mostly it's kind of relied on exact matches in, like, a multiple choice situation or sort of unambiguous matching.

And.

But since, you know, we're kind of very well positioned to do more complicated stuff surrounding math.

Right. The problem is that at different levels of math, I mean, to a calculus student, x squared minus 1 is equal to x minus 1 times x plus 1, but to an algebra student, they're not supposed to be considered equal.

Yes, that's right. That's right. So if we're just calling it math, we need to support all those cases.

Yep. Okay.

So, yeah, so from the. I have this demo notebook that's kind of working at a very high level, and then we can get into the weeds as we move on, if that works. So, you know, you can do some, like, question object, and you can, you know, put the answer in here. And we support things like tolerance where just the total difference between the answer has. Has some acceptable tolerance on it.

So, so hold on, hold on, hold on. People are not using slide rules anymore, are they? You know, a slide rule, right? You have to read off the numbers and you might have a slight mistake.

Right.

The only way people are going to have different things is if they're using, like a calculator or something and they didn't type in as many decimal digits. But otherwise I don't understand how they would get 11.01.

Right. So maybe they're using a calculator. Right. So, yeah, the idea is not necessarily that the student is solving these in Wolfram language, it's that it's the teacher's tool and they're presenting the notebook and the student solves it, how they solve it.

And you might have a question, but I'm disputing for an algebraic case that numerical tolerance makes a lot of sense. By the time it's algebra, they're probably not doing approximate numbers.

Okay, well, I mean, if. If the coefficients here were approximate, maybe it makes sense.

Why would they ever have that? Nobody does that, do they?

Maybe it's an engineering class where they're read from some table in some book. Right.

All right. You know, engineers have tables Because I think. Yeah, okay, fair enough. Okay. For example, do we have unit matching? Do we know, you know, if somebody has a. Same. Same kind of thing? If they say joules versus they say kilograms. Meters.

Yeah, that's been. That's been built in for a while. Yeah.

Do we automatically convert or do we. Do we have the same kind of thing? Because I can imagine. Okay, but let me make a comment about that. It's the same deal to some people. Jewels. And. And. Well, it doesn't matter. You don't need to write it out. Kilogram meters per second squared. Okay.

Here we can do control equals, right?

No, I'm not sure that's the. Wait, hold on, hold on, hold on. For example, kinetic energy would be mass. Yeah. It would be mass times that squared. All squared. Meters per second. All squared.

Okay. I can't mass into it.

I think you can just click it. Just click it. I'm not about to. You accepted it. You can't.

Yeah.

Can you do that? I'll try it. Yep. Okay.

Right. Oh, unable to. That's strange.

I don't know. With the search points. What the hell is search points? Quantity. Hey, you've got. You've got weird error messages there with nearest.

I mean, this. Oh, because I did. Okay. Search radius. Well, that's.

What the hell is that? Bug?

That's a bug.

Crawled out. Got it. Why do you have to do a tolerance for a quantity? I don't think you should.

This is a bug. This is definitely a bug. Into quantity sampling. You shouldn't need to do a tolerance. But it accepts it and it does.

No, no, but you should be able to do a tolerance that is a pure numerical tolerance, which will be the numerical coefficients. You get what I'm saying? Just like it is in the algebra case. You're not saying. There we go.

All right.

Okay. We'll put in a five there. Put in a five. Is that the quantity? Is the. The as the thing you're assessing, too. Okay.

That just happens. Yeah.

Yeah.

Oh, I guess they're. Yeah, they're equivalent. Yeah. All right.

Okay. But. But hold on, hold on, hold on. Stop for a second. Whatever framework we can give for the equivalence of algebraic expressions, we've also got that for unit expressions. We've also got an issue for unit expressions. Do you understand what I'm saying? For some classes, joules are the same as that kilograms, meter squared, blah, blah, blah.

I see.

But for some they are not.

Yes. Right. So, you know, what we've had for a while is a quantity comparison method with an Assessment function which assumes that you want to automatically convert.

I know. I'm saying there's probably a case where we don't want that.

Right, right. We also have like that where it could just be like a held expression is a comparison method that we have that.

Okay, yeah, fine, but that's too detailed. I'm just saying, I'm just flagging this as another thing, but let's keep going on the.

Right, so with so within math. So you know, what we're looking at today is, you know, you have your answer key and then you have Math expression is what we have named. Right now we can. And then within this we have a method option that allows. Is going to allow all of this, you know, customization.

Okay.

And so right now we have two primary methods. Match, what we've called match and what we've called equal.

Do we have people from the Wolf Malpha math team here? No. We probably should. Yeah, maybe. I don't know who because. Because what I'm going to say is, you know, even a communal garden LLM now knows things like what fifth grade math is about.

Yes, I. We can, we can talk about LLMs if we want. I have that. I have an example that we can get to later.

Okay.

Right, so, so that we have match which eventually feeds into match Q and we have equal which eventually feeds into, you know, double equal.

Okay.

And you know that there's those are, you know, the equal match Q of.

The unevaluated expression or match Q of the evaluated expression.

By default unevaluated. We have sub options that let you control that evaluation and you can customize it on. And yeah, the, there's sub options here that get into the weeds. But we've tried to make the default very sensible. By default, when you're working in match method, it's unevaluated. Everything is either inactivated or held. Right. So you know, here's an example.

Can you give me big picture of. Of the my X squared minus one example. How do we deal with that? Right.

So if you have. I'll just do assessment function of X squared minus one.

Yep. Yeah. X squared minus one. Right. So.

So you know that's always, that's by default going to be considered math.

Right.

We have some heuristics that look at it and say it's math.

Okay.

So so we do end up on this framework automatically and by default right now. And this is something that we want to discuss. Yeah, it's gonna pass.

Is it?

Okay, that's right. It's gonna.

Okay, so now put an expand Copy down. No, no, no, leave that, leave that, leave that. Copy it down. Put an expand on the right hand side there. No, no, no, not there. Okay.

So Chris, is this going to match by default then or is this going to. Oh yeah, this is going to equal by default. But even equal, even the double equal thing, you know, X double equal Y. If one polynomial is expanded, one is factored. Yeah.

It won't consider it equal.

Right, right, right. So that, that's what's happening here is it's getting passed to something like this and it's not determining that it's equal. So it's giving incorrect here. Right. And there's two main sub options that we've introduced what we've called inactive pattern, which isn't the best, most beautiful name, but it's, it tells you what it does and something we call transformations. So inactive pattern lets you control what evaluates and doesn't. Basically whatever you put here gets passed into the second argument of inactivate. So it's heads there can be accept. I know, but people on the live stream might not. So you can, you can very much control the, the evaluation. So you know, maybe the only thing you don't want to allow to evaluate is the multiplication between these terms, but the pluses where the minus signs are, you can allow or you know. Yeah, so it, it works out. So you can reorder terms here, but you can't expand terms, that sort of thing. And it'll be correct. Right. Here's another one where we've inactivated power. So you know, the division evaluates and. Right. So and then the second main option here is transformations. And these get applied in the case of equal. They get wrapped around the entire equality. Right. So you have, you know, transformation one.

So it's F of one equals, equals F of two. Basically.

No, it's F of X equals, equals Y of. You know, which is important because even if you have like the factor polynomial expanded 1, if you do simplify on both sides, it might not end up in the same place. But if you just simplify around the equation, it's gonna, it's gonna really find it. Yes, yes.

Okay.

Yeah, yeah, yeah. Danny Finn pointed out a great example to us early on of that exact use case. So we, we decided that the transformations in this case apply around the entire quality. In the case of the match method the transformations are applied to each half independently. So you know, the, the transformations for the equal case are much easier to come up with because they don't have to like operate through a hold generally so you can, you can use normal like expanse and fly in the match case. You know, you can do things like replace and that works just fine.

Yep.

But we're also, we've also come up with some named ones that we haven't documented yet. But we think this idea of the shorthand name transformations.

Yeah, I agree. Okay, let's get to the end of the agenda. Let's come back to this. What, what's the next thing you're going to talk about here?

Okay, here's an example of one that came out of Chat GPT. So we can keep going into sort of lower level details. Get into that.

Okay. No, I think I'm understanding that you have a concept here of equal or match.

Exactly.

Why isn't it called sameq versus equal?

Well, it's not passing the same queue, it's passing to match queue. We want to, we, we can support things like blanks. You know, you can do a hold of x plus blank.

Why the hell would you want that for math?

Oh, I mean you don't, you know, you don't necessarily want. Imagine an indefinite integral where there's a constant and you don't necessarily need to use the big C. You can use anything, right?

No, you can't use anything because you could put an X in there and you would have the wrong answer.

Well, well, maybe, maybe you have a panel.

No, no, nobody's gonna do that. Okay, let me ask a meta question. All this stuff, have you looked, you know, talk to the folks like Bradley here who do math content?

Yes.

Who'd you talk to?

We've had multiple meetings with Bradley and.

Okay, fine. Yeah, okay, okay. Because the thing you just said is a non starter of plus blank. It's just wrong. You know, some kid is going to write plus sin X. I think a.

Sophisticated teacher could come up with good pattern based.

Yeah, they're never going to do it. If we do that. We want a thing called indefinite integral. It's just a string name. But we've done the right thing.

Yeah, yeah, yeah. That totally is the sort of thing that we're planning on. Yeah, yeah, we don't.

Right. Because integrals are kind of complicated because they can have a constant difference. Right, exactly.

Right, right. So. Yes, yes. Yeah. We haven't specifically talked about an indefinite integral one, but that's the sort of thing we want to build out more.

I think the blank thing is a non starter, I think. Right.

What do you think the advantage Of Same Q vs Match Q is?

I don't know as well what match Q actually does. I mean, same Q is a well understood thing for comparing expressions. I don't know what match Q does with numerical precision and things like, I assume it uses the same cube, but I don't know, which is kind of embarrassing after all these years. It's never come up. Match Q has never. Has never come up as a thing trying to match approximate numbers for me, I don't think ever.

Chris and I definitely discussed saying Q versus match Q and we decided that we might as well support patterns, so we might as well use match Q.

I think that's offering people too much freedom that they don't need. I mean, you could support match Q as well. You could support all three.

We could.

I think that makes more sense because I think it's. Now the question. Okay, so let's walk through what the issues are. Okay. Issue one, is it held or is it not? Issue two, how do you do the comparison? Is it. Is it same Q, match Q? Whatever. Okay, Issue three, what transformations can be run? Either a single one or multiple ones on either the full. You know, there's no point in running a transformation on the same queue because the same queue is already blown up by the time the transformation happens. Exactly. But there is a point in potentially having, you know, F of blah same Q to F of blah.

Right, that's. Yeah, we'd have to do the transformation that way within same Q.

Right. But, but. Okay, so let's just walk through. I mean, did I get all of the options here? All of the different cases? Are there more cases? Right.

Well, we have the tolerance, which we can discuss.

Yeah, no, I know, but that, that's part of the. That's something that will be fed into the comparison function. Like does match Q take a tolerance? You can look it up. I don't know.

Oh, I don't think so now.

Well, that's complicated and weird and I don't know why it doesn't. I mean. Yeah, I don't know why it doesn't. I would have thought.

I think it's VAR args. So options.

No, I doubt it's varargs. No, there is no. There is no tolerance in match queue. No. Yeah, but does it use sameq or equal for its way of comparing numbers? I bet it uses sameq. Doesn't the sameq have a tolerance? Maybe some. Well, why don't we just look it up, bring the documentation up. Does same thing.

Yeah, it has the same test, I think. I don't know if it has a tolerance.

Oh, it's varogs. As well. Yeah.

Oh no, it doesn't even have same test, does it?

No, it doesn't. Okay.

Yeah, it's. Yeah, it's vargs and it works on rules, so.

Right, okay.

Okay.

So did I capture all the major. Major.

Yes, you captured the major issue as well. Yes.

Okay. All right, so let's go through these then. All right. I'm not keen on the way you've done this with the sub options for methods and so on. To me, I mean, isn't assessment function. Doesn't it usually have a second argument which is the assessment method?

Yes, which can be inferred and that's what's happening here. But it's the same as putting math expression. We definitely considered instead of having this method value having a math match and a math equal and a math same.

Well, I think that is way too complicated for most of our users.

I thought so too. I thought yeah, having. Having this with heuristics that usually get it right is.

Is. Is math expression. Is that second argument annotatable? Do you have other things where there's like a list there?

Not. Well, it can be a full association where that becomes a thing. That's comparison method.

And what are the other.

Element wise assessment goes to true where you know, it treats a list as a bunch of an. I don't know, we can look at the. You can specify a custom comparator or how to combine the scores when you have multiple parts.

Yeah, okay, right. I remember some of this stuff. Right, ok go back then to the other. Go back to this case. Okay, so and this is perhaps a Bradley question. If you just look at the math hierarchy, so to speak and you know, first grade, third grade, fifth grade, I don't know what. Right. Is there a way in which the equivalence of math expressions gets ever looser, so to speak? I mean, you know, 4 over 6 is not the same as 2 over 3 when you're learning fractions.

Correct.

But later on nobody really cares. I mean somebody would say, somebody might say points off because you gave four over six in the answer to your integral rather than two over three. Would they say points off for that or would they not.

At I feel like at like the calculus or the university level, you stop getting points off for things like that. I don't know if AP Calc in particular would mark off. Probably they would mark off for things like that.

Okay, fine, but then we've got some waypoints like the AP calculus rubric like you know, we got these various rubrics. Seems to me that's what we should be concentrating on, if there are rubrics for this, you know, have a version of math expression that says, you know, math expression, AP calculus or something, Comparison method arrow. AP calculus, comparison method arrow. You know, I don't know, basic algebra or something, I don't know, or some class name or something like that. Because each one of these things, there's going to be a lot of complicated stuff of whether, you know, you can consider.

Right. I thought we would have a bunch of things as, you know, just here, AP calculus within math expression as opposed to a different comparison method.

Possibly, yes.

And it also depends on the problem type. Right. It's, you know, a lot of the problems are, you know, write this in a particular form, in which case the expected form is very strict.

Right.

If it's a form of a line equation or a form of, you know, mixed number of a fraction versus a improper fraction, things like that.

What is a line? Did you say?

A line expression like point slope 4 form of a line, whereas y equals mx plus b versus.

The amazing thing is the way that the, the, the, the jargon of all these different forms has evolved over the last 50 years or something. Anyway, that's just my amusement. But, but, but say you're looking for the point. What did you say? The line form of a thing?

Yeah. Point slope form of a line or standard form of line.

Okay. Point slope form of a line. So let's say you're looking for. The answer is, you know, 4.4 comma 27 or something. And that's the thing you're looking for. Right. Okay, well, I mean, then, then is.

A point slip form an equation or is it just, it's just a pair of coefficients.

I'm sure it's a point. Which means, which means an actual XY coordinate. I would say so. Right.

Point slope form of a line is, is that the line equation? The Y equals MX plus B and it's that particular equation.

Okay, all right, yeah, fine. But my point is here that, so if somebody said, you know, U equals 5v3, that that would be an A. No. Right. You don't get to do that. Is that correct? You couldn't use any variables other than X and Y true or not true?

I think sometimes true. But the problem statement would typically be, you know, use X and Y or use U and V as the variables.

Okay, but I mean, basically the issue here is there's a lot of kind of content knowledge in, oh, it's an indefinite integral. But you, you know, like, for example, okay, so here's an example for an integral. Okay. It's a Maya G function. Well, actually it's just sin squared X, but I give it as a mayor G function. Am I right? Am I wrong? It's mathematically equal. To most calculus teachers, the mayor G function is not considered the right answer. But why not? In our. In our formalism, so to speak, probably full simplify would say it's. It's equal to sine squared X. Am I making sense?

Yeah.

You don't need to look it up.

Okay.

I mean, many, many integrals can be done very, very mechanically in terms of MAGI functions.

Okay.

Right. When I used to be in the business of doing, of doing integrals. I don't know whether I have a test type things after. I was really quite young, so I never had the opportunity to hand in a, you know, a. Here's the answer. It's a MAGI function. I didn't know about MAGI functions. The hypergeometric functions, what I would have said in those days. And, you know, it's kind of like. So I never did the experimental test, so to speak, but it's certainly exotic and, you know, but the whole point is those things are deeply mechanical. You need to know a lot less to say the answer is this as a major G function than you need to understand how to, you know, translate it into sines and cosine and so on. Okay, so my question is, if we were to take various rubrics for things, AP calculus, this, that, and the other, how many of these are there, do you think? Badly?

Oh, gosh. I mean, the, the College Board stuff is probably the best one for the U.S. i can't really speak to the.

Okay, the GAO cow. Who knows what the gaokao does? I, you know, we. We don't know. Somebody probably knows, but I, I know.

The AP, like grading heuristics specifically mentioned things like, you know, radicals in the denominator of a fraction and things like that. So.

Okay, so we would have a fighting chance of being able to. I mean, to me, one of the things that's complicated here and that we could potentially handle well is, well, you've got the right answer, but not in the right form. And I. We have a way of specifying sort of partial credit, don't we?

We do. We do. Right now, the assessment result object always has a Boolean correct value, but it also has a score value that can be anything, and you can pass back an explanation that's presented to the user.

All right, but, but I mean, to me, this Is a really, you know, we gotta ask educators and things what, what they need. Because I, I, I don't know. I mean.

Right.

You know, for me the hypogeometric function is the right answer as far as I'm concerned. It's not the best answer, but it's a, you know. Now in terms of, of. Let me think for a second. Right.

I mean what's great is any choice.

Okay. We may be able to have some el cheapo things. I think they will not be widely understood. I think the only way to do this for real is a, you know, an actual content specific curated rubric XYZ type method. And I would even, I mean, I think isn't the term rubric a fairly common one for describing, you know, what's considered acceptable and what's not? That goes beyond the just right answer. Wrong answer. Is that, is that not correct? Somebody educating, education oriented respond to that, Bradley?

Perhaps I don't know what the official term is.

Okay. We should find that out. We have plenty of people in the company know this stuff. I think you guys should do a separate meeting with some of the folks like Alec in the UK and all kinds of people who know about this stuff. I do not. Okay.

We can set it up right.

Every right. I mean and then, and then what you want to ask is okay, so for what you're doing right now, I mean I have no idea what heuristics is math expression using to determine equality.

Right. I mean by default it's you know, inactivating local symbols and sending it to equal.

Just to equal, not to full simplify.

No, that if you, you can do transformations goes to full simplify if you.

Want that people are not going to understand that.

But you know, one of these named methods we have is simplify that it will do that.

Right?

It's just a convenience. On top of equal we have a doc page Simplify.

You want full simplify? I think by the time you do, I, I don't know. In some cases full simplify is going to blow up your exercise, right? If you say is sine 2x the same as 2sin x cos x people? You know, I mean, this is why the only way to do this in my opinion is, I mean, okay, one thing we do not have is all the possible mathematical transformations that we can do. We don't have a big list of them. I, I, I tried that actually in S and P in 1980. People didn't find it very useful. Maybe the world has changed and they'd find it more Useful today of just saying these are all the transformations, you know, number them, label them, you know, which ones do you want to apply? Okay, so, but Bradley, do you want to comment some more on what you think should be done here?

Most of my familiarity with this stuff is through working on problem generator. One thing that we did there that I'm not seeing here is a, A atomic test and not, not capital A, atom Q or anything, but we, we checked to make sure that there weren't, you know, symbols like integrate or Meyer G or things like that. So.

Right, right. That's a good. Right, right. Okay. You know what the basic bottom line here is this is going to be a big heuristic mess and we shouldn't pretend otherwise and we should provide content dependent, you know, names here. So I don't even think, I don't even think math expression is the right concept for a comparison method. It's not a comparison method. It's a type of thing. I mean, you know, mathematical equality would be the right thing to call it, if that's what we were dealing with. Well, which is what you've said, math equivalence over there. Yeah, the method option. I mean, comparison method. Math equivalence is a reasonable thing to say.

And math equivalent other comparison methods we have, if we want consistent grammar, 10 are mostly nouns. We have things like quantity and geoposition and.

I see. Okay, well, fine, fair enough. But then I think math expression should mean we try as hard as we can to find out if these things are equal, which means we use full simplify, we do all these kinds of things. And everything below that is some kind of rubric for some particular educational level and so on.

Okay. I mean, choosing defaults is certainly hard. If you think the default should be heavier weight. I don't have.

I think the default should be. Is the thing mathematically equal? I think, I mean, because I don't really see other than, you know, it then comes down to specific operations, you know, commutativity. Do we include associativity? You know, commutativity, two plus X is the same as x plus two. Okay, we believe that right. Now what about associativity? You got extra friends in there.

Right?

You know, I don't know. I mean, for some, you know, four over six versus two over three, you know, are they equivalent, are they not?

Right, Commutativity. We, you know, for the case, for the match method, Chris did work to make that work. For things that are orderless expressions.

Yeah, well, right, maybe. I mean, I, It's a total mess, right? I mean, to me it's like, is it mathematically equal? Which means we try as hard as we can. You know, you're doing a class and control theory or something, and an unevaluated integral will be considered. Okay, perhaps. Right. Get what I'm saying? Right.

Here's the convex optimization problem. I'm done. Yeah, those things.

And, You know, I think. And I think for, You know, it's calling math expression. I mean, maybe we should say. Maybe we should break that more into algebraic expression, because I can see not wanting to do transformations. I think it is a more advanced kind of math where you're doing, you know, let's say it says algebraic expression words, algebraic equivalence. Okay.

Okay.

That means you're doing. Not sure if it's a comparison. Algebraic expression. Yeah.

Equivalence, you said.

No, I said. I mean, if you're using nouns, you got to say algebraic express.

Okay.

But that means you're not allowed to do sin 2x goes to 2 sin x cos x. Right. That just means, You know, I mean, conceivably there's an arithmetic expression. Algebraic expression. Math expression. Okay. An arithmetic expression. What does it do? It. It doesn't do. Yeah. Algebraic expression would take sine sine x squared minus 1. It's the same as x minus 1, x plus 1. Arithmetic expression. I don't know.

I mean, I think it would inactivate everything except a small family of symbols and then feed it to equal and wrap it and simplify or full simplify.

Bradley, what do you think about this?

I think this is the right track. Again, with problem generator, we have specific kind of answer checking parameters for different problems. Right. So we have one, you know, the.

Entire universe of problems coming in.

Yeah, essentially we know. Right.

Okay, what. What is that? What are the options there? How many are there?

Dozens. And. And some of them rely on each other.

Right.

I mean, there's. There's problems that expect an integer. That's easy, that's simple. There's ones that expect a particular form of the fraction. So there's two or three of those. There's ones that expect a particular form of a polynomial, like the factored form or the fully expanded form, which are different heuristics for each of these. So there's lots of them. But it's very problem specific. And I don't know how to generalize that until you get to the algebra level or the calculus level.

Well, the problem specific ones, remember, these are specific problems that we have here so people can, you know, if they know it's supposed to be 2/3. They can just say it's supposed to be two thirds. They don't need to do any fancy, you know. But the problem generator is different because it generates problems. So it doesn't know, it doesn't know what the answer. You get what I'm saying? Whereas here the teacher can figure out, oh, I think the answer is blah, and put that in. Right. So I think the problem generator is a more extreme case. Look, I think you guys need to go back to talk about math content here. I, I think, you know, as an. My external view would be that arithmetic, algebra, math are perhaps reasonable levels where math means, you know, your calculus. And above, I think you're going to need specific things for things like indefinite integrals where it can differ by a constant.

Right? Yes. So you think this differentiation between, you know, arithmetic and algebra should happen at the comparison method level and not at the method option level?

Yeah, I think so. Because people really want that. I mean, they know they're teaching algebra class and, you know, it's a, it's a big thing. They don't want to be filling in a method option every single time for every single.

That seems, that seems reasonable to me. The, the line there is blurry about what should be comparison method.

I agree, but I, I think in a first approximation, you know, if you are teaching a particular course, unless you are making a very specific point, you should be able to use a comparison method that is somehow keyed into your course. See what I'm saying?

Yeah, that's a rule of thumb. That makes sense.

Right. So I, you know, I think we need to iterate this. You need to go asap, talk to a bunch of people within the company who know about these different courses. I mean, I don't know who's done, you know, I know we've done a lot of stuff matching to grade levels and Common core standards and things like this. Whoever's been involved in that will be a good person to ask about this. Okay, then I think you gotta, gotta code up the. I'm, I'm gonna say pick like five rubrics. Okay. And maybe we make a rubric up that's, you know, like algebra. I don't know what grade people do algebra. And I, I have no idea. Fifth grade. I don't know what, what grade do people do? Algebra.

7.

Okay. I wasn't so far off. I was on earth by two grades then. Okay, fine, But you know, you know, define math levels, algebra, calculus, you know, whatever, arithmetic, and have comparisons appropriate for those levels, I think, and try to have the general case of those and then maybe there are special casing for people who are, you know, teaching a very specific, you know, class, etc. Etc. Etc. Make sense?

Yes. I think that this, that still falls into this framework. I think all of these things are basically still going into the controlling evaluation, applying transformations.

Right. So, okay, so let's just look at the, the do it yourself approach.

Right, so this is the do it yourself approach. Right. Do we want to stay on the demo or do you want to look at the doc page that we've.

Let's look at the documentation page.

Okay, so this is a new kind of documentation page for comparison methods.

Good.

Where this is the biggest one. All the other ones are pretty.

I think this is wrong. I'm sorry, I just think you should break this up. I think by the time it's a math expression, we just, you know, kind of know what we're doing. By the time you're talking about, you know, equal versus same Q and so on, it's not a math expression anymore. If it's a math expression, we're expecting it's an equal comparison, blah, blah, blah. If you want something different from that, I think you want to call it symbolic expression or something.

But I mean, there are a number of these. It's useful to use, match and apply transformations that are specifically designed for mathematics. Fine, but you don't want associativity or communicativity.

No, look, look. Okay, look. There is a function which generates a canonical form either of the left hand side and the right hand side or of the combined equality. I don't know whether it's useful to do it the combined equality. I don't know whether our functions actually pay attention to that. Where the full simplify basically maps across the two. Two sides to get what I'm saying. I think it does.

No, it does the full expression on. On inequality.

Okay, well fine, but then that's. So, you know, so those are the two cases, it seems to me is either you're doing. I mean, please check that with somebody who really knows because I don't think that's right. Okay. I don't think it looks separately looks at the two sides together, but I might be wrong. Okay, but the more common case, and I don't. Yeah, I mean what I would do is just say you've got an equality expression with a double equals, not same Q. Right, Just double equals. If it's math. Right, you can provide a function which is the thing to bash that with to try and resolve it end of story. Just a function you can provide full simplify, you can provide whatever you want. No, no quoted strings, just there is a canonicalization function and maybe that should be an actual option.

Can you hold or inactivate no sub expressions?

Why would you do that?

Well, otherwise students can just pass in the question as the answer. They could just give code and it'll evaluate.

Well, I mean, that's what you have to do. When you say math expression, you've got to inactivate all those things. That's, that's what it means to have a certain level. I mean, maybe that's the definition of math expression as opposed to symbolic expression. Maybe symbolic expression is whatever it is with the bashing function, with the canonical causation function there. And if you say math expression, it will deactivate, it will inactivate everything that isn't math, everything that isn't resolved math, if you understand what I'm saying. Right. So inactivate, integrate, inactivate, factor, all those kinds of things. Right. Just don't inactivate the mathematical functions, the numeric Q, you know, numeric math and some other non numeric mathematical functions. Does that make sense?

I understand what you're saying. I'm sure that there are counter examples to why that's going to fail, but I'm not coming up with them at the moment.

Okay, but that's my suggestion. Inactivate everything that isn't math. Yeah, there probably are examples. I mean, Michael will be a good person to ask for that. Right?

You know, functions that you're thinking of as mathematical functions that you don't want to evaluate. Because it would ruin the question.

Yeah, fine, whatever. I mean, I know they're going to be exceptions, right. We're going to give people various rubrics and just like, you know, a code of laws or something, people are going to find loopholes.

Right, But I think we want to keep this do it yourself open framework that we have here.

I don't think. I think it's too complicated. I think it's not the right thing. I think you should give a comparison function. It's either equals or it's same Q, whatever. And a canonicalization function. In fact, the whole thing that could be wrapped together. It's just the essentially assessment function, which is either full.

I think the only difference between that and what we have is that we also have the inactive pattern option, which I think you need.

What is an active pattern? The thing that.

So before it passes it to the test function, it calls inactivate with Whatever's here.

I thought we were agreeing that there would be a bank of those things for math expression.

I, I think that's. I think we can come up with a collect. Yeah, collection of named.

No, but then you're going to want to both activate or inactivate those with respect to those names. See what I'm saying? You could say, oh, but for my particular class, I want the binomial coefficients to evaluate.

Right, but I also think we should have one that's just open and you can specify things directly force people into named classes.

I know that, but, but look, they're not going to think of which things you have to inactivate, okay? They're going to get it wrong all the time. Do I have to do subtract, Do I have to do minus, do I have to do this? Do I have to do that? Yes, they're not going to get it right. It's quite complicated.

It is, yes. I don't think we're going to get all the classes right either.

Right, but. So I think we should just punt completely on this and if they want some detailed inactivation thing, they should just write code to do it. And there can be a function repository entry for some specific case of a class about binomial coefficients or something. And that's how it's going to work. Rather than us trying to partially do it and have people use inactive pattern on things where they're going to get it wrong and then they're going to be unhappy. I mean, I, I think it should either be completely. You get to do the comparison, you write the comparison function. These are not incredibly complicated, these comparison functions. I think having, you know, either that or it's completely kind of. We babied you and we did it for you with a, with, you know, with a named comparison. Those are the things I think we should be doing and I think we should be clever about the name comparisons. Now whether whether those name comparisons need sub options, I do not know. That depends on details about math education that I don't know make sense. I mean, what do the CBM guys do? What have they done for math expression comparison? I don't know.

I don't know that they have.

They do.

They do a lot of free form.

Math inputs that they do have. Math inputs. Okay, So I, I think this, I'm sorry, but I think, you know, it's what happens in these design reviews that the, you know, I think, I think this didn't go quite in the right direction. I don't think you want These weird, complicated sub options. I think you either want, we're just going to do the right thing, math expression, algebraic expression, whatever else, or it's pure, do it yourself. And I just don't believe, I'm sorry, I just don't believe in these, these partial things. These partial things may be super useful for us to implement, you know, algebraic expression, for example, but that's on us. And you know, I, I think that's what we have to do. Either people use the, the, you know, the, the things that we've created or they write their own. And possibly we can put in the documentation page, we can put the code that's used by our comparison functions, possibly if that, if people find that useful and then they can just modify it, take out a transformation, add a transformation and so on. Does that make sense?

Yes.

Okay. Anything else we should make sure to cover? I'm afraid this is going to have to come back another time, so please, please work with the people who know math content stuff.

We have and we've done plenty of collaboration.

Okay, well, I'm afraid it needs. I think I gave you an outline. Unless you violently disagree with it, I think we should wrap up here.

Okay. Other people, Chris, Stephen Schroeder, Avery. Have questions?

No, I just wanted to say that probably with the low level development that we have done here, we can answer all these specific problem. So that was the point actually to say that having this inactive, having this transformation, we can build a specific answers for that. Sounds good to me. Right. The point that I'm making is I don't think this intermediate layer is terribly useful to users. I think either the complete do it yourself where they understand they can write code, or where we've given them a fully babysat version of. I think those are the two useful endpoints. I think this intermediate one where people. Look just, just to explain this, you know, anytime people have to understand a new sub language of, you know, a whole nest of options and so on, that's asking a lot of them. And it's only worth doing if it really has really high payoff relative to they just learned the generic Wolfen language story and you know, that's where they go, okay, what's the cell?

But we only need to teach it to one person. We only need to teach these, these, you know, sub options to, to the LLM.

Yeah, right. Well, it won't get them. It won't get these. It's a, it's a long training path. It's got to have lots of examples. It's just not worth it. I mean, it's going to have a much better chance of writing the generic code than it is writing anything with C sub options. It's never going to learn it, it's not going to learn.

I understand, I, I, I, I understand what you're saying. I don't think we have to throw away code. I think very much what we have, we'll be able to use, do these things. I think we'll have less documented in 13.3, but we'll have better stuff when it does get documented.

Right, but, but let's, let's try and work as quickly as we can to get a few rubrics that we can document. Okay, sure.

Yeah, that makes sense.

Even if they're experimental and even if we find, oh, there's a loophole here, we have to change this, etc. Right. I mean, you know, I think again, I don't understand the education stuff well enough, but, you know, it seems like grading is somewhat, you know, it's, it's somewhat subjective at some level. I mean, whether, I don't know, you know, in, in, Yeah, I, I just think if we had to change the rubric slightly because of a corner case, I think that's okay. But I think we should try to get these things appropriate for different levels of courses. And I, I would say there's a, you know, there's a standard course sequence by grade level, so let's make sure we hit each of those in an appropriate way. Makes sense.

Okay.

All right. Good stuff. Okay. I mean, this is going in a good direction. Thanks. Okay. Bye, guys. Bye.