Live CEOing Ep 753: Language Design Review of Question Framework for 14.0

Okay. Hello everyone. Welcome to another design review as part of our version 14 season of design reviews. We are looking at our question framework and let's take a look here. All right, so partial credit.

So this is something that's been in for a couple versions where if you have, if the student. So this, the student's answer is that 3, 1, 2, list. And if that is a list and you say element wise assessment is true, then each element gets assessed on its own and the total assessment, it can do partial credit and it kind of combines the scores sort of thing linearly.

So if they get two by default.

Linearly, we have all sorts of advanced options for customizing it.

Okay.

But that doesn't support common questions that we want to support for other cases where the student's answer is a list. Right. So the example right below, you know, find the roots of the polynomial. There's four roots. They need to give them all. If we used element wise assessment, you could just give a subset and that's. Yeah, more usually you want that. It's want it to be an all or nothing, not a partial credit situation in this case. Right.

So wait a second, I'm sorry, the. I'm sorry, you want them to give all of them, but they can be in any order.

Yes, so the teacher wants to. Yeah, exactly. The teacher wants them to give all of them in any order and they don't want to have to write their own comparison function for.

Okay, so we just need a thing where it just says allow any order.

Right, Right, right. So that's what we're doing. So we've named it List Interpretation. See the next one down in the light blue?

That's a crazy name. I mean, I'm happy to rename it Incomprehensible name.

Right, so it's. If the student gives a list, how should it be assessed is what it's saying. Yeah, list assessment might be better. That's one of the.

So what are the other right hand sides?

So if you scroll down a little bit, I have some pseudo documentation here.

Okay.

Right, so we've. Element wise assessment used to be the, the key and it took a boolean value. Now it's the right hand side to this new option. And, and it'll be backward compatible, but we'll change the documentation to the new thing.

Okay, Okay, let me see. So look, we can't be inconsistent, ordered, list unordered. I mean, okay, what other things do we have in the, in this assessment? Function options in that association.

Right, so. Right, so you can give like a Custom function. Instead of using bunch of.

What are the names of these things?

Like comparator selector. Okay, yeah, that. The score combiner and answer correct combiner. Those are the things I was alluding to earlier where you can, instead of just totaling up the partial credit, you can have a custom thing.

Okay, look, I suggest we call it like list handling or something. And then we say. And then it's like something like orderless, Ordered.

Let's read this. Order list and ordered are also element wise. So if we're going to change a name, we might as well change it significantly to.

Oh, wait, wait, wait. What do you mean that Also element wise.

So the assessment function is applied to the elements of the list as opposed to when it's none, the assessment function is applied to the full list.

Okay, well, so I think then that those should be two different options because you're either applying it to the whole list. So I mean, basically it's listability, right?

In some sense, yeah. It's analogous, I would say, I guess, you know, if it's. Yeah.

Well, okay, so. I mean, So the case, the default case. I mean, so then the other ones might be element wise, ordered. Element wise, orderless. And now is there another case? You've got three cases up there. Element wise assessment.

Right, right. So, okay, so the, the other tricky part of this. So, so it's comparing the. How we've done ordered, ordered list and unordered list. So we have it implemented. Now, the answer key, the values of the answer key are lists. Right. So it will make sure that the values in the student's answer correspond to the complete teacher list.

No, that's not what you just told me. You just told me it was element wise.

Yes, but they're compared element wise. So if you do, if your comparison method is, you know, calculus, then it's going to use the calculus comparison method to look at one element of the student to one element to the teacher.

I understand, but does the student have to have the same number of elements as this?

Yes, that in, in these new methods it does.

Okay. Except for the element wise one.

The element wise assessment is different. Yes, that's why it's different. In that case, typically the. The answer key is not a list. Right. The values in the answer key are not a list because it's only. It's doing in each one. Yeah.

So I don't understand why ordered isn't the default. What's the difference between the whole thing and ordered?

Well, the whole thing, the assessment function that like the internal assessment function gets passed the entire list. So you can imagine treating it like, you know, a vector. Right.

Okay, explain this again.

So it's instructed to consider the case where you had some tolerance. Right. So if you. If there was tolerance in here and you were looking at the whole. If you had none, what we're calling none so far, then the tolerance would be like the norm of the differences between the vectors. If that makes sense. Right. The whole things are being passed to the comparator. Whereas if you had tolerance, and you said what we currently call list interpretation goes to ordered list, then the tolerance would be applied to each element. So each element in the student's answer needs to be within tolerance of a distinct answer in the teacher's key.

Okay. I mean, I doubt this is all the cases. So this is.

Yeah, there could be more in the future, but this covers, like, the Wolfram problem generator question set.

Okay, so let's just understand this one here. So this one here is. It has to be all the elements.

And they have to be in the same order as in the key. But the comparator is applied one by one.

Now, I don't understand why both of those. So whole list, you say, can't be orderless. Yeah. Okay, so this here, they might have a different number of elements. Correct.

Separate elements. I'm not sure what you.

That's element wise. That's what you were calling element wise.

Okay. Yes. They might have a different length, a different. Yes, right. I like to think of that as, like, you know, name as many vice presidents as you can.

Yeah. Okay, Let me think about this. Come on. You guys can do better than this. Come on. This is a model. How about anybody else speak up? I mean, you've been thinking about this for a while. I'm coming into this cold, so.

I agree. I agree. I agree. Yeah. I mean, I. I thought putting them all on the same property made sense.

Well, fine, but. But let's list.

Partial.

Well, element wise. Look, look, look, look, look, look. I mean.

I mean, what. You have, the line above that seems perfectly reasonable.

I don't like it.

I. I mean, whole list is the same as what we were calling none.

Yeah, that's right. That makes sense to me. I mean. Okay, so hold on. This is. Okay.

We could even call separate elements partial credit.

No, It. Well, I mean, the obvious thing will be some elements, but that's rather silly.

Any.

I think that suggests they're very few. I mean, It's like you. You can have as many as you want.

We could just call it element wise.

But the.

There's a function called contains any. Right.

Is.

Is there an analogy there? Like all those contains this stats.

Yes, but. But here it's counting.

So maybe different.

Well, maybe. Maybe we say count elements.

Count any.

No, but what about this? Because it is counting and the other ones are not counting. You see what I'm saying.

That I don't love that count elements.

Well, but you. That's what you're doing, right? The. The score is the count of agreeing elements. Correct.

Right. Well, what other words can we use of the agreeing?

What's that?

It's the total of the score of the agreeing elements.

I mean, those kind of say what they're doing.

I mean, that's. That's an accurate name. It's just a little long. I don't. It's fine.

I think it's better to have it long and accurate than to be incomprehensible. All right, that's what I suggest.

Okay. There's a typo on all elements orderless.

Yeah. Okay. Should we move on?

Yep. So, yeah, we had. In the. In 13, 3, we had arithmetic result, polynomial result, and calculus result. We're adding algebra result. That's the previous subsection. I don't know if you care to look at that. It's pretty consistent.

So tell me what class this will be useful for?

Algebra.

Okay, so the calculus one is useful for a calculus class. The algebra one is useful for an algebra class.

That's the idea.

Okay.

Right.

Okay.

The quintessential problem for this is, you know. Yeah, here's a polynomial cell for X.

And.

Right. And it allows. So the student can, you know, it allows evaluations in this answer that are. Don't contain things like self. Right. So.

So it could say. They could say in this case, four over A plus B plus four over A plus B.

Exactly. And that'll be more correct because it isn't. Doesn't differ by algebra. It only differs by arithmetic. Right. Okay, so that's the idea.

What about A squared minus B squared divided by A plus B versus A minus B?

That would work too. Right. So just can't have things like solve reduced. Fine. Instance.

But I mean, is that really what people want? That. I mean, you know that A squared minus B squared. I mean, there are things which. What the heck. Okay. And now I say, let's say just together of that. See what I'm saying? So which functions does it do and which does it not do? Because I think that one. Many people would not mark that as a correct answer. I would think. I mean, I'm far Away from that world. But I would think that would not be considered the right answer if this is what they're looking for.

Right. So if it doesn't contain a symbol on our blacklist, it basically does equal, you know, double equals and wraps it and simplify. And if it's true, it gets marked true.

Okay, but I, I, I, I mean, I don't know. You, you have to teaches algebra whether what, what they like for this. Because I was not to me, this would not be the right answer, this thing up here yet if I go this, you know, double equals A minus B and I say simplify, I'm pretty certain that that will say true.

Yeah, right.

Okay, so I don't think that's. I, I, I don't think that's right. I mean, I think that, that perhaps there are certain set of functions it doesn't simplify, have options for which functions it can use.

I don't think so. It takes assumptions, but I don't think it takes, I don't know.

Well, I think it does take methods. I think it does take some method options.

Okay.

Now transformation functions.

Oh, okay.

You got to ask somebody who teaches this stuff. I don't have the slightest idea. Okay. I just don't think this is right. Am I making sense? Who have you asked to actually teach us this stuff?

Well, we have the Edtech group and we've talked to them.

Okay.

I don't know how many of them have middle school math teaching.

Yeah, we need somebody with specificity to tell us what they would do in this case. Use this case as an example. What would they do? What would they expect?

I mean, we have comparison methods that would prevent that. Both.

I understand. I want to know what. Whether this one, which you're proposing as the default algebra classes, is right. Okay. I don't know. I mean, you're asking me. Okay. I simply don't know. And I know that I don't know. So you got to ask somebody who does.

Okay.

Do we have to, can we get. No, you got to do that offline.

Yeah, yeah, we, yeah, we don't need to do it in this meeting.

Okay. I mean, one thing I want to say is that I do think that these comparisons have got to be fairly stable. I mean, we can't say, oh, we just changed a version, and then somebody goes back and remarks exam papers from two years ago and all the scores change. Do you understand what I'm saying?

Yes.

So there's a certain argument that we should make sure there's versioning information in these Comparators, Am I making sense?

Yeah.

Again, you want to ask somebody who's more of an expert on how this works on the ground, in the field, et cetera. But my feeling is that we want something where it's like algebra and it's like version, you know, comparison, you know, assessment version arrow 14.0 or something like this. Do you see what I'm saying?

Yes, I understand exactly what you're saying. Yeah. I mean, do you think we should just name these things like algebra result 13 0?

I don't think we should. I think we should, we should have a separate thing for versions because some people will be kind of sloppy. I mean, look, if people, if this is for, you know, it's, it's part of the documentation and part of the, the things we have to train people who are using these for like high stakes exams, et cetera, et cetera, et cetera, because there's no, you know, if it's an intermediate look, what do I know about this stuff? But my impression is if it's an intermediate thing in some class and you know, in different years it works differently, it's fine to get what I'm saying, if it progressively works better because we have more sophisticated, Finer grained control, etc. Etc. Nobody's going to object. The thing, the scenario that I'm concerned about is that there was an examination that was done and somebody tries to rescore the exam and they get different results.

Yeah, I understand.

I, I don't know how important that effect is. Okay. I don't know, maybe people, I, I don't know how long they even keep the exam documentation. Maybe they, they probably throw it away after some period of time. I just don't know. Make sense?

Yes.

So you, you've got to have somebody who is actually an expert in how these things are used on the ground. Right. And there's people in the, you know, you know, ed tech group who do know this stuff, who probably should have been in this meeting. Okay. But I'm telling you, my, my version of this, I, I think you need to look more carefully at this. I mean, can we go live with something like this? Probably, yes. We didn't have this before, so people are not going to immediately pour it into lots of things. I mean, it probably should be experimental.

Oh, it's experimental, sure.

Okay.

You may also want to field tested with actual students as opposed to not just teachers. Not just, you know, like have people, you know, algebra is high school level, right. So you want to have maybe some high school level students Try this out and give you feedback.

Let's do it through people who are actually dealing with the on the ground deployment. Okay. Which includes things like that. But you know, we're designing this in a very theoretical way here.

Okay. So the next thing here is a feature that we do know is important from people in the EdTech group and that's, you know, advanced tolerance support.

Okay.

So we have two different new math comparisons that support advanced tolerance. That one of them, this one that we've so far called numerical approximation, just adds tolerance to each numeric value in the teacher's an. In the, in the answer key. And the next one that we called engineering result, only adds it to, you know, inexact values in the answer key.

Okay. Oh, come on. Okay, Okay, I understand. What's it going to do by default? Presumably it's going to use precision control. But how does it know what the precision that's being asked for. What the hell.

The tolerance. Oh, that, that's going away. That's an implementation transient thing. It's not going to be okay.

But there's a tolerance number that's going to be given.

There's a tolerance number that's going to.

Be given and that number might or might not be applied to exact numbers. Right.

That number, that engineering result, it will not be applied to.

Exactly. I understand, I understand. So that's what I'm saying is that the choices, you know, now normally when we ever do a tolerance for equal, simply if it's an integer, if it's an exact integer, then, you know, there's no tolerance ever. Right.

What's the case if we do, you know, one. One point double equals one gives true. Right. There's a, there's a precision.

I know, but I don't understand why in this case here they don't give the comparator as, or the, the thing that they're comparing to as inexact numbers and make it so that whenever there's an exact number, well, that's going through.

Fit, which automatically numericizes things. That's why that one.

It'S there. Okay, but, but I don't understand. I, I'm just saying there's an answer. Right, right. That. That is supposed to be the right answer. If that right answer is, is numerical, then do a numerical comparison. If that right answer is exact, then do an exact comparison. And that seems to me much more intuitive than some weird option that decides whether to do this. And then just say the rule is if it's exact, it's an exact comparison. If it's inexact. It's not.

Yeah, actually it's what it's doing in the first method, so.

Yeah, yeah, I understand. But I'm saying, why isn't that just the way it works?

I don't understand what you're proposing.

So I'm proposing getting rid of these options and I'm proposing simply telling people, if you want it to do an inexact comparison, don't say the answer is one, say the answer is one point.

How do you say what the tolerance should be?

I'll have a tolerance option. But the tolerance option only applies to inexact numbers.

Right. Well, we can't use a tolerance option with some of the other methods. It just. We don't have a way to implement it with every.

I'm sorry, which other methods?

The. We can't do it with anything that is going through equal. We can only do it with things that are going through match which. So calculus result and algebra result would not support it.

Well, I don't think anybody wants an exact algebra and calculus results. I don't think anybody wants them.

Right. But we need, we need comparison. We need name comparison methods that support.

But look in the, in the algebra and calculus case, if somebody gives in exact numbers, we should just do what equal does because it's a corner case.

But what if they give a tolerance?

Who gives a tolerance?

What is it the teacher? If the teacher says, yeah, well, I.

Think it's not something they're going to do, so I'm not worried about it. I mean, in other words, if they give. If they say if you type x minus one point and if you type x minus one point. You know, I just, I just don't believe it. I don't think it's a case we need to worry about. You know, one point. This will mark it as. Right. Forget it. No, they don't need it.

We definitely want to have this where you can specify exactly what the tolerance is on values.

But no, you don't. Not for algebra.

There are problems where we want it.

Absolutely. Which are numerical problems, not algebra problems. Right.

So we need a name, we need a name for the comparison method that accepts tolerance for something like an engineering problem.

Okay, but I don't understand why we're not just saying tolerance, arrow, whatever, a number.

That is what we're saying. Yeah, that, that is just. That is the transient thing that real won't actually be in there.

Okay, so the tolerance then will probably be something like, you know, I don't know whether it's, you know, significant Digits. I mean, there's, there's any one of these things. It matters whether we're talking about significant digits or absolute or whatever.

Yes. Right now it's all absolute. But we also, we also have, are going to support scaled for relative, but we're not doing significant digits.

You know, we used to always talk about when we're doing designs, it's like a boat and it's taking on water and you start running the pumps because more and more things are like, oh my gosh, we got this issue, we got that issue. Run the pumps faster. There's not the right way to do this. I mean, in, you know, possibly the right thing to do is, you know, what, what if you've got three numbers and one of them is supposed to be this tolerance and one is supposed to be that tolerance.

We have that. Right. The discussion point at the very bottom of the notebook right now.

Yeah. Okay, well, how is that supposed to work?

So you, they, you would mark up individual values with a wrapper. Right now it's. Right now it's Jonas, come on, come.

On, come on, come on, come on, come on.

And all this will be written through the authoring notebook where, which nobody's going.

To understand by the time this is being written that nobody's going to understand it. All right, so look, let's talk about what's actually needed here. Okay. It does matter whether they get, you know, 18.23. Which, how many digits are they supposed to get? Right? Let's say that the answer is supposed to be 18.23 meters per second. What happens if it's in different units? Do we deal with that? What happens if they say centimeters per hour?

Right. So if the answer is a quantity, we have good support for units in the quantity comparison method. We have a comparison method called quantity that works on units and it already takes a tolerance. It's unambiguous. But when the answer is a formula.

What does tolerance mean? Is the tolerance given units?

The tolerance is given in units.

Okay.

It's absolute. So it's an absolute tolerance and conversion works on. Yes.

Okay. Let me point out that a randomly generated problem, absolute tolerance may not work.

I understand. Yes.

Okay.

That's the most common thing in just a naked number on the right hand side of the tolerance option should mean absolute.

Yeah. Although I think, you know, significant digits, I. E. Precision is an even more common thing. Get it right to X number of significant Fig.

Again, I don't ever taken a question, a quiz where they, well, what do they say? I mean, they, they Say, you know, within, you know, within X, get the value, you know, within this many. And they mean an absolute thing.

Really? Well, I mean, I'm out of this game because I've never taken a multiple choice. Like, yeah, it was a thing when I was a kid. Wasn't yet a thing. So multiple choice wasn't yet a thing.

I mean, I guess they do say express to so many significant digits. That's that certainly.

Do we have anybody from Wolfram Alpha team here? I think we just have the wrong people for this meeting because we need people who've actually, you know, who are actually doing this stuff on the ground. Because I don't know what the right thing to do is here.

We can ask them outside of this meeting.

Yeah, we can continue this with a larger group.

Yeah, I think that's what we have to do. I just don't know. And what I would do in the first instance is if it's exact, make it exact. If it's inexact, use a tolerance. But I don't know how you do that when there are a list of numbers and some of them are a 0.01 and some of them are a thousand. How do you use a tolerance then? If you've got a list of numbers with very different magnitudes.

Right.

So what do you do?

Well, we. I mean, one thing we've. We've supported within these, like, engineering result, what we're calling engineering result, numerical expression is you can do tolerance goes to scale the. Yeah.

Oh, that's horrifying. That's absolutely horrifying. What's wrong with working precision? That's what we use everywhere else or something like that.

I mean, that's a little bit different. That's a little bit different. Right. That's the precision. The computation should be happening, not how things should be compared. Right.

So it's comparison precision. But that's something we don't have.

Right?

We have that in equal or something like equal.

No, there's no. As far as I know, equal is hard set to like a machine precision.

I mean, if this is done on a test, the students would be calculators. Right. I mean, this is like. This would be deployed as a web page or something. Is that right? That like as a homework problem.

Yes.

And as far as I know, all high school students in the entire world still, you know, use calculators for this sort of stuff. So that means that, you know, like the 0.5 meter per second tolerance is, you know, that's probably way too big. Right. Like a calculator could get this right with, you know, four, five, six digits of precision. And that's probably, you know, expected, like it should at least have.

Calculator usage has been declining largely as a result of Wolfram Alpha and related things. So just for what it's worth.

Yeah. I mean, my students still use scaling calculators either on the phone or an actual calculator when they're doing algebra and this sort of, you know, physics problems. So.

Yeah, okay.

So if the problem is given with like, you know, 18.23, then I suspect that the answer also has to have at least, you know, those two digits of precision. It may be, you know, one less or one extra, I don't know. But I don't think it would explicitly state that 0.5 meter per second thing in any.

So what you're saying is the digits, if they give, if the teacher gives the answer as 18.23, they're expecting that the actual result given will lie. So is 18.22 an acceptable result or not?

Well, no, because that's not the answer.

Right.

The answer will be something like, you know, 25. I don't know what the question. I try to understand the question. I don't even understand the question.

But let's not worry about.

Suppose the answer is, you know, 512.18 or something. Right. Like that.

That's.

I imagine that, you know, the answer will want to be with some, you know.

Okay, what I think we should do.

I mean, the question we're. Sorry, the types of questions we're considering here is where the answer is 512.83 times X plus A over 5. Right.

You know, that's, that's, that's a big weird corner case.

But I mean, the case where it's just a number isn't complicated. Right? We have had that.

Yes, it is complicated. Yes, it is complicated to do it. Right.

I think tolerance, the tolerance option is unambiguous with the tolerance. Yeah, but it's a, it's a bad thing, the tolerance. I'm pretty sure it's teachers.

It's a bad way to do it. It's fine if you have a single number. If you have a vector, for example, and the vector is this. Right. It's a bad way to do it. Sorry.

So on vectors, we do the norm of the difference.

Oh, for goodness sake. And no teacher is going to know to compute, to specify the tolerance for the norm of the difference.

You know, if they're saying the comparison method is vector, then what. Right. If what hasn't done.

I know, but if they think it's element wise, they think it's element wise. What is the comparator? Because they say I want the kids to get it right to within, you know. You see what I'm saying?

Right, right, right. Then what we've had for a couple versions is just absolute.

So don't we have a round for that or whatever like.

We do, we do, but.

Okay, no teacher wants.

Who will really know this? It's Michael. Because this is the whole. Michael and Jose are the ones who are really going to know this. This is subtle. This is not something where we're just going to hack it through.

A round is not part of the solution, I don't think at all.

Well, I'm not certain about that. I'm not certain. I think, I think we could imagine a solution for very persnickety teachers where they give an around specification or an interval specification and they want the student.

To be within a certain number of standard deviations if possible.

Or no, they want them to be within the plus minus in the around.

That's not a plus minus, it's a normal distribution.

I'm aware. I know. Right, I know. So it's a way, it's an alternative to giving an interval which is, it's a more convenient way to specify something like an interval.

But is a normal distribution.

Right, I'm well aware. I know.

So, so the question is, okay, that.

May be around just the wrong thing, but like there's other things, right?

Like I don't think, okay, there's interval and centered interval and so on. That certainly persnickety. Okay. But again, Jose has worked on precisely this for around, etc. Etc. Etc. Let's get him involved in this. Okay, let's just not, you know, break this for, you know, whole generation.

But the computer science, by doing this in a sloppy way, the computer science center line design might not be that education center.

I don't care. I. I don't care. So let's get in this meeting. People who actually understand the math of this and the people who understand how it's used in education and we have neither of those groups here, so we really can't do this in this meeting. Sorry. I mean, this is going to be subtle and we're going to end up with probably several options and it's going to be quite subtle and we're going to end up with some reasonable default that people will use and that will do the right thing most of the time.

And a teacher wouldn't Necessarily need to know.

Teacher won't know squat. Teacher will most of the time just.

Use the default for which you get the default.

Right. And my guess is that a very important piece of information in the default is. Is what the teacher typed as the answer.

Of course, yeah.

Yes, but the teacher might have typed this as the answer. And what does that mean? How much precision does that have? See what I'm saying? Because that has that amount of precision.

Right?

Okay. So this is quite subtle, right? In other words, when they say n of 1/7, do they mean six digits? Do they mean 16 digits? What do they mean? See what I'm saying? So let's, I, you know, I'm afraid we need another meeting with the right set of people here for this. Sorry. I mean, again, I would recommend that you meet with these people separately.

I'm telling you, we meet with EdTech people all the time.

Okay, but I think you need Michael and Jose as well. I think you need those people. I think this is subtle. I think, again, Michael knows all these things that come from all the standards that NIST and so on have put out about what significant figures actually mean. Let's use all that information and all of that tradition, so to speak, here, okay? And not kind of hack it with some random thing. It is subtle. And, and there will be corner cases now, again, people will use the default 99% of the time. So let's get the default as right as we can. And the number one piece of information is, what did the teacher say the answer was? Am I making sense?

Yes.

And, you know, in, in a more.

Complicated, I mean, I think equal. This is all built into equal, and that is what the default is.

Okay, well, let's ask these guys because I'm not certain that that's all. That's the way everything works. If that's the default, then we don't have to have this discussion. The only discussion that have is does one point mean the same as one?

No, that's not the only discussion.

Well, no, in that case, where we're.

Just saying we use the implied precision is not the only thing teachers are going to want.

Okay, well, what precision are they going to want? And how do you.

They're going to want to specify tolerance.

No, they're not. No, they're not. Not in the only. If it's a single number. If it's a single number they want to specify, how do you think they think about it? How do you think they're going to think about it? If the answer is 18.23 how are they going to think about what the student could enter? They're going to think about it in terms of the characters, the student types. I think.

Yep, they are.

Or are they going to think about in terms of the math? In other words, if the student enters.

And they're going to think, I want the student to put in two digits which they're going to think means a hundredth. So they're going to want to put a hundredth somewhere. They're not going to want to put in two.

Well, no, the question is because they're.

Not going to think about it.

Some teachers would say this is not the right answer. That's too much precision. I would say that.

You would say that? Yes, yes. But I don't think that's.

Well, let's find out. We don't know. I think a lot of people, if they don't currently mark this as wrong, they should.

Yeah, I, I, I, Okay, I don't know either. But I think students know how to round, you know, a digit like the last, you know, like.

Well actually I've seen lots of examples where they quote calculator where they copy every single digit from the display of a calculator.

Yeah, yeah, but, but like if wrong, if the prompt asks for it, you know, like give two digits after the decimal point, they should know that it's the default.

The answer should not be give something like this. But by default if the, if the input contains numbers to two digit precision, they should not give this by default. Even if their calculator or you know, WOL language said that. Am I making sense?

Yes.

I consider it the wrong answer. And that to me is something where, you know, we will need an option that determines whether you think that's the wrong answer or not. Okay. I'm telling you this is subtle. Let's try to get it right. Okay. I noticed on our live stream, James is asking is there a negative marking function? In other words, if you say this, you get minus.

Yes. That's been in for the. Yeah. In the first option to assessment function you can give an arbitrary number of values that are compared against and each one can have be correct or incorrect and give an arbitrary score. So you could say one is negative one point or negative 10 points and so on.

Let me just make a comment about a generalized comment here. I mean the vast majority of people who are building edtech software don't have a clue about all of these issues, all these subtle issues about numerical analysis and so on. We do have a clue. Let's not get it wrong.

That's a good point. It is our.

Yeah, we actually have a bunch of world experts in this, so let's not not use them to do the best we can to get this right.

It's a good point.

Okay, so we need to come back to this, but you should do a separate meeting first and then we'll come back to this.

Okay.

And you know, I'm telling you it's subtle and I think we're going to want to have things like sent an interval numbers. We're going. You know, some people who are doing some very fancy, you know, high stakes exam might well want to actually write in sentinel interval type things. But people who are just sort of, oh, I, you know, I think the answer is 18.2 or something. Just give that and et cetera, et cetera, et cetera. Okay.

All right, so this next part we can do if you want. It's not really difficult. Well, if you had run anything else previously in the notebook, it would have loads the packlet and you could run it.

So but we'll just load the packlet.

No, no, no, just run anywhere it says question or just do question object shift enter and it'll load the pack lit and then you can run it. All right, so this is very much not design review stage. That should have floated. You did the packlit install the tap, right?

Yeah, I did, yeah. Okay, well, I mean I do this one then.

Oh, this looks great. 18. Yes. Yeah, maybe there's a problem with that. I can, I'm happy to demo it if you want.

All right, briefly demo it. Yes, I grab the sh. Sure, you grab it directly from me.

No, it won't let me until you give it up. So. So this is, you know, rebuilt question authoring notebook.

Why the heck is the highlight color light blue?

This is not. Design has not gone ahead of pass to this at all. So. Okay, so the main thing that I want to show here is the complete change in ux. So instead of having like an area here with a bunch of input fields that had a whole bunch of complications to it, if you want a GUI editing thing for a specific question, you hit this and it opens a secondary dedicated notebook just for one question and you, you know, and you fill it out and, and then when you're done, you can test it within this notebook, you can tweak it, you can add random variables, make it a question generator.

Yeah, this is a good idea.

And then when you're done, you hit finished and it, this is a little slow now, but it collapses back into here.

And if you want saved or is.

It, it's just stored in this notebook. So you would want to save this Untitled 60 to save it.

Okay.

But you can re edit that.

You can. If you hit this button again, it'll open it again and you can go back and forth, add infinitem. And then we have this toggler here, which I made up this icon. It's not right to switch to code mode. So if you want to just write the code, it's infinitely customizable improvement.

It got over overbuilt.

And then you hit the toggle again and it opens the notebook again and you can keep going. Right. So, and you can, you know, there's tools for markings. You know, this A is, is, you know, going to be filled in by a as a slot. It's not literally a sort of thing.

How did you make that slot?

It's an inline cell. Is that what you mean?

No. How did you type it right now? I wasn't paying attention.

Oh, I hit this button, I highlighted it and I hit this button in the toolbar.

Sorry. What does that button say?

It says select for formatting, which isn't great. I don't know. In Danny's version of this that inspired it, it was called something with something with math in the name of. I didn't love that either. It means that it'll be formatted. What do you mean it'll be formatted? It'll in the question object. It'll, it'll, it's not formatting.

It's computing a random variable.

To put it well, well, it, both, it does both.

So of course it formats it, it shows the, the value.

Yeah.

Right. I mean that's basically just like in a template notebook, if you start a new template notebook, there's a thing exactly like that. Okay. I think it's just an expression. Template expression. Right. It's, it's the thing.

Right, right.

That evaluates that. And we could use the same visual, another metaphor there.

Okay.

Right. Because that's going to be evaluated on notebook load in the case of template notebooks. And this is the same thing will be instantiated in the same way.

Right. It's, Yeah, I, I, I'm not defending these terms. Yeah. We could certainly call it something like template Template expression.

Something like that.

Yeah. Template expression might make sense. Yeah. And then when like it. So we have these, that map to comparison methods where, you know, I don't.

Know, by the way, whether template slot would be useful for these things. I don't know. I mean, I Could imagine something where if the person's name is Bill, give them this and if it's Bob, give them that.

Right. I mean you're going to need to write code in that case for sure.

Yeah. I mean maybe it's a silly case but I mean I could imagine. Yeah, it's a silly case.

Right. And you can use the same, you know, you can use the same tools in the answers like that makes more sense when there's a multiple choice. Sorry, this isn't multiple choice. So like if it's a, if it's a multiple choice answer, definitely. I don't know why this didn't turn gold. It should have turned gold. There's still bugs but.

Okay.

Yeah. So then you know, some of these have more options that you know, you can get more fine grained.

Well, each one should have some kind of mouse over. I mean, that's okay.

Yeah, yeah, yeah, yeah. We need more info buttons, we need tool tips, we need lots of stuff. I, I know I was really. Well, I like the General 140 but it's just not going to make it in time for the freeze because there's a lot to do and I just wanted to sort of show the general UX of the secondary notebook and toggle.

Yeah, I like it. I think this is good. Great improvement. I think it uses the learnings that we have from things like the function repository that you know it's going to look easy when you start typing something or nobody will ever start and it's.

All implemented on that same definition of the client that we're using for the resource.

Oh, that's nice. Yeah, good stuff.

Okay, great. If you think this is the right direction then I think it is certainly plow ahead without really any other comment.

Okay. The other thing that those numerical precision things, those we've got to. Yeah. Okay, we should wrap up here. All right. But you, you will, you will iterate on that. Okay, thanks.

Yes, we will.

Okay.

Yeah, thank you.

Bye.

Bye.