Live CEOing Ep 685: Language Design in Wolfram Language [Around, EchoFunctions, and More]

Sam. Hi, everyone. Welcome to another orphan language design review. This is part of our Language Completeness and consistency series and we have a number of topics. First up, this is a horrifying one around and min and numerical order. Right. Okay.

So we think we've untangled this at least the symmetric case and. Right.

The.

The question it was, you know, when, how, you know, how do we sort these, et cetera, et cetera. And sort of one of the issues is, you know, since these are all probabilistic, you know, when do we say that they're actually different?

Yep. And so the question is, is a one, you know, a one equals. Equals a two? Obviously, for this particular symbolic thing, it's just going to sit there, but actually it will know.

So the key new idea is evaluate those two and compute a one minus a two. So the key new idea is that we will consider two arounds equal if their difference is not significant, meaning their mu1 minus mu2 plus is smaller than the uncertainty. That's it.

Yeah.

Or proportion. Right. So now we can continue. So we have to worry about these 24 functions. Numerical order, equal, same Q and order and min. And these are the proposals.

Right. And basically order and same Q we.

Don'T worry about because I'm not sure I agree with you on that. But we'll get to that in a minute.

Okay.

Wait a minute. Assign plus minus one. What does that mean? Oh, numerical order. I. I got it. I'm sorry, I wasn't reading properly. Okay. If the ABS of the difference. Why the heck are we doing this? Is less than twice the standard deviation.

Yeah, I mean, I think the idea is some multiple of the uncertainty is going to be like a tolerance with which you compare the centers of these around objects. And so it's like the tolerances that we use for floating point arithmetic. Equal we allow seven bits of difference. Same Q, we allow one bit of difference. I think Jose could correct me if I'm wrong.

That's correct.

It's doing some multiple of the uncertainty and taking specific factors. One for equal and one. Well, you're not using a different one for same queue, but that's a different story. Same Q for you is purely structural, I guess, on these.

Yeah, yeah, yeah. So we are imitating what numbers do here in the sense of having equal being more tolerant than numerical order.

Right.

And the question is, is two a good factor? So, I mean, obviously it's going to need some kind of.

Well, this is like two sigma. Is that, is that right? Is that how you think about it?

Exactly. Yep.

There are some plots below in the statistical discussion section. Right. So the key point is what we just basically said. Right. The difference is, you know, significantly different from zero. And, you know, Jose and I are suggesting we do 1sigma for numerical order and 2sigma for. For, you know, less or for equal.

I guess so two, that's, that's 5%. Is that right? What's the, what's the confidence? Is that 95% confidence?

Yes. And one would be 68.

Yeah.

So these curves were what we were suggesting the previous time, in which there was no notion of equality. It was always either plus one or minus one. And now we have this intermediate region, the green region, which is equality. And so you see how now greater and less do never overlap.

Which is good.

Exactly.

Yep.

And so this is just scary.

I mean, this thing about having, you know, it's a bit arbitrary, but of course, it's been arbitrary for years. What we do with finite precision numbers, and it has to be.

Yeah, it's a judgment call where we put the boundaries. But now the new thing is that there is a region of equality, and These are the 1sigma, 2sigma, 3sigma, the red lines.

Is there any way of. For our usual tolerance, there's no way of adjusting that. Nobody's dead in 30 years.

Some people have cared. And there are internal variables that they.

Can.

You know, fiddle with.

Right.

There's internal dollar equal tolerance, I think, and same cute tolerance mark you, but.

You'Re the most likely person to have run across these kinds of things.

I wrote them.

Okay. Okay.

Then, you know.

There are situations when you need to adjust the tolerances. Like in hashing, hashing of numbers, you need to have a zero tolerance, otherwise you get like a floating point drift of things. When you're doing like counting bins and stuff like that. And then if you do one method using a hashing and one method without hashing, and it all gets. It gets to be a horrible mess. It gets to be method independent of the quadratic method and the hashing method and everything. So you, in, in any given meth, you need to fix the tolerance of things that you're working on. And then you need to set that kind of visual universe and then you need to kind of work in that.

And so, so how's that going to affect this, this stuff here?

Well, it's. It's a similar idea to what Jose's and it, I guess, are proposing here. I mean, it's. We could argue about whether the factor of two there or one or three or every is Too large or too small, what it should be. It doesn't have to be a whole number. It could be like we use one bit, which is 0.3 decimal digits. The same Q tolerance of floating point numbers. So, I mean, but, but the concept is very similar to what we do for floating point numbers. So I think it's the kind of. The right concept.

Okay, fine. We just need to kind of decide.

Which functions, what the tolerances should be and that sort of thing.

Okay, so. But we have this notion of. Oh my God. What are these called? System. Oh my gosh. What are the system settings? Low level system settings.

They're internal. I mean, they're internal things.

I know, but what's our usual wrapper for internal system settings? What's it called?

System options.

Yes, system options. Yeah, yeah.

I don't want to make it as public as system options because these really are things that you should put in a block so they get reset within a specific small computation. System options are more things that you set globally, like the number of threads or something. Like you just work with them in your session. But these are very local, private things that you want to do on a very specific function.

And you don't think these are end user. These, you don't think typically this is, this needs to be end user. I mean, the difference.

It's scary. The code is not prepared for everywhere. So that's why.

Let me make a point about this. With numbers we have control. You know, if you have. What happens if you have high precision numbers, what happens to this there?

You have essentially a very small interval, so very small uncertainty. So the uncertainty is around the last digit there.

Okay, so it's still a one bit uncertainty, is that right?

We do have an example of that in the notebook, right? That's what we're seeing right there. Right?

Yeah.

But Stephen's using very high precision, which is. Which would be a tiny, tiny interval. Your intervals are almost the size of the number there in your examples in the notebook.

So hold on, hold on. I'm just trying to understand. We don't do anything different with high precision numbers than we do with low precision numbers. In terms of the last bit is ignored for comparisons or whatever else. We're doing that correct?

That's correct, yeah. Well, Order, for example, takes into account the precision. So it will look at, you know, the more precise one. It has a note. So order isn't purely structured. And that's one of the questions I have for Itai and Jose. So order the floating point numbers actually has some knowledge about the numbers. It's like doing an infinite precision kind of comparison where it's like considering all the bits and then it's kind of comparing the scales of them, the precisions of them and stuff like that to, to determine which one comes before the other. So it's not doing a subtraction, but it is doing analysis of a floating point number.

Right, but it's based on the center, isn't it? Mark of the interval?

Yeah, yeah, it's based on the center of the interval. Exactly. Yeah.

But that avoids having a sort where you have bunch of things and, and you wind up with the first one much bigger than the last because. Yeah, because of drift. And with your precision I could, with your definition I could put together an example of a list that would sort in the wrong order because of your, your tolerance in numerical order.

Yeah, that is kind of a problem. And that's another thing with it, with Italian and Jose's proposal then numerical order proposal is having a non zero tolerance for the. I think you're proposing one. Yeah. Which kind of is a little bit scary to me.

Wait, wait a minute. Your point is. Okay, hold on, let me just understand this. So right now with numbers we have a comparison criterion that is based on a fixed number of bits in a number of any length. Those are allowed to be fuzz. Is that correct?

But not quite.

But not.

We have a comparison. We have a comparison function that takes how many bits to allow when you compare the two things and that can be zero bits. That means it absolutely needs to take the entire number. And that's what order uses. Or it can be numerical order that says go all the way up to where the interval of uncertainty starts. That's like a kind of zero tolerance. Sorry, order would be like minus infinity or something like that. And then we have equal, that has like seven bits. It allows like seven bits of the, of the number to be different. So like a precision of 2. When you subtract them it will allow numbers that are close to each other too.

Okay, so. So it's seven bits still here. Even though this is many more bits.

When you do equals, equals. Yeah, that's. Yeah, that's correct. But it's seven bits in the difference of, of the, of the, of the two numbers that you're saying.

So if I, if I do this to the power 60. So this is plus 10 to the minus 60 times 7 bits is. I don't know what, you know, let's say 32 log 2 of base 2.

Of 10 times 60 or something like.

That it's percent 409 equals. Equals 410. Yeah, I see. Okay.

Yeah.

And if you make that like a little bit more there, I mean, you might be just on the boundary with 32. So. So, you know, make it like 128 or something, then it should be different.

So. Yeah.

Right.

And I guess finite precision numbers suffer from equality being not necessarily transitive.

Yes, that's right.

Which you know, and that's a problem.

With sorting.

You know, which our proposal also suffers from. But I don't see a way around it.

Yeah, yeah, but I mean, explain why you want numerical order to be fuzzy for these versus the way it is for numbers.

Well, I guess, yeah, because numbers. It's a zero tolerance.

That's what Rob, it's not true. Look at the example on the left where it says 10:1 and 10.11 and those are still considered equal in numerical.

Order because when you. That, that 0.1 is an insignificant bit. So when you subtract them, you've got a tiny, tiny precision there, Jose. That's the thing. Your interval is so wide in that number that it's effectively only one digit.

But, but the number still remembers the 10.1, so the input form of that number still remembers. So I think there is some tolerance.

If you take X minus Y there of that 10 and y 10.1 back to, back to 1. If you do X minus Y, you'll get an input form of that. You'll get a zero. Yeah, if you look at the input form of that.

Yes, but, but the input form of, of Y is still remembering the point one, if I remember correctly.

Yeah, but, but.

So, so in a sense something is ignoring that, that point one.

God, this is weird. I mean, welcome to floating point arithmetic.

But yeah, if we, if we want to have numerical order with no tolerance with arounds, that's also fine with me. I mean, the thing is, is to have this, some sort of choice of the levels of tolerance that we have, we want for these functions.

Right.

And. Yeah, right. And I mean, personally I think it's quite weird if just. Well, I guess part of the question is what do we do about min and. Right.

And I think, but min will just be based off the choice that you use the numerical order. That's all it will do. It will just take whatever the result of numerical order is if it's one minus one or zero and it will do that or either swap them or give you the first one or whatever. So min is just a consequence of numerical order.

I Think. Yeah. Let me just understand the point of contention. So we are agreeing that equal is okay. We don't know if this two is the right number, but it might. Yeah.

Yeah, maybe.

Okay. All right. So we're agreed on that. So the question that remains is numerical order. Mm. And the argument that's being made is the numerical order criterion here, Which is. Wait a minute, is this the right way around or should that be a half for 95% confidence? If the difference of the means. Surely that should be a half there.

No, that wants to. Because if you have two things that differ by a little bit, right. They're equal.

I'm sorry, I'm confused. This thing is less than. This thing is saying this quantity X.

Right. We're saying that we. That then it's the, you know. Right. Then if this is grit, right. If that quantity is greater than two sigma, then there's a 95% probability that that difference is not equal to zero.

Oh, I see. I see. Okay. Yeah, I get it, I get it, I get it, I get it. So it's like the. When that thing on the left is getting really big, that's when it.

Right. That's when we can just say. Oh, we base it based on the centers because the probability that they're actually swapped. Because these are Gaussian distributions. Negligible.

Yep. Okay. Okay. So the thing at issue is a numerical order case. Okay. So what are the arguments in. So what is. Mark, what are you arguing for in that case?

I mean, we in the arbitrary precision code use zero as the tolerance. So when you subtract the numbers, you must get a floating point zero.

Yeah, but I'm actually looking at it, Mark. Numerical order is different than order for this.

Yeah, yeah. Your order is going to use even all the insignificant bits. It's like minus infinite position. So numerical order is going to. Numerical order does this. It exposes a few hidden bits, does the subtraction, and then fiddles around and checks if the precision of that thing is going to be zero or not.

And.

And checks if it's going to be.

Like a zero precision zero test is analogous to the comparison with the. With the standard deviation, is it not?

Yes. Yeah, but it's with zero. With zero tolerance, right?

Well, yeah. Right. Because you have hard borders. Right?

Yeah. I look at this directly analogous. I mean, if I say precision P, I'm losing P bits of precision. That means I have two to the minus P. You know, if I'm. What is. I'm not correctly seeing the math of this, but I Mean, basically, you're saying it's a box whose size is determined by the precision. The, you know, if the bits falling off the end of the precision thing. If it's precision P, any bit that is. Yeah. It's two to the minus P, isn't it? That the boxes of width 2 to the minus P, give or take, if.

We'Re using P bits.

If you're using. If the number is accurately, a precision P, there are nothing smaller than that precision. Any bits beyond the P bit are undetermined. Therefore, the size of the box of indeterminacy is 2 to the minus P.

Right. Well, I'm just pointing out that when. Normally, when we say P, you know, precision P, we mean P decimal digits. So that's why I'm being.

Oh, I see. Okay.

When you're saying P, if the precision is P bits, then.

Yeah, yeah.

But I mean, internally it's all bits. So Stephen is just thinking, like, closer to the computer, which is how it's actually implemented, rather than level hard borders.

2 to the minus P. So what you're saying is numerical order, if it is not within those hard borders. If it is within the hard border, it thinks it's equal. If it's outside the hard border, then it does its, you know, whatever ordering it's doing. Right? Yeah. Okay, so the question is, what is the analog of the hard border in the case of a round. And the problem is around doesn't have a hard border because guess what, it looks like this. You know, it's Gaussians. Right. So what do we do? You know, in the case of numerical order, we, you know, these things will be. Will be step functions. And, you know, I have to say I think it's reasonable. If I'm not being confused, somehow it thinks it's reasonable to use a criterion like this in the case where. Because there is no hard border. Now, the question is.

Yeah.

The question is what is the constant in front of that square root? That's the question.

I know. Okay, fine. Okay, so.

So I'm fine with decreasing it and having a half or something like that, to be more strict.

Which means that you have to be in a closer.

Right.

In order to go ahead.

Right.

That.

That, you know, the probability of overlap is even. Even greater than.

Right.

So the centers are much, you know, that much closer than the standard deviation. So, right. There's. You really, you know, at that. Right. At that point, the probability that you quote, unquote, have the wrong.

Well, let's compute those probabilities. It's just an earth of some kind. Right, right. So what is the actual function of. Well, this isn't the same as like the student T distribution or something, is it what you're doing that's different. That's.

No, no, it's. It's right. I mean we have that computation in. In. In the statistical discussion section. Right, right there. Irf. Irfc.

Okay. Okay. Yes.

And it's interesting to see that that quotient that we have been looking at appears there naturally.

The two times.

The two times the difference divided by the square root of the sums.

You see.

It'S the elements of the subtraction of the two arounds. Center and standard deviation.

I get it, I get it. But I mean. Okay, so that's the abs. But now there's a square root of two here which we could presumably take out of the earth, could we not? I don't remember the relations for earth. It's not like an exponential where we can trade it out, but that square root of two. Okay, so if we. If we take this. We're asking the question as you change that number. I'm. I'm pretty confused.

You should look at the plots. Maybe looking at the plots.

The plots at the bottom.

Right. And that, that one right there I think is the most helpful. Right. So you know the blue area is the.

I. I think it. I think the three at the bottom are clearer. The ones for one sigma, two sigma, three sigma.

Oh, okay.

Yeah. No, no, further down. This three. So for example, if we have the two first Gaussians, do we consider that those are clearly separated? That's the meaning of one sigma. And for example, we might say there is a lot of overlapping. But if you think the center of the yellowish distribution is clearly at the tail of the blue.

Okay, you know what this reminds me of? It reminds me of like the area criterion in diffraction.

Yeah. So. Right. So why, you know, that's a function right there that you can write. So why don't we plot 1/2 Sigma and see what that looks like.

1/2 Sigma being what you are proposing. If you had a 1/2 instead of a 1. This is. This is what you're currently proposing for. Okay, can somebody. Whoops.

Oh, did we need to find sigma? Oh, sigma is defined. Sorry, didn't realize we had that extra input there.

Remind me what the area criterion is? It's that the first minimum, you know what I'm saying? You guys remember this in diffraction, right? The 1.22 lambda over D business.

Yep.

Right. So that the area criterion Is that the first minimum of the diffraction peak that you are. I mean, maybe it's different because it's a question of what you visually perceive there. So it's a different type of thing. But this is the same kind of question, I think.

Right? I mean, it's given a wavelength lambda, what's the smallest, you know, area you can resolve based on pattern, Right. Or. Or length scale you can resolve?

Not area.

Right, right. But. Okay, so listen, let's understand what you're saying here. So you're saying the criterion is, in this case here, the two sigma case is the case where those two things are considered equal as soon as they are closer than this. Is that correct? That's the equal.

Exactly. So those two Gaussians will be. That's the border of separation between being separated or equal. And it's quite far already.

Huh? You really think it's okay to consider them equal when they're that far apart?

That kind of worries me, too.

So.

So that's why I'm saying two equal to sigma may be already very tolerant.

Tolerant. Even for equal.

Exactly. Yeah. Yes. I mean, equals sometimes feel a bit too tolerant, but.

Okay, so one criterion is, you know, you see that little dip there? For what, you know, this is like. Are you a bimodal distribution or is it all smooshed together? Mm. For what? Sigma. I mean, and clearly that's going to depend on the width of this thing here.

Yeah.

I kind of think that this. If this is all a smoosh distribution, it never goes down. Of course, we're not seeing the sum of the distributions here.

That's a very arbitrary criterion. I mean, there is a. We could try, I figured, with the formulas. But there is the formula for what's basically the P value, that the means are different. Given the two means.

That I think is the student T distribution, is it not?

Well, right. When you have a small sample size, use the T distribution. But if you have a lot of samples, I think there's just a formula given the means of standard deviations of the normal distributions.

And we do. We're effectively assuming infinite number of samples. In this case, we're assuming we actually have the distribution.

Right. The sample number doesn't enter into it.

Right. So you're suggesting, in the shocking kind of. You're suggesting a P value criterion.

Yeah, maybe. I mean, I think it's better than something arbitrary like just how the plots line up with each other.

Right. And then. Right. And these plots are a little misleading. Right. Because exactly how much they look like they overlap depends on the particular Sigma 1 and Sigma 2 chosen.

All right, well, so what p value. Assume that we can do this with p values, which we can clearly do. Right. What p value, what probability of the means in fact being different would you want to take for the case of equal, Would you be satisfied? Would you consider them? If the probability that the means are right and Right.

And so my objection to one sigma is. All right, so you're going to say that this is less than this when there's in fact a 32% chance that these numbers are equal. Right. That if you draw an infinite number of times 32% of the time, the values will be reversed. Essentially.

Okay, hold on, hold on. Okay, the question is, given these two distributions, what's the probability that the true mean. Let's see that the. Well, you're asking for reversal, but I'm just saying the probability the true means are equal is not something you can compute because that's a point thing from a continuous distribution, is it not?

I don't know what you mean by the two means are equal, right? I mean, we know what the means are and they're not equal.

Given that.

We know the distributions exactly.

Yeah, yeah, right, right. The correct question is given. Okay, what the heck is the criterion?

I mean, this is why we started out by computing the probability, you know, given, you know, our MU1, our, our MU1, which we took to be zero, and our MU2 that we call to be mu. What's the probability that, you know, X1 is less than X2? Okay, and then, then we.

Right, and then what on our live stream suggests that we look at this thing here. Receiver operating characteristic.

I have never heard of this thing. What field we're talking about?

Binary classifier system. Discrimination threshold is varied. This does seem relevant.

I mean, we can study more probabilistic characteristics of this problem, but I think we just need some sort of judgment call to make this work. And then if we have to adapt it in the future a bit to change the confidence levels, we can, but this seems relevant.

Thank you for the suggestion here. I think this is relevant. Oh, look at this.

This type of approach would require a new function in which we can specify in a separate argument the level of confidence that we want or the levels or whatever. But I think we want something automatic.

No, I agree, I agree, I agree. What is this whole thing?

We have a setting like we have for equal tolerance and same Q tolerance, which is hidden but could be set in a block. That would be the.

No, I understand, but that That's a. That's an escape valve.

Well, sure, sure, of course it is.

But let's look at this receiver operating characteristic, which I have never heard of.

Yeah, we will.

All right.

But we're going to pick two numbers, and they're somewhere between a half and two, probably, and that will be our default. And we can have an escape valve. And I think that's where we should go.

And around. Around is still experimental, correct?

It is, yes.

Right. And so. Yes, and. And you know, which. Right. I just mean, I. I was just saying, you know, we could maybe make the. The escape valves perhaps slightly less hidden or just sort of tell people, let's make them hidden.

Let's make them hidden. I don't think they're ever going to be relevant. Let's just, you know.

Well, I'm just. I'm just thinking if we get feedback, it's too strict, it's not strict enough, then, you know, it might be useful for people to be able to tweak. But.

I see this, the thing that's interesting here in this curve. This is, you know, false positive rate, false negative rate, true positive, true negative. So that's a way of characterizing if you've got these two distributions. Right. It's the criterion. This will be our false negative rate is our rate of thinking they're the wrong way around, so to speak, is our rate of having the variables assigned the wrong way around, etc. That makes sense. Mm.

The. Right. I'm not sure we ever heard or. Or maybe I just missed it. You know, Mark's objection to.

To same Q. Yeah, no, sameq is fine. It's just structural. But I was worried about order. So you talked about numerical order. That's numerical values. But what. How does order sort around objects? Well, I think it doesn't take it.

By center, by centers, and then using the true order on the numbers.

Because.

Centers are by first. Yeah, but that is like numerical order. But that's like. With an infinite. Or using all, you know, ignoring the uncertainties.

Right, Exactly. Exactly.

Yeah.

Okay, then that's absolutely fine. That's exactly what I think it should do.

Yeah.

Okay. Right.

Exactly. So.

Yeah, no, I just thought it didn't take a.

No, I did.

Yeah. So it is kind of related. It's like having the constant zero in your.

In your order.

And same Q are already doing the right thing here because of the structure. Because having the center as first argument.

Yeah.

All right. Yeah, we can do collaboration here.

Okay. Do we want to talk a bit at all about the anti Symmetric case.

Oh, you mean the asymmetric case.

Yes.

Asymmetric case.

Yes, yes.

Right.

At which our proposal is basically to do the same thing only, you know, we first, you know, compare the centers to decide which, which sigma faces the other and use those is the basic idea.

In the asymmetric case, the distribution is broken into two pieces. Is that correct?

Correct?

Yeah.

Yes, it is. The distribution of one asymmetric around is, is, you know, is two half Gaussians.

Right, right.

Which are. It's a discontinuous as a function of. Of, you know, variable value. It's a discontinuous P of X is discontinuous in X.

Right.

P. Right.

I mean, one could have a discussion of. Is that the right model? But that is the model we are currently using.

Yeah, that's the model we're using. I think, I think any other model, I mean, you know, you're chasing, you know, the whole point around is you've got some rough estimate of how much error there is in something thing. And you know, by the time you're saying, well, actually it's this particular, you know, asymmetric distribution, well then you're using completely different technology.

Yep.

Okay, so you're saying look at the facing sigmas, so to speak, and then apply this criterion.

Right.

Which means that, I mean, other pair sort of paradoxical cases where you have a very, very wimpy, you know, opposing sigma in which.

Exactly. Right. So you, you can have a case where, you know what the center that is higher has a probability of being, you know, upload asymptotically approach 3/4 of being less than the lower center.

There are examples. Willow, if you open the statistical examples.

In the statistical discussion section, if you want to take a look.

So this case, for example, we consider them equal because the overlapping is large. The long tails are facing precisely each other, so they overlap a lot. This criteria would say that these two are equal, while if you face them the other way further down, they are.

Different.

Because they overlap very little. And so that side is very precise in a sense.

Oh boy. Well, I mean, in that receiver operating characteristic thing, for example, what matters that, you know, it seems this whole question about what's the area of overlap. Right. Seems to be the thing worth thinking about, which is an integral, which means that that number isn't a pure ratio of standard deviation to difference of means. It's some inverse earth type thing about. I would have thought that the, you know, the criterion should be something about the size of the overlap integral as compared to the size of the non overlapping parts.

Which is essentially what we're simplifying by. I mean, so right, in the symmetric case, that's basically what we're doing. Exactly right. We're just, you know, using a multiple of the standard deviation as opposed to. And here, yes, we're cheating a little, but it's not.

I, you know, what is your proposal here for what to actually do?

Right, so we take the two centers and, and order them as if they were exact or you know, whatever precision numbers they are. And then we, you know, use that to determine which of the plus or minus is facing the other center and use that in the standard deviation to compute an effective standard deviation. And then use the symmetric criterion. And then in the top place where the large standard deviations are facing each other, right. We see a large area, you know, there's a large area where they can be oppositely ordered and we declare them to be equal because they're both sort of near the middle.

I understand, I understand why the criterion isn't if the overlap area divided by total area is greater than blah, blah, then they're considered equal if the overlap area is less than blah, you know, let's say the overlap area is 95.

First of all, define overlap area.

Well, it's going to, in the end it's going to be similar. It's just that we want a criterion which is fast enough to be computed. This, this.

Okay, so you have to approximate that. But I mean, if what we're thinking about is what's the probability that there is.

Right. We're looking at the negation, right? Our equality function, right. Is one minus the probability that, you know, one guy is less than the other, minus and minus the other other way. And we're looking at when is that large.

Huh?

So we can test your idea and I think it's going to end up being something very similar.

But fine, I'm sure it is too. I'm sure you can approximate it very well by some purely arithmetic criterion, so to speak, that doesn't involve ERFs and so on. But I mean, you know, assuming it was an integral criterion, what would we want the ratio in order for the things to be declared equal? What do we want the overlapped probability to be?

I would say very similar numbers to the ones we use for the symmetric.

Case, which is what, 90%, 95%.

Yes. If we are using like 1/2 and 1 Sigma, 1/2 and 2.

2 Sigma was what we were using before.

Yeah, 2 Sigma.

Okay, so final question then. Numerical order as compared to a Quality test. Same criterion or different criterion?

I would say numerical order has some sigma multiple which is smaller than one. I would say 1/2, 3/4 of Sigma.

All right, let's just see what this means for the overlap integrals and then I think we're done. Okay. Can we move on?

Yep.

Okay. All right. Bra and cat.

So this is, I mean, something we discussed possibly pre pandemic. It took so long for somebody to go and do this, but it's, you know, so now we're just talking about the documentation.

BP as opposed to AP before pandemic. No, we don't have an ap. Anyway, who knows? Okay, so the only question here is whether it's. Or whether it's a list and it.

Wants to be a list, which is.

Why does it want to be a list? Because it.

Because when you put a bra and a kit together, that has to be two arguments.

Wait a minute. Why does.

It's for consistency with bra ket.

I see. Does make sense. Yeah. Do we have anybody here representing the quantum framework? Is this going to be. Is the quantum framework going to use this directly?

I don't think we have anybody representing the quantum framework.

Okay, well, can somebody check with them? I mean, it looks okay to me, assuming they're happy with it.

Okay.

Yep, we can check with them. Okay, let's go echo here. Right.

So, I mean, this is, you know, a big, messy discussion. So I don't know that we.

Well, let's give it a try for a minute here. Okay. What we've talked about previously is, let's see, where did we get to last time? Didn't we already have a discussion?

Okay, well, we had a five minute discussion at the end of.

Okay, all right. Okay. Okay. Okay. So let's describe what some of the things we think we want are. Goals. Okay. So, You know, put a wrapper around a whole evaluation. To sort of penalize, slash, limit, slash, whatever echoes within it. That's one type of thing. Another thing we want is for a single echo, make it not generate too much stuff or penalize itself. Is that a fair concept?

I thought we had shelved that idea.

Okay, well, tell me where we've got to before.

Well, there's the list of the functions that were discussed in the last time we had a serious discussion. And, you know, I'm not sure we're. We're going to, you know, page in and move beyond that in the few minutes that are left, but.

Well, okay, so you're suggesting that we table this for a custom echo discussion, Basically.

Sure.

All right. What else do we need to prepare for that discussion? Do we think we have the use cases understood?

Well, I, I mean, there were. Well, it's not clear to me that we have the use cases understood.

And that's what I was trying to write down, what some of the use cases are. I mean, the basic point is. You're trying to balance the number. I mean, it's essentially, Okay, echo is being used as a, as a diagnostic, but once you've seen it a few times, it's enough. You don't want to. I mean, the cases where I've literally had the thing running, it's generated, you know, 100 megabyte notebook, just echoing random stuff out. When I didn't know it was ever going to trigger, right. I was just like, is it going to trigger or not? You see what I'm asking where you don't really want it, you know. Okay, you're asking the question, does it trigger. Okay, you see, five. Five triggers. Five, you know, actions of it. That's presumably, you know, then that's enough. You don't want to see it a hundred thousand times. That's one use case. What's that?

Well, I mean, I would agree you don't want to see it a hundred thousand times, but I, I disagree that the lower limit is five. I mean, for. If I'm, when I'm actually debugging these things, it can easily be dozens.

Fine, fine. Okay. The question is, okay, what's the criterion? What's. What is the criterion to shut it up? Okay, so A, you've seen it more than X times. B, you've seen more than X echoes altogether in this input line. What's another criterion? Right?

And you're, you're imagining something where just all you have is echo and there's no outer wrapper.

No, I'm, I'm saying we could put an outer wrapper around the whole thing to determine this. The issue is, okay, so you know, for example, we could have, you know, an echo, okay, echo control. Right. Of expert. And maybe we have something where we can say, we can put an echo control there. It has a nice sub value. You say echo control and you say five or something. And that means, I don't know whether that's per thing or whether that's, you know, maybe this means that's the maximum per thing and then this is the maximum total and things like this.

Right.

Right. I mean, so I, Right. I. You'll notice that I have a suggestion of using echo, Echo quiet. The same way. They're right in the middle of quantity. Yeah, but of course you. You would act. Right. We would need to implement, you know, actually implement hold all sub values before something like this could work.

I see, right.

Otherwise all the echoes would evaluate before the quiet echo.

Yeah, okay, fine. But we could do the case of this without sub values without solving that problem.

Sure.

But sub value case is more useful. But okay, the cases that. Okay, again, again. Another. Another goal is some evaluation produces lots of echoes. When it's finished, get rid of them. That's echo cleanup or cleanup echo. Yep.

I don't think you're right. I don't think any of this is significantly different from what we discussed last time.

Yeah, I agree, I agree. So we need to home in on what we really want to do. But. Okay. Dan, our livestream is suggesting. What about redirecting echoes to a different window or file?

Yeah, so. And one of the things I was going to ask, not being an echo expert is. So echo has the side effect of creating a thing in the front end. Potentially. Does it have any other side effects?

No.

And the reason I ask is because. Because managing a stream of echoes looks a lot like a log file.

Sure.

And what. One of the things that makes log files useful is being able to process them symbolically.

Lots of interesting points.

I mean, you know, how do people process a log file that is hundreds of megabytes? You know, they're doing things like filter by time, filter by the name of the event, filter by, you know, some other criteria that is being symbolically collected. You know, and then show me only the relevant stuff that's like echoes as.

So reap kinds of things.

Kind of. Yeah.

So I mean that, you know, you could imagine something that says reap echo.

And so, you know, I mean, maybe sometimes, you know, it's. Maybe you kind of want to see the hundred megabyte stuff because. Because it's going to take some exploration to figure out what the relevant portion of it is. And I.

When.

So when I say see it, I don't mean literally embed it in the notebook. I mean, but have it available to you for exploration.

Right. I mean, this is. Okay, the other. Another case is redirect the echoes somewhere else.

Like print action.

Yes. Which we don't have freckle. Right.

We do not. Right. That was another suggestion that I thought we had tabled in favor of the list that we had at the top here.

Well, okay, you're telling me basically we can't solve a problem right now.

But.

I really think this is the thing we should, you know, we need to address. It's a very useful piece of functionality, but go ahead.

Yes, well, there are details we need to get into, and each time I bring this up, you get distracted by this grandiose vision and never discuss the details. So. Okay.

Example of detail.

Well, like this. Dollar echo limit. I. I don't know.

Right.

You know, a. A, the name is wrong. B, I'm not sure what it is. We're actually. You know what it is. We want to apply here and. And how we get there from here.

So this is a more complicated discussion. You're right. We either. We. We. We. We kind of run out of time here, so probably we should wrap this now and expect to do a custom meeting just about Echo. Okay.

Or just have it first on the agenda for a change. It's just that the around stuff, you know, since we could foresee getting around stuff done for 13.3, that took precedence.

Fine.

Okay.

Sounds good. Anything else for here right now?

Nope.

Okay, see you all soon. Thanks to folks on the live stream. Some good suggestions there. Much appreciated. See you all soon.