Live CEOing Ep 757: Language Design Review of ExternalEvaluate functionality and TestCheckpoint

Right. Hello everyone. We are continuing our version 14 design reviews and Ricardo is about to present some things to do with external evaluation. Please go ahead.

Yes, so, okay, so today we're going to talk about changes we are introducing in External Evaluate and later to some addition we have done to the testing framework. Mostly the external evaluate is just a review. We already seen internally most of this stuff, so I expect this to be relatively smoothly. On the other end, we need to talk a little bit more on the testing stuff because we have more issues to resolve there. Okay, so for 14, we are introducing the ability in external Evaluate to basically provision and create environments on demand. The syntax is what you see here. Basically there is a specification that you can put where previously you were putting already Python executable. And this specification allows you to basically.

Shouldn't it be capitalized in that. In that sentence, creating Python environment. I mean, this is part. No, no, no, no, no, no, no, no, no, no. In the, in the progress monitor.

Oh, yes, I see, yeah.

Okay, so I mean we're supposed to have a procedure now for reviewing progress monitors. I don't. That only just got launched, but this should go through that.

Sure.

Well, consider that everything that is coming out from the progress monitor is coming straight from STD out. But yes, I control the dial and where possible we can do the fixes.

Okay.

All right.

So as you have seen, basically has been what, 30, 40 seconds, the process is done. And so this guy installed the environment under this.

I don't understand why register external evaluator doesn't come back with some kind of object. Well, it comes back with a naked path. Doesn't. What does external. Isn't an external evaluator a blob represented by a summary box?

That is only the running session we currently don't have. No, the external evaluate target is always going to be. Historically it's been always a path to a file.

But when you do register external evaluator, usually what does it return?

Okay, yeah, this is something that we would like to change. Basically it will return the uuid. Until now, it returns the UUID of the evaluator.

It's never returned. We've never had an external evaluator object.

Before.

But that's surely what we want now because I mean, returning a uid. Can you show me. Just show me the documentation page. Show me what it actually normally returns.

Register external.

Wait, here you go.

And so normally when you. It returns the key in the registry there is associated. Yeah, basically the key that it creates inside the registry.

Okay, this is wrong. What it's doing now. I mean, this is not consistent. What it was doing before is kind of a hack, right? Returning. Look, what it should be doing is returning something, presumably I can say external evaluate and then one of those UUIDs and then it will external evaluate against that registered external evaluator. Correct.

We are adding disability now. So previously disability wasn't even there. So the Register external evaluator is returning a uuid, but you cannot really start a session by referring to the uuid.

How do you have to start the session? How do you refer to that uuid?

Okay, you go to this. So we are adding now a third argument in external evaluate, which is basically it allows you to do. To give it a name. And now you can do something like Python name goes to and you use the. Basically the ID that you. That you. The name that you specified. So now basically we are adding the ability.

I'm totally confused. Would you go back again? Go back again? Okay, so you said. I mean, I don't understand why this isn't sort of an abusive registrar external evaluator. This is essentially creating a. What would you normally do with registraxel Evaluate? What would you normally. What would its arguments normally be?

The first argument is the system. The second argument is the file on disk that basically is divide on executable or any other.

Okay, so you're replacing that by the association that gives the dependencies.

Exactly. So that dependency now get processed by Register external evaluator and it returns to you the evaluator that was just created in this case.

I guess.

I guess.

Lovely. But now the thing it's returning is so totally different from just returning a uuid.

Right. This is the changes we.

I don't think it's right. I'm sorry, I just don't think it's right.

One thing about Register external evaluator though, it's never. It's just a side effect thing.

Right.

I don't think anyone has ever used the return value from Register external evaluation. You mostly use it.

It's got a uuid. How do they refer to that installation?

Well, that's the thing. I mean until now you don't have really a way to start from external evaluate uuid.

Okay, but how do you pick which of the seven pythons that is writhing around in your computer you're going to use?

Yeah, so previously there were a version that allows you to. This is all stuff we are trying to solve basically. And we are redesigning the. Also the return value of Register external evaluators so that you can Directly use it.

And I think, I think, Ricardo, I think I agree. I'm having a hard time understanding the change without understanding how it was intended to be used before, like why the change is necessary.

Yeah, and so the thing is that before it was returning the idea of the evaluator, but you don't really have a way to use this ID inside an external evaluate call, so I'm not really sure either or people were doing it before.

Well, one thing, registration evaluator has always been basically a side effect type function. One thing it does is it, if you say register external value of Python, blah, it makes that Python the preferred evaluator for Python. If you just call, if you just say external evaluate Python comma two plus two.

Right.

The question is, how does that pick the Python to use? Well, it picks the most recently registered evaluator for Python and that's always been kind of ugly, but it's always been basically a side effect function where the, the return value is not of any use. You, you register some things to set your system up and then you can either, you know, use the Python, use the evaluators by path, or you just rely on the default behavior register. External valuer makes this evaluator the default for that language.

So are you saying this makes everything.

Much, much easier, right? I mean, that's, that's a good idea from Ricardo.

I'm sorry, could you say that again?

Naming the environments, which is what Ricardo is proposing, is a really good idea. I mean, that makes everything so much easier, right?

So now you have the ability to assign a friendly name and that's the side effect, right? You assign a friendly name to this evaluator, you can either use the full path to the evaluator, which is another argument to register external evaluator, or you can, and that's the return value. If you want to use the return value, it's now useful. But you can also, I think you.

Should be able, whatever happens, in my opinion, you should be able to use the return value from register external evaluator in external evaluate. If it's a random UUID that refers to Ruby or something, then so let it be. Right. I mean, in our system, the more common thing would be that it's an object represented by a summary box and you pick that up and you use it rather than a naked uuid.

We could, right? We could have in the past or now invented a new thing, external evaluator objects sort of thing that, you know, a new symbol in the system that would have a summary box, but it's just the thing is that people don't program with calling register. People don't call register external evaluator very often. It's not part of your, it's typically not part of your programming where you, you have that in, in your notebook and you call it every time you, it's. You just call external evaluate Python blah. And if, if you want to set up which python that is, well, then you, at some point in life, you call registration.

So this, this thing, is it going to be persistent? Given a name assigned to a python with certain dependencies, what's the level of persistence?

It's persistent. It's definitely persistent. You would not call registrational evaluator. Typically in your session you do it. It's the sort of thing you do once and then from then on you, you don't, you don't use, you don't call registrational evaluator. That, that's why the return value isn't necessarily all that useful.

I mean, also, the main idea is that there are a variety of specification that can go into the second argument. So typically, yes, it's a file or it's a string. In case of a database, it's a database reference object, you pass it in. But it could be a URL, it can be also string. The guy normalizes to what it's appropriate for every evaluator. So for example, for SQL, it's a database reference object, and even if you put it a string or a URL, etc. It gives you back database reference. So in this case, the normal form of the evaluator is a string. And so the typical return value here, it's a string. Because this is what we historically use in order to refer to basically evaluator on disk.

I'm sorry, show how you can use the return value right now from register external evaluator if you want. Right.

So I just want to be clear. Once you've got a return value from register external evaluator, whatever it is, it should be a thing that you can persistently use inside external evaluate on that same machine.

Yes, so this is what I'm doing here. So basically you register the evaluator.

I got it. And then it returns that raw string.

Exactly. And this is the path to the evaluator, and then you can use it directly.

Right.

I don't understand why the evaluator goes like that. I. No, no. The return value from register external evaluate should be something that you can pass nakedly as the first argument of external evaluate, not with some kind of weird wrapper where you're telling it again that it's a Python instance. You see what I'm saying?

If it's a uuid, we have to. We'd have to either have it return, well, a uuid, which. And then. And then have some sort of, you know, reverse lookup of the uuid, it's just not. Or we would have an external evaluator object or something, which I think is a better idea.

Well, hold on. I mean, is that really. I mean, if clearly individually registered evaluators are associated with a name, but they should still be distinct, right? Like I'm. If I. It's kind of my fault if I register a Python evaluator with what, you know, called foo, and then later register a Julia evaluator called Foo. Right. I mean, that's. It's one namespace of named evaluators and it's my fault if I have a collision from different languages. So in that sense it is redundant to have to respecify the language, assuming we can count on people to not make duplicate.

Where are you giving the name in register External evaluator. Where do you give the name?

So it's a third argument and basically, yeah, you can do.

But right now that name doesn't stand alone as a.

It needs to. It needs to. You need to give that. Okay, which is really straightforward. I mean.

Well, it's not necessarily. It's sort of, but. But right now, you know, start external session, you can just say the string Python. Right. So we have.

And if you do that, then you have to give that as a second look. If you do that, you can either use Ricardo Env, or you can use list Python comma, Ricardo env, or you can use a uuid, all three of them.

What about just Python, but Python also.

Fine, but then there's a default, which is. I don't know how it determines the default.

Right, but what's the one String. If you just give a string as the first argument to start external session.

Or external value, I don't know how it determines which of the seven writhing pythons ends up sticking its head up and grabbing.

That's not so much the issue. The issue is just that if you're saying that this name could also be used as just the naked first argument or startle value.

Yep.

Then it. It sort of lands in the same namespace as all the names of languages on your.

Exactly right. So if you're stupid enough to name that Python, you will get confused. Right. Just like if you reset plus, you will get confused.

Well, okay, so basically, I mean, but.

What if you say, what if you say, but what if you use a language that you, you know, you don't have? What if you say Haskell or something, right? And you don't have. We want to be able to say, oh, we don't see Haskell. Let's look for a packlet that says it has external value for Haskell. But right now that first argument as a string is treated in a special way.

So look, here's the deal, just like we do elsewhere in the system. If you defined it as a user defined thing, then, and you've got a definition for it, then boom, you use the user defined definition. If you don't have a user defined definition, it goes and tries to use a system one or if necessary, looks for a packlet that does that. Am I making sense?

Yeah, that is consistent. We could just say, right, you say Haskell there and don't have a thing called Haskell. Then we do the, the lookup that.

I was talking about.

Yeah, let me do the external lookup. This seems reasonable, but I just think whatever you return from register external evaluator, you've got to be able to use that now.

Okay, then, then we need to revert this change where basically register external evaluator returns either the UUID that is created or the name. And so the we, we keep the register external evaluator return value to be the name or the uuid.

No, I don't think you need to do that. You can have it be that path if you want. You just have to have a hash table that has that path in it, just like you have a hash table that has the UUID in it.

Not really. I mean, this is risky because now this thing is a, is a string that refers to the Baidon thing. And I will need to. I mean, why Biden not node JS or not?

No, no, no, listen to what I'm saying. You've got a uuid. The uuid, you know, goes into the table of external evaluators, right? And it knows everything based on having hit that elements of the table of external evaluators, right? Think of this string as another key, another hash code. Basically for.

You could have multiple. You could have multiple registered external evaluators for Python that use the same path. I don't know why you would, but you could, right? This path isn't necessarily unique by looking up in that.

Not only that, but also there are expressions that can be complicated. I mean, the, the database reference contains all kinds of expression. I mean, ashing it. And, and yeah, I see. It's.

I don't think, I think the correct place to go, which we probably can't get to for this version, is an external evaluator object.

Okay, then I think we need to delay this change just to be on the safe side. Basically we have register external evaluator, return the name or the UUID so that we don't make a breaking change for nodding.

Fine.

And then we, and then we.

Fine, but then the thing you're adding is a name for the external evaluator. That's a fine thing. And then, then when you call the.

Thing, then you will put it under name here. So basically right now, in the current implementation and everything, you distinguish Ricardo env. And then you can put the return value into this key name.

Why do you need that explicit name? I can't just be Python. And then that list.

We can try to implement that.

Yeah.

Wait, what are you suggesting?

Next question. I'm just suggesting you, you should just say list Python comma Ricardo End. I mean, is there a good reason not to do that?

Well, if we, if we are going to move towards an external evaluator object, then we don't even need the Python.

Right.

It would just mean.

That's correct. All right, leave the name for now. Okay, but here's the question. In a WFR that needs a complicated environment, what does it actually look like?

Okay, well, in a more completed environment.

Basically more complicated, even this complicated, I just want to know what does somebody put when they write a wfr?

So basically when they write a wfr, they will write just this external evaluate and then you do python and then ghost 2. Let me go to all this stuff. And then you do two plus two. Sorry, no, it doesn't make sense to us. What are you doing?

Put the quotes.

I have no idea, but it's not working.

But you're not allowed to say base directory anything. I don't want you to see anything with the Python runtime. Nothing like that, is there? Nothing like that, is there? This is a thing to be distributed to different people's systems.

Sure, sure, exactly. So in the base directory you're not going to change it, and you're probably not going to use this specification. You're going to put something like 3.11 or 3.10 here. You are going to R code all your dependency and pin them to the right version. And so basically this is done by. We are going to do a workflow for that, because in the end you need to do pip, install all Your packages, then you need to run.

Wait a minute, we're doing that, right? Because this is. Listen, listen. The whole point here, which if we don't achieve this, we might as well just hang up, hang up this effort and go home type thing, is that people are supposed to be able to write a WFR that makes use of a Python package and it is fully encapsulated. They don't have to know the user of the wfr.

We got that, Stephen. We got that. When Ricardo said the author of the WFR has to perform this step of getting the dependency strings correct and that that's the author of the wfr.

But who's pipping anything?

I mean, that's author of the wfr.

Yeah. So the only issue is that you can write this, Stephen, you can write these, or you can write only this. Only this string. The problem is that with time software changes and this thing will bring down dependency. That. So basically the right thing to do is to freeze all your dependency.

And I got it. And there's a way to do that using some kind of PIP command.

Exactly, exactly. I mean, if you're using just. Most of the time, you.

Why don't we provide a WFR that runs PIP and that gets you that dependency string.

Yes. Into the workflow or.

So why don't we provide a WFR that's called Find Python dependencies or something, or find Python dependency string, something like that generates that string or generates that dependency list from whatever it is that you would give me the current versions of all these things so that I can freeze it, you know, even though the Pythons will continue to writhe around. Right.

Yeah, I suppose that's feasible to write a WFR to do that. Right?

Yeah, but that's very nice. So what you're describing here is the more. Is the less encapsulated version of that, where somebody is just sort of putting it on their system for permanence on their system, rather than having it be part of a more ephemeral WFR type thing.

Yes. So basically what you do is that you define some variable.

Why are we defining a variable? What value are we defining?

They don't need to. They don't need to.

No. So they shouldn't.

Yeah, they should basically put right in there what I'm saying. Yeah, they. They need to save this association somewhere or they can put it.

Save it permanently. Because the WFR is a fully encapsulated object.

Right, right. Hold on a second. Car is talking about the author of the W. This is what the WFR author puts in their phone. He's not talking about users of the.

Yeah, okay, that's fine.

They don't do anything.

They don't get to have a persistent variable. They get to have a function.

Sure, sure, sure.

Right.

They run the function and internally the function will always call this association, which takes care of basically updating the system if needed or just use the existing system if everything was already installed. So yeah, I mean, I think if there is nothing more here, I would like to basically move on because.

Right, what's the neat example?

Well, the neat example was just syntax ladder that is built on Python. And so you see the first time I run it, it's going to create the environment. So you see that when Python is already downloaded, it takes four or five seconds to build an environment. It obviously depends from how many dependencies you got. And basically this example is using a syntax letter to syntax.

I like it very much. This is great. I mean this is a good example.

Yeah, yeah.

And yeah, it works for like or something, right?

Yeah, I should say.

Yeah, this is what they use inside. Yeah. In Pygmans.

Well, why don't we get it fixed? Yeah, let's get it fixed, please. Indeed there should be alternatives, but let's get it fixed. I've said that actually three times before, I believe. How about we actually do it this time?

Right. So Aubrey, make sure that I know someone did something with this.

Yeah, let's make sure it actually talked.

About the last time.

All right, next item.

Right, okay, so let's move to the testing framework. So in the last version we created, we added to the system a function that is called test create that allows you to basically, you know, we divided the evaluation step from the creation step. Yeah, the addition we are doing this time is different. We allow you basically to have what we call checkpoints. And we are not really sure about this name, but tentatively is this one instead of test.

When did we introduce test grade? Was that in 13.3.

13:3.

Okay.

And so the overall idea is that by. If you just run test checkpoint, it doesn't do anything special. It returns the evaluation of the first document, but in the meantime is emitting a message to the testing framework.

Yeah, good idea.

More like rep. And so basically the. The test collects all the event and they are stored inside the. Inside the object.

I don't think it should be called checkpoint. Shouldn't it be called test logging or something?

We can get there.

We have a long discussion about the names which we'll get to.

Yeah, just a quick real World example of how this will be used. So supposedly you want to export a string into your disk and then read it back and make sure that you know everything, all the unigot character there. Then you want to delete the file, etc. Etc. So you see that now at every step you can record and freeze what the values were. And that allows you to quickly debug and see. Yeah.

Wait a minute. What are the arguments to test checkpoint here? One of them is an expert that you're just evaluating.

Yeah, it is the same order of the arguments of test create. So the first argument is the expression you want to evaluate. The second argument is the matching the expected output and. And then you can have same test in order to customize. You see that the second document should be.

Wait a minute. These are conditions under which test checkpoint issues a checkpoint. Or I mean if I just want to always put in my testing thing, I reached this point, what do I do? Do you understand what I'm saying? I always want to put in that test object. I want it to declare that it reached this point. I'm here. How do I do that?

I believe that's just what the default behavior of test checkpoint is. Because test checkpoint, it never alters control flow. It always records, you know, this condition.

Was successful or what argument then whether it records it or not.

No, no, it's secondary. It's exactly the same argument structure as verification test. It's the computed value and then the expected value.

In this case, like a subtest, it's like a test within a test.

What does it do? So I want to know how do I get it to always.

It always does anything. That's just what it does. That's just by definition, that's what it does.

Even if that second thing fails, Even if the second the understore string question mark file exists independent of that value, what does it do?

It will continue to run.

What does it put in the test output?

It adds an item to that table below where the failure type is something that describes, you know, your value. Didn't match that test.

Or it did, as the case may be.

Or it did.

Yeah, but I just want it to throw a thing that says I reached here into.

That will be something different. We are considering exploring something that breaks also control flow. But this is not what this function does.

But I'm not sure I understand what you want in terms of this. I mean, it also suggests that the name test checkpoint is confusing because it's not. It's not like just. It's like a Subtest within a test. I don't know what you mean by wanting to test.

Why is the subtest. I don't understand. The thing is just running through and this is not affecting the control flow, but throwing other data into the test result logs.

That's.

Yeah, I mean, it might be helpful understand the motivation. So the place this came from is that we're trying to make it easier for people to encapsulate what are currently written as multiple separate tests into a single test that's testing multiple properties at the same time. So the idea is that. And the example we started with is Todd has tests testing the packlet manager and there are certain tests where it's one logical test that's testing five different fields in an association. Right. Like there are five fields in the packlet object association that he wants that test to validate. All have the expected value. If he were to write that test as some kind of a single verification test where the expected value was an association and the computed value was an association and only one field from that association didn't match, it would be difficult to deduce what the problem was. Test checkpoint is intended to allow him to rewrite what are currently five tests as one test with five test checkpoints. So then he can see individually, okay, four of the association values, say, are correct, but one is incorrect. But even though one of them might fail, he doesn't want the test to stop running. It should still test all the other properties as well. So that's. That's the motivation behind Test checkpoint.

Yeah, also there is also the motivation that is one. And there is also this kind of motivation where you have a procedural program with several things that needs to be done in a certain order and you want to make sure that at every step they are done, the evaluation keeps going here because we feel that if you want to stop the evaluation, you can simply throw or break control flow with other work from language construct. And so we feel that basically this checkpoint should keep the rest going in order to regard all possible information as much as possible.

Let me just try and meeting a bit because I'm going to run out of time. Top of the hour. Just give me a sense of what else we have to figure out here.

No, not much. We need to figure out the name and we need to review a couple of properties that are coming out from the. There are a consequence of this name. So basically there is a property that allows you to extract checkpoints from. From the test. You see that this is the inner test and you can ask for the first level of the checkpoint that is inside, which is just one because there is only one checkpoint point here, but there are actually two because another one is nested. And so there is another property that allows you to extract recursively all the test checkpoint. You see that internally the checkpoints are just another test object like the other one. And. Yeah, more or less. I guess with the time that we have, we should try to figure out what the name of test checkpoint should be.

All right, tell me again. Test checkpoint. Test create. Let's go back to test create.

Sure.

What's the difference between test checkpoint and test create?

Well, that test checkpoint, it's only emitting an event. And yes, it's performing a test, but it returns to you the actual value of the evaluation. Test creates the test. And inside Test create, you're gonna have those sub test.

I understand. What does test create do? Test creates a test object of some kind, right?

Yes, yes. It creates a test object that has not been evaluated yet. Then you evaluate it and. And, yeah, you got an outcome back.

Okay, but at runtime. So this is the runtime analog. So if I just. If I put. Okay, let's say I say test evaluate, and then in there I have a test checkpoint. Test evaluate returns a test object. True statement or not true statement.

Test evaluate, return. Yes.

Okay, show it to me. Test evaluate of 2 +2. Show me what it does. No, no, no, no. Test.

No, no, no, no. It doesn't. Doesn't work this way. No, if you do Test evaluate two plus two, it doesn't.

Just show me what it does.

I'm actually not sure what it. Yeah, because it needs to take either a test object or an association. So you need to already construct the test by using test create. Then you pass it to testify.

Why is it the case that, you know, you could want pure runtime testing.

That is verification test.

That is verification test. And it's what we already had, like, since 10 years. And yes, it's working fine. People use it all the time.

But the problem is that show me Test checkpoint. Okay. Test checkpoint is another runtime test. Right. It doesn't sit there symbolically. It actually evaluates. Right. Correct. This is super confusing. Just like verification test actually evaluates. Right. So you're bringing in another example having. Having said verification test is wrong because it actually evaluates, you're now coming back and saying, well, actually, we need something which actually evaluates. Am I wrong?

It doesn't. Test checkpoint is typically inside a test object, which doesn't matter.

But just answer my question. Test checkpoint, like verification test, actually evaluates after the whole production about how important it was to have a test object that doesn't evaluate.

Right, but test checkpoint isn't a standalone thing. You put it inside a test object. It does evaluate outside the test object. If you just. But, but that's not. It's. It's. It's not a standalone function. It. It lives inside a test object.

Yes, as a thing to.

It's kind of like echo in the sense that it evaluates and has a side effect. Right. Of recording whether this worked or not and then goes on.

I think the confusion is that I think, Stephen, you think that test checkpoint does evaluate eagerly even inside test object, and that I think is not true. It evaluates eagerly only if it's standing outside. But if it's inside, then it's only evaluated when test evaluate is applied to the unevaluated test object, but not before.

No, I get that. I realize that. Because test object holds everything exactly. How could it possibly evaluate anything?

Right.

Okay, so test checkpoint, I don't understand why it isn't just. Why does it have yet another test in it? You're saying that. I'm just completely confused. And your example.

Well, I don't understand it.

You say as sock equals blah and then you run test checkpoint on the various keys of the asoc. Is that right.

There?

I mean, you're not returning anything there. If it's like echo, it's trying to return something.

Well, look at this example. This is basically more or less like a multi step thing where you export and you check that a file exists, then you check that the return value is a certain thing, you delete the file, etc. Etc. Etc. So otherwise basically it start to understand what. What was the thing that was going wrong exactly. Might have been that the file was not there. And then you need to figure it out by a message or anything else. So basically it allows you to then structure a big test into. Into. Into several step.

Well, I think it's. I mean, maybe one thing that Steven's getting at here is that the fact that it's evaluating immediately. Right. Means that in theory, if that test checkpoint, certainly the second one, which is just a statement, if that was just verification test, in many ways that's not super distinguishable from what test checkpoint is doing because it's just. It's not using a return value, it's just executing that test immediately. It's really the fact that the first test checkpoint returns something that makes it different. And you could make the argument that instead of having test checkpoint where the return value is meaningful, you could rewrite this code as export path equals just that export statement. And then inside the body of the with you have a test checkpoint comparing export path to that condition. So the fact that we've merged those together into one is.

It's like echo is convenient, but you could use echo as a pure side effect, or you can use echo with its return value.

Yeah, yeah, right.

Which you can do with the same thing with test check. I mean, it's a.

Okay, hold on, hold on, hold on, hold on.

In fact, here it's, it's a percent effect.

I, I know, I realize that.

Okay.

Although I don't understand why I need a test there. I mean, in other words, I just want to throw something that says in my testing results. And I don't know if Arnode is here, but he would certainly have an opinion. Yes, he is. I mean, I would assume that it's very useful in testing to be able to say the code reached this point. No test, just the code got here, I made it here. Am I making sense?

Yeah. I mean, many people do that with just a print statement.

I understand that, but this is now doing that test objects, you can test the code made it to this point.

I think that is in effect what test checkpoint is doing. You end up with a log in the resulting outer test of which checkpoints were reached.

Yes. I mean, you could use it for that purpose just to do nothing more than just have some sort of phony test in there that just pass to indicate that you've got here.

Yeah, well, I think you need that. I mean, you need that case.

Well, you can do that. Right now you just say test checkpoint of, I don't know, one, comma, one or something. And then in your results for the test, you'll see that checkpoint there as passing. You can see.

I have a feeling.

But that's not what it's for. It's for testing parts like think of it as subtest. I think the name test checkpoint.

Yeah, I mean, the reason basically why we're doing this design is because, you know, we wanted to have a, to have a test group. But basically we realized that it was difficult with the existing framework that we have to have test group and it would have been a better fit to have sub test inside test. So basically.

So look, I got a suggestion. You've really got two things going on here. The pure. I made it here just put something into the test output. You know, the idea of a hacky thing where you type One. One is just ridiculous, if I might say so.

That's ridiculous. But I don't. Why would you. I'm not sure why you would, but.

For the reason I just explained, to be able to put into your testing the fact that you reached this point in the code.

But that's another way. Yeah, right. I mean, if we had a way to capture during test running, you know, just like block of dollar output kind of thing to capture events, you know, that something was printed, something was echoed, that might be just as useful and arguably, I mean, you know, more that would be useful.

That is a separate thing.

We could have something like test event, you know, that would fire some event that would be recorded, perhaps, you know, test event.

Yeah.

Or test log. Maybe just test log.

Right. How does all of this relate. How does all of this relate to.

The assert function that we already have?

Because, I mean, that's kind of like.

What you want to do when you want to see things.

That's an assert. Which test makes a test. I'm saying you just want to say, I made it here. Okay. What you're doing there is, you're saying, you know, evaluate an expression, test its value. Okay. And then record the result of testing its value, which, God knows why. I mean, I don't know the. It's really. And then in the manifest of how many tests succeeded and how many failed, if there are a bunch of failed test checkpoint tests, I assume that those are counted against tests.

No, no, they are not.

Let them finish. It's.

It's right, because it's not possible to do that.

But no, that's not the point is that the outer test.

Hold on.

I think you guys just miscommunicated.

Yeah, they're not counting outer tests fail, Stephen. If any of the test checkpoints fail, the outer test is listed as a failure. So that's what counts in the set of, you know, I have 37 tests and five failed. We don't lose the fact that a test checkpoint failed.

Right.

But the other tests, the other checkpoints, the test continues to run past the checkpoint.

And you know, also this, this functionality that you are proposing, Stephen, where basically there is some kind of breakpoint in your code, I think this is a generic functionality that is not related to the testing system. You should be able to put a breakpoint here and it should work.

Well, I don't think he wants a break point.

I don't want.

Yeah, I just want a way to record something in the test results. It is either, because right now the only test result we have is whether the thing failed or not. And now we have these test checkpoints listed as with their results. But if you have a test log, that would then also record something.

Just regard all the STD out that is, that is coming out.

No, I want to record the specific message that I put in to say it got to this point. Record that I don't want standard out. That's a different.

What I'm saying is that when you.

Do echo, we could record the echo itself.

That's fine, but that's a different thing. That's not what I'm talking about here. I don't want echo. I don't want, you know, random crud printed on my screen. I just want to record in the testing system that it got to this place in the code.

So what if.

What if the way we model this is that there's every test has essentially like an execution log and there's three types of events at least so far. Maybe we'll add more. There's test checkpoints, there's test logs, and then there's kind of lower priority anything else that was printed, you know, whether it's from print or echo or message or assert.

That would be useful.

And then you could even filter. I mean there could be filters like hide the low value prints, but only show me the logs or only show me the checkpoints or whatever you want to see.

Yeah.

Because print output is actually, you know, might be useful to capture and.

Program against and the sequence of when these things arose is important. That contains information.

Yeah.

Okay, let's go down to the name of test checkpoint. I mean, I don't like this test log thing. The fact is that, you know, we need a logging framework in general. That's not.

Yeah, no, that's for sure.

I mean that's. That's something we've talked about in the error handling group about. I mean, I think the, you know, we've had idea about function like advise or emit something along those names.

That's structured logging log message or something and then put a wrapper around it that says logging with logging. Why not. Why don't have a function called with logging and then post to log or something or some, some other or. I mean unfortunately log as in logarithm is a. We could just say. We just need a term. You know, with logging would be something.

Where.

Like absolute timing or something, it would return the log messages as well as the value of the executed piece of code. So what I'm saying. Yeah, and then it would have a thing that says you know, it could be for emit log. Although that sounds a bit weird, but something like that. Emit log. And then you just put whatever the heck you want in there. And then you can, you know, lace your code with those things. Or it's a version of echo.

Well, I'm sure we'd have echo log. That would definitely be a function. We add in that hypothetical.

Right.

Which both echoes and throws it in the log.

Yeah.

And returns it. Right.

Yeah. And then quiet. So you could say with logging of quiet, of blah, blah, which will be a way of bank switching all your echoes. You see what I'm saying?

I don't quite follow why quiet is related to echo.

What do we do we have a quiet echo function. Right?

Oh, I'm so quiet. Echo. Yeah.

I said the wrong thing. Yeah, yeah.

But let's. The logging framework is great. But let's talk. Let's try to get a name for this. This. Let's try to get the name.

Yeah.

All right.

Okay. But please remember those two names.

Okay.

Yeah. With logging. I like that.

So it isn't test. So. Because it's not an un. An unfettered. So, I mean, that would be a reasonable thing to do, by the way. I really have to leave. But. But that would be a reasonable thing to do if it was. If it was just throwing a message in there. See what I'm saying?

Yeah.

This is both testing, and.

I think it's easy to think of it as like a subtest. I do. I kind of like test checkpoint. But maybe that name is. Well, I mean, suggests things that it doesn't do or isn't intended for.

I think a checkpoint. People stop at a checkpoint like Charlie.

Or something, and then they keep going. Hopefully.

Yes. But this doesn't ever stop. Nobody ever gets.

But it does record something.

Yeah.

Which is not what a checkpoint is. A checkpoint is, you know, your papers are not in order. You've got to go somewhere different. Right. That never happens here. That's not what this is.

Well, depends.

I mean, no, control float is not, you know, the control flow. The people just walk straight through.

But, but, but, but the actual result of the test is. Is influenced because if the checkpoint fails, the other test fails.

If that's not the right thing to call it a test.

So in some sense, you are either going versus success or verse. Or toward the failure.

I don't think it's the right thing, but sure, sure.

This is why we're here. Okay.

I mean, so I mean, just the ones we liked. I mean, subtest with Test. Yeah, the one that we like the most, though it has other issues, is tested. It's actually the very last one on this list.

Oh, God, no.

The part of speech.

This is not. No way. I mean, let's go back again to what this is. Okay, so what is this? This is within a created test. This is a. It's not a subtest because it's. It is a test of a different kind. It is a runtime test as opposed to a test object. I think if you call it subsidiary test or subtest, it's really obscure. That's not a test case.

There are some other suggestions here, just if you want to take a look. None of them.

Yeah, okay. All right.

But.

What about something like test point, which doesn't have this checkpoint, which bothers me. Because the check.

I like test checkpoint. So I don't mind test point. It seems like. I don't know.

What's the sentence? What. What is the sentence that you say? How do you describe it?

Actually, I think we should.

You know, I've got to go. I'm sorry. I told you, I warned you this is going to happen. I. I can. That's a really obscure thing to say there. You know what?

I'm.

I'm sorry to do this. I can. I can. You guys can talk about this for a minute. I will have to go mobile, so to speak, but I will not be able to do that for another five minutes. So I can.

All right, we can.

I can rejoin.

You talked about this. We could easily talk about this for five more minutes.

All right.

Okay. I'll. I'll rejoin you.

Okay.

Okay. I. I think it's interesting, your text here, which is, you know, whatever. It's just. I think we've ever talked about. But it says create an intermediate test. It is, I think, to me, that's an easy way to understand it. I don't know about the word intermediate so much, but it's kind of like an inner test, inner test subtest.

I wonder, though, because I hadn't really thought about it this way, but like, you know, when we were discussing it earlier, I mean, is there an argument to be made that verification test within, you know, inside of another test should just have the behavior of test checkpoint.

But verification test doesn't return the result.

I know, right? So my next question is going to be. You know, that to me would make sense. Right. It's weird. The test checkpoint that doesn't return a result can't be used in the same way that verification tests can be used. Right. So I mean, that to me seems like a thing that we should make consistent either way. And then the question becomes, what is the utility of this thing? Assuming that verification test is already a way to kind of add an intermediate test just by positioning, not by naming, what does it mean for test checkpoint to return a value? Because I think that that to me is really the thing that is distinctive about test checkpoint. It's not that it's an intermediate test per se, though that's clearly a requirement. It's that it's. It's not, you know, it's not returning the test itself. It's. It's kind of. Yeah, well, it's returning the value of the thing. It's kind of a like test as side effect.

One thing to say is that, I mean, right now if you use a derivation test inside another verification test, it won't be collected by the outer verification test. And this is because otherwise it will be impossible to test the testing system. And so we have this construct that is supposed to be used inside and is collecting messages. So it is different also in that sense that when you use verification test inside another verification test, it doesn't have the same behavior of emitting. Yeah. This intermediate test, Maybe one could say it should do it, but then it will be impossible to test the testing system. Or at least we will need to have another construct that is doing it, maybe internal.

But.

Yeah, just saying.

Yeah, super real verification test final.

Right.

But I think they serve different purposes still because test checkpoint is really much more like something like a. So something that records. But test somehow suggests that certain decisions will be made based on whether it's returning, like true or false, essentially whether it passes or test fails. But in this case, what is being done is that certain checks are being internally recorded in the process of test execution and those values affect the decision whether the overall test passes or not.

Yeah.

So it's different.

Yeah. I wonder if one way to think about this though is because it's really made me think. I mean. So right now we have the confirm family of functions that alter control flow when something goes wrong. But we've talked about, at least in the error handling working group, this notion of this structured replacement for messages. And if we end up having, I think there's like this big kind of set of dual operations that can be defined on functions that alter control flow and functions that don't alter control flow but have similar signatures. Right. So we have confirm. We have confirm by. We have confirm quiet. We have confirm matches. Hypothetically, in the future we might have functions that are like. And I don't know what the verb or the analog of the word confirm would be because I'm going to use check but that's not the right word. But you could imagine something like check by or maybe be log by or log quiet or log matches or something that doesn't ever affect control flow, but it does as a side effect emit some kind of message that people can opt into listening to. And I wonder if like test checkpoint is some early analog or early precursor to that kind of thing where it works specifically within a test. Yes, but its general behavior is that it's a non control flow altering way of putting out a message that something did or did not match.

Yeah, I guess it is.

I don't know. I mean I would wonder if like maybe, you know, I mean it's a question of like the utility of like. There are, there are things about test checkpoint, like the fact that it has the similar options to verification tests that do make it useful and the fact that it's actually a test object and we've talked about, you know, gives you timing information and things like that about specific checkpoint operations. But it does make me wonder if maybe it's. I don't know, there's a more general thing here that we've. If that general thing existed would just be something that the testing framework could understand and kind of plug into. Like essentially like maybe the testing framework would effectively run functions using the with logging idea.

Well, for what it's worth, back when I was constructing this in many ways precursor to these current ideas for R link, I had this with logging essentially in exactly the same way as Stephen suggested that. And I had test logging also pretty similarly I was collecting this log within the final test result object.

I'm sorry, could you clarify, Are you saying you had a function called test logging?

Yeah, I had the function called test log and I had a function test log.

Okay.

And I had a function which worked something like with test log, something very similar to what Steven suggested today. And I also had a function called test fail that basically was a kind of a limited control flow that was returning from a test early. I did not have a test checkpoint, but because I had test log I essentially could record things, although I didn't use that. If I wanted explicit places which decide whether test pass or not. I insert more verification tests inside my bigger test. That's how I structure things so I have verification.

Don't think that logging is something that should be specific to test and obviously test should be able to Record all kinds of stuff, including probably STD out, as I was saying, and logging in the future. If we have a framework to deal with that.

Sure, but, but I don't think we.

Should have any, any, anything specific for testing.

Well, I don't know. In practice it's very convenient to have something like test fail, which is very declarative and I think we discussed it a lot. We decided not to include it in the current incarnation of testing system just because it's something that we could add later. But in terms of utility, I would say it's very useful because it reads very nicely, especially given that many tests are naturally written in a procedural way, not in a functional way. If you have a bunch of statements that import something similar to what you regard and have in your example. In such cases having something like test fail somewhere in the middle can be very useful. So we didn't include it, I believe, not because we think it's useless, but because we think that it can be built on top of what we are building right now whenever we finally decide to include it.

Right.

But I mean, yeah, you can either start inventing all kinds of test related function to break control flow and log or use system generic function and include.

But regard. There is only one and it's called test fail. You won't need anything else.

Well, probably, but not for logging. And we were speaking about logging specifically.

Not about not for logging, but logging. I just wanted to mention that my way of implementing that was exactly similar.

To what's meaning right now. Basically if you want to fail a test, you can just do if something then abort and that will make the test fail.

Yeah, fine. It's just that a single statement was basically what I think if I remember correctly, what I had was test fail was taking a single argument which was basically a reason for failure. Just a single string I think. And that was enough to give very good diagnostics. But as we discussed, we could add it later. So that's why I think we didn't discuss it so far at length because it is clear that with what we have right now, just as you mentioned, since you can use a board, it can be something wrapper like a syntactic sugar based on that. But it's useful in terms of readability.

Yeah, I, I, Sorry, I've been kind of looking at other words, but you guys, you're talking about test fail now, right? Just, just a simple function that makes the test.

Yeah, right.

Yeah, I always like that idea as a super readable way. And that would be one of the failure. You could have the message in it and there'd be a little failure.

Exactly.

Modes for like exactly. Fail because of test fail.

Let's try to put in order the, the, the. The. The kind of functions that we, we, you know, more suitable. What is the best name.

At this point?

Because I'm not objecting to test fail. But it's not the scope of the meaning.

No, it's not.

Right.

We kind of want it up.

Can I suggest we put log matches in there though? Just as a. To emphasize that perhaps a generic thing would be log matches as in just kind of like a test log, but it's more. It's a generic function that happens to be understood by the testing framework.

Okay.

Hi again everybody. Did you figure it out?

We said fact a little bit.

You know, I was thinking about.

Okay, what new suggestion?

Some names. What about you?

You.

We had test checkpoint. You suggest test point. What about test check.

Look, what does it do? Tell me in one sentence what it does. Tell me in one sentence what it does.

I think the sentence here is I.

Can'T see what's on the screen because that we lost the opportunity.

Create an intermediate test to determine whether input evaluates true. Yeah, sorry to expect it.

Yeah, he's an intermediate test. Then why don't we call it intermediate test?

It is a possible name on the list.

What else other words? Is. Is it anything else? Because. Which I think is incomprehensible.

Which one was incomprehensible?

Subtest is incomprehensible.

All right.

So this is create test has an intermediate test in inside it.

Right.

I mean it's. Look, I don't think it's terrible. It's sort of comprehensible. Anything too clever is going to be incomprehensible.

This is true.

What about inner test? That's opposed to intermediate tests?

Well, I like the intermediate because it gives a sense of time and flow which is part of what the story of this is. I mean it's a pass through test. It's a test that is. I mean it would be nice to give the idea that this thing is a just a wrapper. You understand what I'm saying? Not a thing that affects control flow. It's some kind of pass through test. It's some kind of. I think you had test wrapper. But that isn't. I don't like that. Yeah, but you understand what I'm saying? That it's a. That it's a test that you just go straight through. But it has a side effect, right?

Which is. Yeah, I don't know.

Testament.

No.

Test.

No. That's weird. No, that's what. But I think that that's kind of what the checkpoint phrase you go to checkpoint do. It doesn't stamp your passport and go on. And that's the whole point of a.

Checkpoint, is that it can stop things. If there was. If. If a checkpoint didn't ever stop things, it wouldn't need to exist.

Well, that's not true. Checkpoints or something. If you race, you go to a checkpoint, it just records that you came through.

Fair enough, Fair enough. Okay, so look, what does checkpointing mean in programming? I admit that there it tends to mean that you, you know, record something and go on. Checkpointing is a. Rather than a thing where you can be diverted.

Right.

Yeah. I mean, at least within, I don't know, the world of computers. I mean, when I hear the word checkpoint, I mean, I do understand, Todd, what you're getting at on it, but I hear it as like. It's like you're saving state so that you can recover later if something goes wrong.

Yeah, exactly, exactly, exactly. Which is again a different thought. That's yet a different confusion from the one that I was having.

What about test check? You mentioned test check, test point. They kind of have the same. Without actually using the word checkpoint. I don't mind intermediate test. I don't have a problem.

I think it's ugly. I think it's ugly and I think it doesn't communicate that it is a pass through test. Effectively.

Stefan put something in the zoom chat Test. Milestone.

Way too ponderous.

Right? It's kind of a ponderous way to say test checkpoint, I think and, and no milestone even less seems to sound.

This is a muddy concept and this is why we're having trouble with its name. It's several different things.

Saying it's a, you know, what about test assert. But it's not.

No, it's not what it's doing.

Okay.

No, no, no. To speak cleanly, like everyone that we. That we've been talking to, both QA and other people that have seen this thing, they say that this thing is great. And this is exactly what.

It's certainly useful. It's useful, Ricardo. It's useful. No question he meant. But I think it's also conflating multiple. It's conflating multiple ideas. It's conflating this logging idea, which is useful and this idea that it is a. Okay, what is this thing A. What is it again? You're creating a test that is the overall Test that is, it's a runtime test.

This is something. It just shows something into the test results. It's. It shows an intermediate comparison between and you know, into the test result.

No, but it only does that if.

Okay, no, not really. I mean you need to imagine that the whole testing system is event based. Basically every test emits an event and it's just a matter of collecting all those events. And so basically, yeah, the outer test collects the inner events and so on and then the revot is built on top of the events only. So what this does is basically emit an event with the result of the test that gets collected by other system later.

I mean hearing it described that way though makes me feel something like, you know, something with log or logging in the name feels more. Right.

Yes. We don't have the logging.

What about test event?

No, it's not what it is. It's not what it is. It isn't. It is itself a test.

Right, right.

It's more like a pass through test. It's not like a standalone test which is what a test object is.

Right. I mean it's. What about test check? In a sense it's just.

No, it's not, it's not checking anything.

Well, it's not.

It's not checking anything.

It's just recording. It's checking. It checks. There should be some consequence of it checking. It's just showing the results of this. You know, it's just, you know, it's.

A, it's a. I mean one thing I don't like test check is because they're test and check their synonyms first of all. But other than that, I think there is something that is changing when, when, when a, when an intermediate test fails, the other test fails as well. So there is something that breaks later, not the control flow.

That's a good point. One of seven different. We will eventually break this function into its different component parts. Right now it's one of those sort of one man band of seven different things. And those are sometimes useful. They are often useful. But eventually we will realize, oh my gosh, we need a version of this function that just does logging. Oh my gosh, we need a version of this function that just causes the overall test to fail and doesn't throw crud in the log.

I mean right now you can still regular control flow. But yeah.

I think intermediate test is the least bad of our alternatives. Especially given what you just said.

I don't mind it. It is after all the term that I, I don't mind intermediate Test it is, after all, exactly what Ricardo wrote in the usage message for test checkpoint.

Well, I know.

That's why I asked what the usage message said. Let's go to intermediate tests as an experimental. It's an experimental thing anyway. But let's go with intermediate test. I think it's the least bad, so to speak. Okay, okay.

Just play devil's advocate though, since it sounds like we're.

Oh, sorry, I have to go in like one minute. So go on. Advocate for the devil.

Well, I'm just wondering, you mentioned. I mean, we don't have the testing framework now, but if we do. Or. That's right, the logging framework. But hypothetically, if we were to have the logging framework, do we think that intermediate test might eventually be replaced with some more general logging candidate or do we think that's the wrong way to think about this?

No, I mean that some separate logging.

Wrapper that we might use that could log a test where you could wrap a logging emission around a test, for example, or inside a test and you know, the logging thing would do something sensible with a test that might, you know, just look at a piece of the association that is the test, you know, transmission, so to speak.

Yeah, yeah. I mean, the more I think about it, I think having like a. An event log associated with tests would be very useful since that's kind of what's being introduced by this thing anyway.

Yeah, I agree.

So I think intermediate test as an experimental thing is the right thing for now.

Okay. And what are some of the other names that follow from this? There was. There was checkpoints or something as the. As an accessor property. Right.

Well, I guess that will become intermediate tests and. Yeah, which I mean we also want a recursive one that is called flat.

Maybe do we want the non recursive one? Why don't we just. Why don't we just have it always be flat? Why, why would you want.

Well, I guess program.

You could call it that. Look, you could have all intermediate tests.

Or.

You know, intermediate test list. I don't know what.

Or flatten.

I'm afraid I really have to. I'm sorry, but. Okay, thanks. Okay, see you.

Thanks.

Okay.