Live CEOing Ep 737: Language Design Review of Diff

Okay. Hi everyone. This is a continuation of our design review about diffs. And let's see, where do we want to start? Should we review? Okay, so how far did we get last time? What is our next topic?

Well, we got through all the basic diffie mechanisms. We talked some about three way diffs.

Okay. But let's look at, remind ourselves we had a granularity level. Okay, so.

Right.

That was sort of, we'd be talking about generate diff, view diff and so on. We haven't resolved that. No, the symbolic aspects of this. Right, right. Okay. Could somebody glory or somebody else, can you please remind us where we got to last time?

Gloria wasn't in the last meeting. But, but it's actually the list of unresolved issues. If you just scroll down a bit is, is summarized in that, in that notebook.

Okay, but let's not start with multi alignment. I don't think.

Right, well, so there's the, the third argument that, that was something, a point that's worth discussing.

This is, this is what I'm talking about. This is what we have to resolve first.

Okay.

Okay. Okay. So my suggestion is exactly this diff should produce a minimal viewer. Am I making sense? Because we all agree that diff on two big files, you know, we don't want, you know, the thing you want to see is something which is heuristically showing you the right things. If you want to produce, get something programmatic from it, you should be able to do that. But the default thing is that it should heuristically show you the right things which it's convenient to do in a viewer.

Okay, so we've talked about two possibilities that, you know, it produces just a neutral blob and that it produces a viewer. There is a third possibility that it produces a blob that can become a viewer interactively.

All right, I agree. I think the right way to do it, as we talked about last time, apparently we don't remember, is something like the output size limiter.

Right.

Well, yes, I mean, we, I, we, we do remember this. It's just that we don't all necessarily agree and there's two lines down.

The number one thing we need to do before we start talking about multi alignment or anything like that, that's the most basic operation of this thing is I type diff and I have, you know, file one and I have file two. And I want to see something. What do I want to see coming out of that? Okay, so what I think we want to get is something which is, you know, an Appropriate viewer.

So. So what? So, so, I mean, here's a question, which is, what does diff actually return now? I mean, it, it's. It clearly formats as something.

Right now it doesn't return any particular thing. The thing it returns depends on the kind of view that is happening automatically.

So for this. So. So it has some arbitrary head.

Correct. That doesn't necessarily survive reinterpretation.

So I think that's. I think that's wrong. I think what we should do here, this should be visibly a viewer with framing rather than some exotic thing like this where we don't really know what it is. Okay. Once you have a thing with framing, you know, if you want a programmatic version of this output, you know, you have to extract that as a property and possibly it can have a raft or something.

Well, well, well, I don't see why the programmatic thing. Well, I mean, I would argue that the, the. Right, that that should be a diff object and the diff object should format as a viewer. But, you know, if, If I do. If I. If I do, lowercase diff equals capital diff semicolon, that lowercase diff should be a computable object.

I agree.

So as as a reminder, we have two things right now. We have, in the current iteration, we have a thing called diff and a thing called generate diff.

I know, which I think is wrong. Okay. We went through that last time.

No, well, I mean, the point being you just said that diff should create something from which you can extract the object. I mean, I'm just.

No, I'm saying this is what should happen. Diff should turn into diff object. Diff object should format as a viewer, Diff object should have sub values that extract various computable things from it. And diff object itself should itself be computable. So, for example, you know, I don't know, if you diff diff objects, it should do the right thing and so on. I'm making sense.

Yeah. Well, I mean, diff objects are computable right now.

Fine. Then I think we've got.

Sorry, sorry, sorry, sorry. I was, I was muted.

Right.

I'm. I'm just saying. Yes, I mean, I think. I mean, I, I feel like I'm acting a little bit of translate as translator here, but. Right. Basically the, the claim, Stephen's claim is that the, you know, basically the, the formatting things that are provided by diff should be just become format values of.

Diff object, which are the default thing that you see with an appropriate heuristic diff object should display as the thing you're likely to see.

Yes. At least in, in some cases. Right, right. If, if, if it's going to be.

Too complicated, then it, then it will just say, you know, just like we do with any of these things. Right. If it's too complicated, we just show the blob.

Yep.

So, you know, a case like this, it would probably format with this, with framing, basically.

So, I mean, one of the things that I worry about is with anything along these lines is what happens if you end up with dozens or more of these things on screen at a time. It calls to mind the struggles that we had with having a summary blob for, I think it was for polygon that actually rendered a thumbnail of the authentic polygon and in doing so burned a lot of.

So we have to make sure we don't do that here. Just like the output size limiter obviously doesn't blow everything up by actually rendering the whole object.

Ok, so here's a different possible issue. So we have this diff object and we have these different views. Right? The same, Right. The same diff object can. Can. Right. Right. Now we take, generate the output of, generate diff and then we give these different views. Now give, if we have the one diff object, how do we tell it to, you know, go a different, you know, go to.

Yes, well, okay, so, so I'm saying that diff object by default, okay, there might be an option to diff that determines what the default kind of, you know, attempted diff object formatting should be. And I say attempted because if it turns out that, you know, the thing kind of is giant, it's not going to show the whole thing. You see what I'm saying? So I don't know whether that's an argument of diff or whether it's a diff format, arrow, whatever option, if you get what I'm saying.

So I just realized something which is kind of fundamental to think about here. Many diff viewers allow you to see context, including deep into things that aren't different. Sometimes it's just there by default. Sometimes you have to expand it. So a diff object doesn't carry the original objects around with it, and if it did, it would potentially be very.

Large, except that it's just a pointer. Well, look, listen, listen, listen. If you say diff of two files and they differ only in one character, the diff object is, okay, you're saying that the current diff object, as you have it doesn't carry those files in It.

Exactly, and it shouldn't. It carries around a small amount of context information, but that's potentially tiny compared to the files as a whole.

I understand that. I understand, but whatever it was that you asked it to diff. Okay, so, well, what is your solution? I mean, somewhere you've got to be able to get that context, so somewhere it's got to be available.

Well, what I'm. I mean, what I'm concerned about is that that may suggest that a fully general purpose diff viewer, which could be different from what we're talking about here, but a fully general purpose diff viewer might not be the diff object. It might be a different thing.

Okay, I don't agree. Okay, it's the same. It's the same thing as the issue we've had in videos. For example, is the video in line or not?

Right.

Or is the video referenced somewhere else?

Right. So what, what if you're.

But what if you're.

I don't agree.

Yeah, what if you're diffing two notebook objects? Those are inherently ephemeral.

I understand, I understand. So you have a choice. Just like you do with videos. You have a choice. You can burn more memory and make sure that it's local, or you can say there's no such thing as.

As a. Right. I mean, Right. Notebook objects are inherently right. They're things that only exist in ram.

Right.

I don't have to. I can always get the notebook object. I can say notebook get. And I can get it as an expression. Right. It's the exact same thing as we have with videos. In fact, that's a very good model for this. See what I'm saying? You know, if I do. Okay, look, look, typical thing, right? We all know this. We say, actually, I could do this with a Too big. Or I can do it with a data set or whatever. It's a bad example because it isn't big enough. Oh, come on. But I happen at some point. No, come on. Here we go. Data, not a notebook store now. Okay, that's what I think. We should do something modeled on that idea. Am I making sense?

You're making sense.

I think this is a good idea.

Consequent. I disagree. Yeah, I don't think it's a good idea at all. I think it's just confusing and. And will lead to notebooks that are extremely bloated potentially.

Okay, look, most of the time you won't choose to do this. Most of the time, a diff is for the purpose of, you know. You know, let's look at the scenarios. Okay, let's say you're different pieces of the human genome, okay? And most of it is out on disk. You're not going to put the human genome in your notebook, at least not today. Hey, yeah, I wonder if the, if the. I'm sorry for the silly thought, but I'm just, you know, right now our, you know, digital storage is, we, DNA is a much better form of storage. But, you know, anyway, maybe, maybe the evolved, you know, species of the future will have much bigger genome sequences and then it will work differently. But anyway, never mind. Already crabs and potatoes have bigger ones than we do, so who knows what that means? Okay, so what was I saying? So then, you know, it's conceivable you have two genes or something. You want to send somebody a notebook. Even though these genes are somewhat big, you want to put them in the notebook. You don't want them to, you know, have some random fasta file somewhere on disk that, that was on your local machine and they didn't have it. It's the exact same story as with things like data sets and videos. It's very unlikely, it's very unusual that you, say, burn this video into the notebook. But occasionally you do it because it's more important to you to have transportability than for it to be small.

Well, note that it won't be transportable in any usefully computable way. The only way in which it would be transportable is that you would be able to use the viewer to see the same thing.

No, I, I, I. Well, yes and no. It depends. The actual diffs may be what you want to compute things with. See what I'm saying? And the actual, you know, you may want the actual object.

Right, right, but, but, but, and you'd.

Be able to get that from the diff object?

I would think so. I would assume that. If you store the whole thing, can.

You, from the sum of the numbers, get all of the original numbers that went into that sum? That's very bizarre.

No, no, no. So, so look, why not? Well, but you're saying you want context. If you want context, you can just run the diff in the two different directions.

So, so as a part of viewing. Yes, but not as a part of the object.

Right. I mean, right. If, if, if, you know, if I send Linus, you know, a diff of the kernel, right. And that's, you know, however many hundreds of megabytes or gigabytes at this point. But the diff object itself is going to be what, like, you know, Possibly under, under a kilobyte, but having the, you know, but that doesn't change the fact that there are operations that need both the original and.

And so the reason that that works is because you're referencing the original and it's the generic Linux kernel that everybody, you know that's out there on some URL. If you didn't have that, you wouldn't be able to do this.

So as part of the viewer. Yes, but not as part of the object.

Right, exactly.

Come on, look, why. No, no, look, look. It's the exact same thing as with videos and data sets and so on. Right? The example you just gave.

No, it's not.

No, it's not.

Wait, wait, wait. The correct reference is a web reference. I mean, something stable at that point.

At that, at that, Stephen, at that point. Why don't, you know, if, if somebody has the two objects, they can run the diff and compute it themselves.

I understand, I understand.

So, so then what's the point of burning them into the diff object? I don't know.

I don't know. But I think it will be useful, particularly for these more computational uses of diffs that aren't for following.

I don't think there's anybody here who agrees with you.

Well, I think most people are silent, so. Okay, I'm pretty sure.

Let me, let me point out.

Let's not, let's not address this any further. Right, Go, go ahead.

I just wanted to point out that the current diff functionality has an option that lets you include the full expressions if you know what you're doing and you really want that. If the expressions are so large they don't blow out the notebook file s size, that's fine.

So it's just an option. It won't be the default. It's not what most people will want.

Right, but that doesn't change the fact that for, you know, my use case, I would want to have a viewer where I can give it just the diff object that somebody sent to me and you know, a pointer to our common reference.

Great view. Great.

So, which is why. Right, but you've been denying the need for this separate viewer function.

No, no, no, I don't think we need it. Yeah, I don't think we need it. So I think we need a way. Okay, come on, let' think it through. I, I'm, I'm really sure I'm right here for once, what was a bad sign? What's that?

Well, yeah, we've, we've, we've been there.

Okay. I'm really sure this is the correct way to do it. Okay? So what I'm suggesting is, you know, you do the diff, you get this thing, which is a viewer, okay? Now you say, but I want to give somebody, somebody sends me a diff object, okay? I want to re merge that diff object with a, you know, with a new option to the diff object. In other words, you've given me. The diff object itself will come with a base pointer, so to speak. Okay, you see what I'm saying? I mean, it won't be called that, but let's say it's called base pointer or source Source pointer. Source reference. Now, by the way, we've got a bunch of great precedents for how this works. We don't have these people in this meeting, but resource objects are full of these kinds of things. Okay? Resource objects exactly have different ways to point to stuff. You know, we've got very well nicely developed technology for this. We've got persistent objects and resource objects and so on. They all have this ability to either point locally to local objects, to cloud objects, to URLs, et cetera, et cetera, et cetera.

Okay, so let me clarify a counter proposal here. It doesn't have to be full viewer or nothing. It can be something in between. So the sort of thing that I'm putting out on the table is that diff object can be a thing which formats displaying some level of differencing. But what that formats into isn't the full featured everything you might want to do viewer.

Why? Why? Why not? I don't understand why not.

Because the full feature to everything you might want to do viewer, you know, can do things like. Well, I mean, you know, I can do things like read the source material without having to turn it into the diff object.

But I mean, that means you, you have a function. Look, there's a million ways to do that. You can do it by saying you've got a diff object.

Okay, look, look, and also at some point, you know, this is a bit on from where we want to immediately discuss, but at some point we might want to talk about dealing with having a viewer where you can actually modify things, where you're dealing with a three way diff and you want to make decisions about.

I agree. That's a mutable data structure, you know, back, you know, that's a more complicated thing. We don't have very many of those. Let's talk through those issues. Okay? All right, so I don't disagree. There's no Reason, you know, we have a diff object here. The diff object has a UI that allows you to access lots of aspects of the diff object. The diff object could carry along a source reference through mechanisms like this. And there might even be a way of, you know, like. Like the source reference could by default be a reference just like it is for a video. Right. Or a data set, for example, an in memory reference or an out of. Out of core reference. And then one of the things in the UI might be a thing that says, okay, I want to burn this reference into the notebook here. Am I making sense? What you say is going to be useless, but Lou already put it in. I think it's going to be useful in some cases. Okay, I think most of the time it's not going to be useful, but sometimes it's going to be really useful. Even in the documentation of how to use this, it will be useful and we won't have a bunch of pink boxes coming out because the source material wasn't there. Okay, then.

Well, in my proposal, there couldn't be pink boxes because the limited viewer you would get from default formatting is. Would only rely on.

Yeah, I know, but I think that's wrong. So, I mean, I think you should be able to get to more stuff from there. And, you know, we can handle that in a whole variety of ways. Okay, then, I don't disagree that having a function called diff view or something that has a bunch of, you know, second arguments that are the type of viewer and has all kinds of options and so on, that seems fine. I mean, it's like show. It's like the difference in plot and show. Okay, and. And then this, those options that are provided here could also be provided probably in the original diff object, causing the. I'm sorry, in the original diff, causing the diff object's default display to follow those options in some way. Does that make sense?

Sorry, I missed.

I didn't understand.

I lost. I lost the last one.

All right, let's go through it again. Okay, so diff turns into a diff object. The diff object displays in some default way that is somewhat heuristic. Okay. The diff object has a UI that allows you to specify different forms of display with me so far. Okay, the diff object also has a UI that allows you to burn in by default. It will have pointers to source objects. The diff object would allow you to burn in those source object things if it turns out that it isn't always. Right.

Okay, right. Well, but yeah, I mean, I just don't understand what sort of pointers you're talking about when we're talking about, say, general expressions.

Right?

Either the expression pointers to memory.

Pointers to memory. It's just the exact same thing that is happening here in this data set. It says data. Not in notebook.

No, but, but, but it's, but it's still, but it's still an in of 260, right? It's not in the note. You're worried only about notebook size. I'm also worried about, you know, the object itself. N of 260 still has a million.

Okay, fine, so in another case then is it's file of, you know, or something or you know, the directory of the whole Linux sources, right? That's the other possibility for you. Right. And so burning that into the notebook is kind of a bad idea because it's going to read it all in and it's going to put that in literal form as characters in the notebook, but by default the pointer will be to an external file or an external URL.

That, I mean, that's, I, I mean, you're still not addressing the point that I'm making, which is I, I, I am worried about the full form of diff object.

Right, I understand, I understand. If you burnt in the complete Linux sources there, you would blow yourself up.

Well, but I, you know, I'm not talking about the complete Linux sources. I'm talking about. Right? This, this can, you know, diff two lists and two strings.

Yep.

And you know, I, you know, I, I either the, you know, you're talking about a situation where either the, you.

Know, okay.

Doesn'T round trip or.

Yeah, okay, okay, look, look. And we need to deal with the fact that there are things that we produce as output that cannot be given as input. That's what we've tried to patch over with things like this, Right. If I took this and I just put it in another notebook, for example, it would not work as input. Same thing if I hadn't done this. Right? Same thing with many other structures that we have. Okay, you're commenting that the same thing will happen here. I agree.

No, that's not what I'm commenting. I'm commenting that I believe the full form of, you know, if I, you know, if I, if I do diff of range of a thousand and range of, you know, one thousand of two comma.

Yeah, yeah, yeah, that, that the full.

Form of the diff object should not have two 1 million element lists in them.

Okay, all right, okay. Because as a, because as an efficiency hack, you want to make sure that, for example. Okay, so in this case. Right, okay. Reasonable display there. Okay. Reasonable display. Now you. What, what's the input form of this thing?

You.

You want generate diff to look at the diff object. ETAI is concerned about the diff object.

That's just silly. The fact that there are two functions there. It's just.

Steven, Steven. Hey, hey. We're having a design discussion.

Don't go around. Okay, I'm just getting frustrated because I.

We are having, I think, reasonable.

So ITAI cares about the input form of this.

Okay.

I don't want to see 2 million additional elements.

I agree with you. I agree 100%. Right.

So that has nothing to do with the store in notebook. Object in the store Notebook. Right. This is purely a kernel evaluation question. Has nothing to do with Notebook263.

263.

The diff object, the input form of the diff object.

That's what I just did here. No.

Yes.

Oh, yes, there, there. Yes.

Right. I'm concerned about that thing.

That is what I'm concerned about.

I agree with you. Okay, so here's what we need to do. Right? So this is what we're going to have. It'll say changes here. Ok? So one of the things it might say is, you know, left source or whatever the heck. I don't know what you call it here. Left, you know, or before source or something. Okay, before source, arrow, URL, blah, blah, blah.

But there's no URL here.

Don't. Don't give me.

I'm talking about this. I'm talking about this, this, you know, this case right here.

Okay, you're talking about the case by.

Default, before source should not be in this association.

Okay, I think I agree with you. I think this case. We're out of luck. There's no way to get. There's no way to record something once you've done the diff. Once you've done the diff, you've lost that input there. You've lost that. If you wanted to save it, give. Put it in a variable. Okay, I agree with you. On the other hand, if that had said video or something or you know, cloud or, or.

Or if. I mean, the case I'm interested about in would be, you know, if this were a git object and. Right. I would want it to record the git repo and the sha.

Yeah.

Because then that's something I can send to Jose across the hall and he can do something with it without Copying all the kernel sources.

Exactly. So I'm saying when there is a, you know, this is again the diff object in the case where it is just an in memory expression, before source is going to say missing, unavailable, or something.

Okay, I can live with that.

But if it is, if it can have something that's a meaningful pointer, it puts it in there.

Okay.

And, and I think the UI should allow the possibility of burning it in into the before source as an explicit expression. I don't know how that's. Exactly. And you know, and if you do the wrong thing there, then that's stupid and whatever, but sometimes it's going to be useful.

Yep. And we, we've always, we've long allowed you to shoot yourself in the foot if that's what you really want to do.

Right. Okay, so now, all right, so let's go back here to the question of. And so by the way, this, this deals with the question of whether the diff viewer can have access to context. Now, by the way, okay, so one of the issues here is you want to be able to give something, a reference to this expression without it being in the serialized form of the diff object. See what I'm saying?

Well, you want it, I don't want it.

But no, no, you guys want it. You want to be able to show context. Right? So in other words, if this is range of a million.

Right, right.

I mean, the problem is the diff object itself in, you know, for us in memory, you know, it doesn't matter. Range of a million. It's like, who cares? It's just a, it's just a pointer in memory. The problem is that when you serialize it, you will see range of a million and there as a, as a, as an explicit list of a million numbers. And you don't want that.

Right.

Okay, so the problem is that there is something which has an expression level reference, but you don't want it in the serialized form. And that's what you were doing with the generate diff diff object, et cetera, et cetera, et cetera, which I think is the wrong way around. Okay, now what you're also adding is the idea that I might want to, in the notebook, view this diff relative to this expression, which is in memory, which is big, which I don't want to be part of the serialized form, but I still want to view things relative to that. Does that make sense?

Yeah.

So one of the questions here is, I mean, you know, but yet you think that it is a better thing to be Able to copy the diff object stick in an email, which I don't disagree, is a nice thing, as opposed to having to say, export or having something. Well, here's a question, here's a question. If this diff object were essentially a graphic, not, not an actual graphic, but something like a graphic, and I copy it and I paste it somewhere else, clearly by clipboard magic, it's going to do the right thing, we hope.

Yes.

Right. So in other words, imagine that. I mean, in memory. There's no problem with this diff object referencing. Range of a million, doesn't matter. It's fine with that. The problem is if you try to serialize it and copy, paste it into an email, it'll blow things up. Does that make sense?

You're right. I mean, I suppose theoretically we could replace it with like a weak. Perhaps we could do something like a weak cache and a pointer, you know, a key to a weak, weak hash table. So it'll be small in serialized form, but still, as long as you're in the same session.

That's an interesting idea. Well, or alternatively, what it really, what you're saying there, and it's a mechanism we don't have, so far as I know. If you said local object, for example, that's. Well, maybe that. No, but maybe that is. No, no, that's on. That's on the file system. What you're saying is, you know, that it is a thing which we don't have this mechanism really, do we? Like an internal object where that internal object is resolvable. Surely we have this mechanism somehow where that internal object just. You say, make something into an internal object. Fine. It hands you back a uuid. Okay. And then we have this. We have expression objects. We have something like this, okay? It hands you back a uuid. You stick that UUID in your expression. Were you to need to resolve the thing, it would, you know, it would say, okay, do I have this uuid? Is it still valid? Am I in the, you know, am I in the right session? If no, then sorry, you know, can't find it. If yes, here you are. That makes sense.

So, and my claim has been that's potentially useful to preserve the viewing operation, but other operations, I mean, you know, one of the big operations that you're going to want for this beyond viewing it is of course, applying it. And in applying it, the source information doesn't, doesn't matter. I mean, you actually don't want the source information because.

So, so then ignore the uuid. I mean, if the UUID is before uuid. You know, if it's a thing called, you know, source UUID or something, or source source pointer, whatever it is, and that pointer is to a UUID thing, and that UUID no longer resolves. Well, then you're cooked. If you don't care about the uuid, well, just don't look at it. But it's not taking up a lot of memory.

Yeah. And so that means the default view will be something that should be recreatable from the diff object minus the source information. So that's, you know, I refer to this limited diff view as opposed to the full diff view. Full diff view requires the source material, but the limited diff view doesn't. It can just work with the information actually in the diff object.

And obviously for the before source is something that needs to be resettable because one of the most common operations in a git workflow is to take the same patch set and apply it to different shaws and see what results.

Okay, so the question then is, given a diff object, what is its sub value? Does its subvalue. Do we actually treat a diff object? Do we actually not make it a diff object, but call it a diff function and have the diff function be something that actually applies?

So, by the way, you can see what's implemented today, Lou, right? It just takes properties and you can see it. Is that right?

Yes.

Yeah. So just type in bracket properties, bracket after that output or percent, whatever. Yeah.

Right. That's its current sub value behavior.

Yes, yes, that's what's happening.

And so now it has a separate function that's called apply diff. Is that correct?

Correct.

It's not obvious to me what's right. I mean, I think this is not obviously wrong. I think it's very. I mean, I just don't know how common. I mean, if you had a diff function, how cool would it be? Or do you not care?

Sorry, how cool would what be to.

Have a diff function where you just use as a sub value? You just apply it, you know, you want to say, let's say, for example, apply this diff across a hundred files. You know, being able to just say diff object diff function, slash at, you know, all the files or file, you know, like for example, oh, oh, oh, oh.

Using the diff object as a head against the thing you want to apply it to. I.

Yes.

Surely there would be concern about overlap with the properties value.

No, no, no, no. I'm saying, I'm Asking you the question, if there was a thing magically called diff function. Okay, forget about the conflict with properties. Okay, if there was a diff function, how excited would you be about that? Is it. Is it something where that would look really nice or is it something where. Who cares? Because there's so much other equipment that has to be around when you apply the diff?

I'm leaning towards the latter, but it's possible I'm missing something.

Okay, all right, So, I mean, the issue is, okay, so right now there's a separate function, apply diff. So you've got diff, which creates the diff object, apply diff, which applies a diff object. Right. And perhaps you have diff view, that is some kind of show like, construct for diff objects. I mean, I have to say, my slight preference is to have diff view rather than some crazy thing with some sub value with options and blah, blah, blah, blah, blah.

Hey, what do we do for video if you want to change. So for image, I guess we wrap it again in image with new options.

That is correct.

Do we do the same thing with video or something else?

I assume. Yes. Okay, but I mean, with video there are more complicated operations like video recode and things.

Sure, yeah.

Right. So again, it's a more sophisticated. I'm guessing that some things we can just wrap it and I mean, like, like for example, image resize is an operation, not just a wrapper. I'm saying, I mean, I think if you want to change the exif data.

Right, sure, yeah, yeah. It's just, it's. It's always bugged me a bit when you. On. On an arbitrary object that you just want to do some option surgery and, and maybe there are better patterns for doing that than I'm aware of. It always seems clunky when I try to do it.

For what? For the, for the.

Well, for any object where you're preserving, you know, the central object's integrity is going to change its options.

Grof is a good example.

Yes, yes.

And graph is one where it does just have the outer wrapper. Okay. So as a very serious user of the graph function, Okay, I can say that most of the time it works very nicely. There are corner cases, some of which frankly are bugs and don't need to be broken, but they take years to find because people just don't really understand that it is a bug as opposed to that it couldn't be done.

And I'm sorry, graph is like image, where you just wrap it and graph.

Okay, yes, yes. And it works. I mean, I use it every day. It works very nicely most of the time. Except that, like, Nick, who's here recently found some corner cases that are wrong. Okay. But, you know, I bet those have been seen before, but nobody knew they were bugs. Nobody was pushy enough to say it's a bug, but. But in general, it works nicely. And so I think it. Go ahead.

I mean, the. Sorry, I was momentarily distracted. But you know, sort of the one issue with. With sort of, you know, continually changing options is. Right then there isn't an easy way to say if two guys are the same. Or maybe we.

We.

We'd have to think about, like, are these two diff objects the same except for, you know, viewing options? How do I understand?

I understand that's complicated, right? I don't know. Again, you tell me. Is the canonical gradation important?

Yeah, I mean, this is, you know, like, my example would be that we don't, at least, as far as I know, we don't have a similar sort of operation for manipulate. It's not something that people would use that much if we did. But on the rare cases, when you want to kind of. How do you do it? Oh, you have to end up doing some pattern matching surgery or something.

Yeah, yeah, right. Okay, look, I got a suggestion. So with diff objects and comparisons, you know, triple equals is probably not a good thing to use with diff objects.

Right.

You know, it's conceivable that doubly, if it's an important operation, double equals could work and ignore viewing options. Or alternatively, hash. I'm not sure if hash should. I don't think hash should ignore viewing options. I think it's too structural. But you could have a property of diff object that's called char or hash or whatever it is. Right. And that way, you know, in the.

Right.

Yeah. Now that I'm thinking through this a bit more, I think probably what you want to compare is not the diff object itself, but if you know some way to.

For example, you want to know if the diff object is empty, that represents an empty diff, Right.

Or you want to know if these two objects are essential, you know, have the same changes. Right. Basically, if they represent the same operation, irrespective of, you know, source data or anything else. Because that's certainly something that I, Some sometimes need to do where it's like, okay, you know, are. Are these two branches basically making the same set of changes? And so then I create 2 diffs and then I diff the diffs and ignore the, the stupid parts.

Okay, so look, look, I think we're going to probably need a diff diff, so to speak. I mean, a diff. Yes, we could just diff the diffs. But, but, and that might work given that they're all expressions or. I mean, so for example, the, the, the representation of an empty diff should be something very clean, certainly in the viewer. It can be clean, but it should be easy to determine that the thing is an empty diff. See, the problem is that you might want to carry along viewing options even for an empty diff, just like a show of an empty graphic might want to carry viewing options. Yeah, but you know, I don't think it's terrible to imagine a little cluster of functions, you know, diff equal and things like this. I think that's better than hiding all that stuff in weird options or weird properties.

Okay, well that gives us a design direction then. If you're kind of signing off in principle on functions like diff equal, diff empty, that sort of thing, then we can, we can start to design around that and iterate.

Yes, yes, but I mean, I think. Okay, so let's review again. Okay, so I want, I want to go back to some of the other things which were in here. Okay, so first thing is we need a graphic design for the default diff viewer and how it unfurls itself. And you know. Okay, I want to come to another issue.

Yeah. And unfortunately, we'll need probably a multiplicity of designs because it will depend on what it was you actually diffed.

Yeah, right. So, but it needs a design, you know, a, an overall, you know, sort of brand approach, which I think won't be the same as the, as the output size limiter. It won't be. It'll be a different thing than we've had before because I think these diffs, like the one you displaying in 261 there is, you know, I think it wants a little bit of light framing, but not too much. I think you want it on white, probably. I don't think you want to, you know, have some weird thing with blue and all that kind of thing. It's going to be complicated enough without doing that.

And by the way, just as a, an aside, one of the things that's been brought up is that the typical color experience, which I think we want to use because, you know, lots of people expect it to work that way, but the typical color experience is not colorblind friendly. And so it would be good to have a colorblind friendly Option?

Yep. Wow, I'm surprised nobody fixed that in all these years.

Well, so. Our internal git hosting solution has, you know, it uses green and red by default, but then, you know, there's a personalized option that you can pick, you know, which goes to that blue and yellow, I think it is.

Okay, so we can have a theme type thing, you know, just like we have plot theme. We could imagine a diff theme possibly that we use as an option to the diff viewer, which, which I mean, my, my claim is that most of those options like diff theme, let's say, which I don't think would be a stupid thing to have, can be both options to diff itself and options to the wrapper diff viewer to be applied to the diff object. Now what I'm also interested in is when the diff object returns some of these things, like removals, some of them are just purely lists of things, but some of them need structure themselves. Do you see what I'm saying?

I, I, I don't quite know where you're going.

Okay, so look, look, look, look, look, look. If I say changes here, okay, what the hell is that?

So that is the changes element of the association from the. So I can describe what all these numbers mean if you'd like.

Oh, no, I'm not interested in that. Well, I am at some point, but I mean, the point that I'm trying to make is there's yet more structured stuff here, okay? Just like a diff object is a structured thing. I mean, we could, okay, the comparison sequence alignment is something very un, you know, if I just do sequence alignment, we made the decision in sequence alignment which might or might not have been correct to say, you know, Okay, that's a sort of incomprehensible thing. Okay. It doesn't have structure, it's just, you have to take it apart yourself. Am I making sense? As opposed to something that's a whole association with, you know, because in, wherever it was in this form here.

Right, okay, okay. This is part of why we've been having a bit of trouble tracking. Because by structure you mean at least in part, you mean human comprehensible structure.

Yes, I think, I mean that, I mean, I mean things where it is self evidently easy to go in and write code, that does it. Whereas here, I mean, I've done this many times with sequence alignment, you know, of writing something that, for example, right. Puts these things on top of each other and so on and so on and so on. I'm saying, I mean, so so in other words, I mean, I might say something like. I mean, I don't know, what would I do here? I would say, you know, if it's of depth to, you know, stack it, is there an easy way to do that? If I say column of the whole thing, it's going to be, well, I mean, okay, so let's say, I say, no, it's not. There's not really an easy way to do it. I have to say if depth of hash greater than one column of hash, otherwise hash, Well, that did the wrong thing. Depth greater than two, perhaps. Okay, something like that. But that's kind of messy. Am I making sense? And so, I mean, there's a trade off here of, of some things when it's just a list, You know, additions, for example. Okay, so I, I assume additions. Maybe it isn't. Maybe that's complicated too. Oops. I see. Well, I don't know what, what I would have to do to get that. But I mean, is that going to be a complicated str. You know, thing with a bunch of detail in it, or is it just going to be a list of additions.

Will be the pieces of changes that represent additions. So changes is giving you all possible changes. Some of those might be additions. In this case, there's only one item in there that's not a match and that's a change, an edit.

Okay, but, so what I'm looking for is some things here will just be a list of things, 50, 51 or something like that. Other things will have. And what I'm a bit concerned about is the incoherence of, oh, we got one structure for changes, we've got another structure for conflicts, we've got another structure for this. Or is that not a valid concern?

Well, so changes is just a list of all of those things together, right?

What's each individual thing? What's the definition of what it looks.

Matches, additions, removals, edits, conflicts.

No, no, I understand that, but you're giving this. In the case of a list, you're giving it with part numbers, correct? What do you do? If it's a string, for example, then.

It'S string, index, position, numbers, character counts.

Okay. Do you know do. Is that obvious? Does it show that?

Does it say the word character counts?

No.

The diff object, however, does say that it's a diff object from strings.

Okay, but, but when you get the thing out, if all.

If you just look at a piece of it, you're going to lose the context that it came from. Strings other than the next.

See this here. Okay. I consider this a bit iffy, maybe. I mean, this is a decision we have to make right at this point. These are structures. So that structure there is not the same as the sequence alignment structure. But it could be. Am I making sense? I don't understand why that first thing says one one, by the way, why doesn't it say one?

Those are the starting positions in each string of the following substring.

I see, I see, I see. Okay, but. But look, this thing could as well be the sequence alignment output, right? Nearly. What's that nearly?

The one way in which it's not is the elisions that you see in out of 267.

What about this? That's the sequence alignment case.

Yes, but if these strings were hundreds of characters long, you'd see elisions that.

You don't get in sequence alignment, but not in changes. You wouldn't.

Yes, you would.

No. This comes back to the point that the diff object doesn't carry around the entire source objects. In your case, it is because the source objects are so small that they're. That. Because what they do carry around, they do carry around the context of the changes. And context covers a certain region of things that didn't change. If your expression is small, the context is going to cover the entire expression.

Right? We probably need an object that represents that contextualized thing. It's kind of a sequence. It's a. It's a sequence context or something. I don't know what it is. Okay, but. But in any case, let's. So this is another issue is exactly what are all these things? Right? Okay, here's a way to think about it. What's the workflow of. You extract changes and then you feed it into another function. What's that other function? What kind of thing works with it? Can you feed it to take and drop? Can you feed it to replace part? What does it. What can you do with it? Fair question. Lou, would you like to. So there's.

So there's. Yeah, we're thinking one can use the diff. So are there examples of other objects which can be. Whose properties can be extracted and used? I imagine there are lots.

No lots. Sparse array would be an example.

I mean, the big thing that I think we'd want to do here is to be able to modify the diff object itself to say, this particular removal addition. We just don't want it. Remove it from the diff object.

Wow. For example, so I think that's another function. I think you need another wrapper function. That does that called diff replace or something like that.

Well, but what I'm getting at is maybe, and I say what I'm getting at, I hadn't actually gotten there yet.

So.

Sorry. Is that maybe if you start by you know, getting the changes, you know, and then kind of do your magic on those, and then can you convert those, that list of changes back to a diff object if that's the most sensible workflow?

Yeah, I mean, one thing that came up in the last meeting when Stephen suggested that we just allow the output of sequence alignment to be a thing that you can feed to diff and that diff would digest it. I think that would probably cover the case that you're talking about, John.

Okay.

I'm not 100% for sure, but it's definitely the case that there are, you know, it's not common, but there are cases where, yes, I definitely do want to modify diff objects in some way. Whether that's a function that does those modifications or whether I can, you know, kind of boil it down to raw data, process the raw data and then reintegrate it into a diff object. I mean, those are the two possible workflows I'm imagining.

Sorry, I spaced out for a second. Could you say that again?

Right, right. Okay. So I want to make a change to a diff object. I can imagine two ways of doing it. One, as you say, have some kind of diff modifying functions like diff replace or whatever. That's one possibility. Another possibility would be to basically just pull the raw data out of the diff object, process the raw data directly, which might be something like this changes list or.

Yeah, and then repackage it.

And then repackage it. Yes.

Yeah, but okay, so a modification of both of those would be that you could just say changes arrow and then, you know, in the diff diff modify or something changes arrow. Blah, blah, blah. I mean, this reminds me a little bit like in molecules and maybe Michael, Jason, who's here and hasn't had a chance to say anything, you know, when we have molecule modify, for example, that's a case where we're poking at a fairly complicated structure and making changes to it.

Correct.

And the thing is atomic. And that's the only way that you should change it. Yes, sorry. And that, that. And that's might be sort of similar to this. That is you have certain. I mean, in that particular case, you have certain operations that you do with the molecule. And I mean, here I'm a little bit suspicious of. Take it apart into its diff object pieces, you know, modify it with generic code and put it back together again.

So, so, so I will say that, you know, there is a version of that workflow in the git world that I do and many other people do all the time. I literally do it on a daily basis, which is to say that I have made changes locally to my source code and I am going to package up those changes into a commit. Now, the most common workflow is I just add all the changes to the commit and it's done. Everything I change goes into the commit. But there's also a very common workflow where you say, I don't want every change that I made, I want to split this up in some way. Or I just want something to not go in at all. Maybe I have a bit of debugging code I don't want to remove, but I don't want to commit it either. Or maybe I've made changes which logically makes sense as two commits, but I wrote them in one go. I now need to add some pieces to commit one and some pieces.

What, what happens when you do that? What happens to your local copy when you make one of those commits? Does the local copy then rearrange itself to be a diff relative to the now committed code?

Well, so, so git has three areas that are relevant to this. It has the working directory, which a commit operation will never touch. The working directory is just, that's where I'm editing. That's where my editor is pointed to, is the working directory. Okay, so then it has the commit store. And obviously when a commit is finished and done, it goes into the commit store. But then it has a third area called confusingly the index, or to use a less generic term, some people have tried to rename it the staging area. And the staging area is basically where it keeps track of the commit that you're intending to create. And it's by effectively moving these diffs, possibly in their entirety or possibly a line at a time, into and out of the staging area that you're declaring what the next commit, what the contents of the next commitment are going to be.

And there are, you know, options to allow you. Right, so you can compare the index to the working tree, you can compare the index to the commit store, you can add, compare the commit store directly to the working tree, et cetera.

So we should learn from that. But, but I just want to raise.

And I should say, you know, it's, it's not the easiest thing, you know, when you have People who don't have a lot of experience with this kind of walking into it, this, this notion of kind of three different areas holding your code. It's a bit confusing. It's the kind of thing where, you know, you have to learn the tool and really kind of to appreciate its power.

Okay, but, but one case that you have raised is modifying the sources, you know, that is going, clicking through whatever and saying do this, do that, etc.

The process of moving things into or out of the staging area. Yes.

Okay, so let's say in the diff viewer, okay, so I just want to think this through for a second. Whether we can do something without it being mutable. Okay. And what our analogies to that are.

Well, and John, in your case, you're not actually changing the diff, right? You're selecting pieces of the diffuser for the staging area.

That's correct.

And you're not selecting other pieces of the.

I mean, in essence I'm, in a way, I'm creating a second diff from the first diff.

Okay, so may I make a suggestion? So let's say you have this diff object which is displaying as a diff viewer, okay? Then you go in, you point your, you start editing in the notebook, you click, okay, why not copy down? And why not allow you to actually edit, you know, in the diff viewer? And when, if you hit shift enter, the thing you get is an edited diff object.

Okay, I'm not opposed to that, except it's going to be a bit of a high wire act to implement. But aside from that I'm not opposed to it. But I'm. Yeah, but I'm also rooting for a programmatic way to do the same thing.

Absolutely, I agree with you. Okay, but in the clickity, clickety click, I want to take this one, don't take that one, and so on. But programmatically you might say, you know, I want to take all these diffs that have this feature. But you know, the things which are not going to be easy to do programmatically, the things that are not going to be easy to do by hand, you're saying, you know, so programmatically I've got a list of changes, right? Each change, it's not really labeled, there's just a list of them. I want to check changes three, seven and 18. Yeah, okay, so I agree with you that some kind of, you know, it's basically a diff to a diff. It's basically a, an explicit as opposed. Okay, the thing, okay, so, so one Way to think about it would be that you write a diff object by giving a minimalized association. So in other words, if all I do is to say diff object of the association changes, arrow 5 goes to whatever it is, whatever the heck it is. Okay. And then the canonical. Then the diff object puffs itself out by canonicalizing in some way, or maybe doesn't. Maybe. Maybe. Basically, what I'm looking for is a diff apply of that diff object to your diff. I don't know that that's the right way to think about it, but that would be one possibility.

I don't know if it is or not either.

Right. But I mean, then the. The other possibility is that there is a diff change diff modify function that takes, just like molecule modify, takes certain kinds of, you know, things you might want to do to a diff, so to speak. And, you know, maybe there's a definite list of them, maybe there's an obvious syntax for it. I don't know. But that. That's the other possibility. So in other words, there's diff apply, which is taken. Take an expression and apply a diff to it. And there's, you know, some kind of diff change that changes the diff itself. The fact that those have to be different is disappointing. Yeah, I mean, here's another argument.

Here's another possibility, by the way, would it be helpful? I kind of came up with, I think, a pragmatic example. If it would be helpful in terms of how to think about this, go ahead. So imagine you're doing a notebook diff. And imagine in your copy of the notebook, you, maybe you make notes to yourself, and in those notes, you. You add a private cell tag. And so now you want to send those changes to someone else, but you don't want to send the private stuff. So you basically say, look for any additions of cells with a private cell tag and remove those from the diff object.

Yep. Yeah, it's a good example. So, I mean, to make that as easy as possible, you know, it'll be a delete cases or delete key or something like that. Yeah, delete cases of this thing. And if that's nice and easy, then we're in good shape. But, but the thing that concerns me is that thing at line 267 there, the thing. You know, things like this are comparatively hard to deal with, as I showed down here. Right, sure.

Yeah.

And so that's really the question is how. How straightforward can these things be so that you can just run Delete cases, which is again, this question I was asking before. Given the object that comes out, what are the functions most likely to be applied to it?

Yeah, and my example frankly, was one of the simpler cases kind of designed that way so the example would be comprehensible, but was one of the simpler cases one can imagine. It's, you know, it's only looking for additions and basically if you don't like the addition, you just throw it out. Or, you know, it might be in the middle of three or four other additions, so you might have to surgically remove it from a sequence of cells.

So I can imagine in the case of notebook diffs that the, you know, you'd say diff of these two notebook objects. Maybe there's even a menu based UI for doing that, I'm not sure. And the viewer would initially show you kind of a window onto a newly created kind of diff notebook, but presumably you could pop it out so that you go to the left notebook, the right notebook with markup on it. And of course, if you got that raw diff object and in a place where those notebooks were no longer accessible, you're toast. Yeah.

Yeah.

We have to wrap up here.

In a moment, but yeah, I actually I was, I, I, I did want to kind of bring us to a conclusion with a few questions about next steps, so. But I didn't want to interrupt any.

No, I think, I think we've got a design direction here which I think is really good. And I mean, there's a lot of detail to work through here.

Okay.

Since we have the bioinformatics folk here, I want to ask the question about N way alignment. And just a reminder for me, multiple sequence alignment is this phenomena, is this thing right? You have many sequences and you're asking what their relative changes are. Is that a true statement?

Yes.

Or you've got one little sequence and you want to find it in a whole, you want to find a host of other sequences that contain it or contain portions of it. Let's, let's, let's talk about. That's probably the more common scenario. Okay, so, so we've got, give me something really concrete. We've got the cytochrome C gene of a gazillion animals or something and we are trying to ask the question how do all these differ? Or what, what, what is, what is the question we're asking. No, the typical question is here's a DNA sequence, a small one, where in this 5 million sequences do they occur? And the other ones are typically much Longer. Okay, so I mean, it's. What's the real case of this? It's like saying, I see, I see, I see, I see. Okay, so that case would be of all the proteins in the protein data bank, where does this particular little run of base, you know, of amino acids occur so that I can see whether those things twist up into weird little left hooks or something. It's a little bit simplified in reality.

You allow for a lot of things to be included and still want to count them, but.

Yes. What do you mean included? Like, you mean it's only an approximate correct. It's not, it's not chop. Yeah, it's not. What?

Yeah, it's not, it's not one to.

One if it's not a literal match. Right, but so that's using things like blast and so on to, to estimate whether it's, you know, to get a distance metric. Yeah. Okay, so what we're asking, this is probably going to be relevant to our diff stuff. You know, edit distance is the most basic kind of, you know, change distance, so to speak. Right. So I mean, we're. Because, you know, diffing is ultimately sort of a heuristic thing because it's a question of how long, you know, you, you could say every character is different or you could say this group has an indel or whatever. Am I right? So the question of what, you know, when do you decide it's an insertion deletion? When do you decide it's a reversal of characters or something? Versus and, and when do you decide that these sequences are close enough that you can consider them identical even though one has an A and one has a B in it? Is that right? So in the diffing that we're imagining here in the end, there is a way to get precisely from one sequence to another. There are no, oh, there's an A in that one, a B in that. I mean, we, we could represent that by saying, well, it's an A in one and a B in the other. And it turns out we don't care about that particular element of the diff. Am I making sense to people as opposed to just saying, and I don't know how it's represented in multiple sequence alignment. Just saying, we don't care if it's a C or a G there. In fact, I guess the way that's represented is that there are in standard, you know, bioinformatics land, there are all these different codes that go beyond just A and C and G and T. Like an N means we Just don't know what, what, you know, what base pair it is. There is that right? So we're, I think this covers many, but not quite all of the cases. Okay, but, but okay, but so when the question that's come up here is if we have five files and we want to look at their relative differences, what would be the bioinformatics way of thinking about that process and what would be the structure that comes out? Because I mean, you know, maybe it's just like our sequence alignment function which can only work with two sequences, can't it? Yeah, right. I mean, look with more. What's that think of like the maximum common substructure. But that's a very hard problem on. Even on simple like chemical graphs. But on expressions it would be very difficult. No, I understand, but let's just take lists. I mean we could just imagine if the lists are the same, but we have five different lists and there's this one spot where they all differ by in different ways. We could imagine doing that with the same data structure that we're returning for sequence alignment. We could imagine dealing with more than two strings, I believe. Does that make sense? But it's important that the structure that Lou has developed with changes and things like that. I don't think that will work with more than two. Lou might want to comment.

It already works for three. I mean, it's just if you, I mean we can talk about the details if you want, but it's extensible to arbitrary numbers.

Okay, okay. It's not obvious to me how but, but okay, well, so clearly the sequence alignment function is extensible. The way it works, it's clearly extensible. You just have runs of common sequences and then you have the, you know, areas where somebody differed. And so they've all got to be listed explicitly.

Right. And the changes object has the same information except with indexes along for the ride. Exactly the same.

All right, fine, fine. Well, okay, I think there were some comments on our live stream here, which I had a chance to look at. A comment from LC about making graph and tree like diff objects. And that's an interesting question. Well, I don't know. I mean that's a, you know, this question about representing. But this is a sort of higher level thing representing a whole collection of diffs as a tree, so to speak. But I think it's a higher level thing. Let's see. Boy is commenting. Those are about earlier things about pointers and references. All right. I think we're making really good progress this is great functionality. I wish we'd had this decades ago.

Yeah. So, so very quick next steps. How. What should we do? Before you see this again, think through.

The design as we've discussed it. Try and get some graphic design around the. What the Diff viewer looks like. I think those are all good places to start.

Okay.

And go through. I mean, some of these things, like the granularity stuff. I think we had come to a reasonable point there. I think. Yeah, I think we've got a path here.

Okay. So, I mean, it'll be a little while to process those things, at which point, you know, we'll fall out of context. That's. That's understandable.

I think it's okay. I think we've got far enough that we can, you know, we've got far enough that I don't think we need to maintain context here.

Okay. And at some point before we meet again, we might. We might have something that. That we would be comfortable throwing onto the Pack Web repository. Do you have any objections to that?

No.

Okay.

I mean, I would slightly prefer to see it again before it goes there because I think if people say, oh, wow, this is wonderful, and start using it, and then we're like, oh, but there's this silly thing that we have to change, then everybody's got pitchforks.

We will certainly advise right on the front page, the experimental nature of the packlet.

I mean, I guess I would have a slight preference if you've packaged it up well enough to put it in the packlet repository. I claim it's, you know, good enough for another meeting, but I don't violently object if you, whatever reason, want to put in the packet repository first.

Okay.

All right. Good stuff.

It does make it easier for everyone to try things.

Can't you deploy it? Not in the public. Public backlot repository, but just whatever put. Put it there. It doesn't matter. It's fine. Yeah, I'm not going to complain. Although you might want to put it under a name that isn't whatever we think the final name is going to be.

Okay.

Good stuff. Good work.

Thank you.

Okay, bye.

Bye.