Live CEOing Ep 778: Design Review of DiffTools

Hi, everyone. Welcome to a Wolfram Language design review, an early one in our cycle for the next version of Orphan Language. This is about Diff tools, how you tell if things in a notebook or other type of thing is different. So let's take a look. So where are we starting here?

Yeah, so by the way, the examples in here work, but just, you know, the names are different just because it was what we had to do to get the notebook running. So it's just not in the system yet.

So let's just take a look at the most basic thing. Okay. Wow. Huh?

Wow. This has advanced significantly since the last time I looked at this.

Yeah, this is cool.

So as we discussed before, some of the ones toward the bottom of that list are maybe a bit more on the cutesy side, but. But still potent of potential interest. The. You know, as long as I work.

I don't have an objection.

Yeah.

Okay, so where do we want to start this review?

Okay, so I have four functions that I'm proposing and then some options too, but I haven't written up documentation for the options. But the functions, I mean, we can start right here with diff. And just as a prelude, I'll just say the other ones are diff3, which is the three way version of diff diff apply, which is how you take a diff and an object and get a transform thing and then diff object, which is the thing that diff and diff3return.

That's this thing here.

Yes, yes, that is a diff object.

Okay. How much do. What does the one mean there?

It's an internal versioning.

Yeah.

In case we change radically. Change.

No, no, I agree. I think it's a good thing to do. I was exactly going to ask about that, but we don't expect Diff Object to be opened by end users, so to speak.

No, in fact, Diff object has, you know, I mean, it has down values and it has a. I mean, the goal is for there to be enough down values to get any information you want from the down values and not from exploring the guts of Diff object.

Okay. Boy, this is a pretty complicated system.

It's. It's not simple.

Okay, That's.

I mean, it's part of why it's taken so long to get this together.

Yeah, I know. Okay. Diff3.

Correct. And then Diff Apply was the other one. But diff3 and diff are, you know, they're pretty close to one another, so.

They do have to be separate functions.

They. Yes.

And there's no diff 4.

Well, I think that's a fair question. And you know, I mean, generally the industry has a notion of three way diffs and not, not really a very standardized notion of multi way diffs. That doesn't necessarily mean that we shouldn't do it, but we haven't done it.

Yeah. Okay, well, let's take a look at, let's go through these. Okay. What's the other one?

Diff apply?

Yeah. All right. All right, let's start off then with what these apply to, so to speak. So sure. We've got lists, associations. That's futurized.

How come it just hasn't been done yet? It's, it's, it's just extra.

Is it obvious? What?

Well, well, I mean, you know, associations is kind of like a special case of lists. But, but we might want to kind of tweak things a little bit maybe. Like if the keys are the same but the values are different, we might want to present that in a slightly different way. For example.

Yes. Okay. Are there any other things that we have considered diffing?

Well, we, we certainly considered diffing expressions generally, but that is, you know, that's another mountaintop. Yeah.

And when it comes to diffs with like finite precision numbers, is there a same test story.

At this point? No, I mean, there is a certain amount of. So we do have, you know, this ability to have. There's a diff input function which one could imagine does some sort of standardization of something like that. And we are still finding interesting sorts of cases like that. I mean, one thing which Lou and I were looking at last week I discovered Lou diagnosed was this issue of notebook diffing and whether the thing in the notebook is in the form compressed data or not, which like is, you know, it's the same in terms of how the notebook is read, but it's representationally different.

Yeah. And UUIDs and things you're obviously ignoring.

UUIDs we are. Depends. Yeah. UUIDs we are using for the align cell list option. So, you know, sometimes it's a really good thing to have UUIDs because you can figure out what the user actually changed as opposed to heuristically trying to figure out what the user changed.

Yep, yep.

So, but that's the thing that's under the control of the align cell list option.

Okay. All right, so let's take a look at some of these. So a bio sequence. So do we have some samples here? Do we have examples down here?

I don't remember if there's a biosequence example. Lou, do you have something I could pull up?

My mitochondrion?

No, there is a. There is a biosequence example. Let me just find it.

Yeah, sorry. This is the one that was.

By the way, that should work. That's a bug in. In wolf mouth of pausing.

I mean, sorry, I've played with examples of this myself, but that I didn't keep them.

Why doesn't this work? I mean, and, and. Oh gosh, do I have to do this? Is this what I have to do? Well, that's a mitochondria. Okay, do you want me to pull up mine? I could pull up mine.

Just give me half a second and I'll send you an example.

Well, this is interesting because it's quite long. That's the point. Let's see. Let's just see. Maybe I can pull up somebody else's on the grounds that.

That they're public domain people.

Yeah.

Or they're sufficiently anonymized people.

Who knows? Somebody could target him. If anybody cared. They could.

Sequence some Steven mitochondria and platinum at the crime scene.

Yeah. For example. Or target some weird thing up. What's this? Yeah, okay, but these are toy examples. I want to try a more real example. I want to try an example that is a more serious longer. Yeah, well, that's lovely, but I want to see what it looks like for. For something much longer. Can somebody find me? There's got to be some mitochondrial sequences online. I'm looking at ncbi and it has some. I just. It's in a slightly different format. So just. Just a FASTA file with a mitochondrion is what I need because I can read it in.

What kind of file?

Fasta? Fasda.

Oh, fasda. Okay, sorry, not in my central wheelhouse.

So. Ah, here we go. Well, let's see how this works. This is a mouse squeak, squeak. I have no clue how similar that is the human one. And presumably we have to remove.

So one of the things we haven't looked deeply into yet is performance. Danny Lickblau is actually working on improvements to sequence alignment that might help with performance. Okay, it should help. I mean, his tests show that. Interesting. Speed up. But those changes are not yet.

Okay, well, let's just do this. So let's say diff. Now if I just say human comma mouse.

Now this actually will. I think it will just be a standard string character wise diff. But that's effectively what a bio sequence diff though is. It's really just taking the string data Out.

Okay, so now what does this mean?

Well, so the red is. I'm sorry, I forgot which one was first and which one was second.

I think I forget which one I'm supposed to.

Right.

But the reds are the things you're taking out from the human.

Yeah. And then the green is added from the mouse.

Okay, that's not terrible. Now, what can I do? So if I. If I say something else here, what does character alignment mean that might be.

That gives a more standard bio. Letter by letter conjoining.

It's a bit slow.

Yeah, well, there's a lot here.

Yeah, I understand. But the mitochondrion is actually quite small compared to many things. I see. Graphic design wise, that stem there probably should be. I mean, if we're going to have most of the characters be black, that stem should be gray or something, shouldn't it?

Makes sense. Might it help to increase the line spacing as well between the sort of the, you know, joins.

Yeah, you're right. Well. Or have the stems be shorter? I don't think the stems need to be as long as they are here.

That's a good point.

Right. But I think we.

Right.

But I think it would. I don't think that addresses what. What, no. Is pointing out that, you know, the grouping is a little unobvious and so having more.

I think if those stems are shorter, I think it would look fine. I think you're using characters for those stems. Right. And I think that just needs to be. They just need to be tightened up a bit.

Yeah. I mean, there is actually. You can tell there is a bit of line spacing that's there. It's just not enough compared to what's going on around it.

Oh, look at that. That's nice. It correctly does that.

Yeah.

So why is there such a big. Why is there so much space on the right and not on the left here?

Because we should be fixing line spacing. That's. Yeah, that's Jason. Sorry, sorry. Not. Not line spacing. Line. Line.

Line breaking.

Line breaking. Thank you. Yes.

It's. It's all Jason's fault. We'll just go with that.

But I mean, this is a bug, right? We're not.

Yeah, it's. It, It's. I mean, once identified, it's a very simple thing for Ludo fix. So. Yeah.

Okay. What's character. Should I try?

Probably the same thing happens with any string diff.

What the hell is that?

So this is giving a peek into the raw data from the diff object.

But why isn't there a More button or something here.

Because it's minimal, this could be a browser, but it's currently not. Details will give you the full story if you really want it.

Okay.

Maybe consider minimal the equivalent of short on the details.

Right, but so now this format where it's position arrow. What's happening? But I don't understand this. What does that mean?

It means it. There's something in the first list and nothing in the second list or string. Sorry.

No, I notice.

Notice for example, two down, which is considered an edit where a thing has actually changed from one.

But that's position 808 in string one and 370 in string two. Is that correct?

Yes, yes.

So it. But that format of position arrow value, is that used anywhere else?

Not arrow. But the, the idea of a position is. Yes, I mean that is the format. Yeah, that's actual diffs.

Yeah, I mean that's. That, that, that goes all the way back to, you know, the original command, you know, UNIX diff. And I don't know if, if things.

Predate that, but RCS or something probably predates that. Maybe. I don't know. But anyway, but be there as it may. Okay, so here's a question. Sequence alignment has this thing which I found quite nice, which is string, string and then comma, list, case one, case two. Do we have a format here that looks like that?

We don't.

I don't think so.

Right. I mean, the unified is sort. Right, the unified is sort of that. Right. Only instead of less one, less two, we have red and green.

Yeah, I understand, but what I'm saying is I find it programmatically useful. Like if I do, you know, if I do wherever. If I, if I do this, if I say sequence alignment, What I'm going to get is something probably the same data, but. But see this. I like this format. I. I mean, right, that's.

That which is. Which is. Which is fine in this case, but it is ambiguous if dealing with general expressions.

Right.

This, this is fine because of, you know, strings have exactly one dimension and are.

Fair enough point. So. But to me, this is a sub value that should exist for diff object. If you know what you're doing, you could get this. You should be able to get.

Sure.

Yes, that would be a trivial. Yes, I could, I could add a property that gives you this. Zero effort.

Okay, all right. Okay, let's, let's keep going then. So I think I'm, I'm sort of understanding what's going on here. So I better look at a notebook diff how do I. Where do I have some samples of that?

So we don't have samples. I'm not sure if there are any examples in here, but Lou has some.

There should be one on that page, John, Unless you removed it.

No, I didn't. I didn't even touch the examples, to be honest. When I was reviewing.

Let's scroll down a bit more, focus on the examples.

By the way, these things like diff granularity shouldn't be a string. Go ahead. What were you saying here?

Right, scroll down a bit more please.

And to answer your question or to answer that, yes, we know this is. You know, it'll all get fixed up once we have some blessings on some names.

Okay.

Right, so that last example just above generalization and extensions. I mean it's a pretty trivial notebook. But.

I completely don't understand this output. I mean I would have expected to see some graphical representation.

So click the view as.

Copy it over to my. I can copy this. This is self contained, right?

Yes.

Yeah. Report and then choose like report for example.

Well, that's a bit indecorous.

Can you be more verbose?

Well, I mean it's just got freaking less than signs. I mean.

Okay, so just. Just the notebook expression is what you were.

Yeah, I'm complaining about the formatting here. The formatting could be nicer here. Okay, but. But how do I see this? Okay, what would people expect to see? Look, for goodness sake, we need an actual notebook that we can diff here.

Right?

And I just sent you an input that. That are. There are a couple of actual notebooks in the packlet.

Okay.

And that input will summon them and diff them. This is taken from Edits to the dynamic tutorial I believe that Teo made along the way.

Okay, what do I do here? Okay, this is obviously horrendous.

It is the details.

It is what you would expect from details. Yeah.

Oh, that cell bracket should not be remaining lit.

Well, it's dynamic. What can it say?

Well, I. I think that's a bug that might already be fixed.

That so big, that one. Okay, so these blue mean what?

Right, because this notebook doesn't have all the documentation styles. Right.

So. Yeah, so. So this is one. There is a problem of embedding a notebook inside of another notebook is that we don't have a way to kind of change style sheets on the inner notebook. So you know, when displaying it as we are here, that's. Yeah, that's something that's just going to be missing.

And. And of course that's not even Raising the question of what if the before and after have different style definitions?

Yes. Okay, what does this mean? What does blue blue mean?

Okay, blue is an edit. It means there was a change from the left to the right hand side.

So, so the diff C mode explicitly what that was.

Yes, that is hard.

But why can't you just run the a textual diff on what's in the inside?

That's what that's in the plan. Yes.

Okay.

Just haven't gotten there yet.

Okay. Okay.

For text, that's doable. Yes.

Yeah.

For graphics, that's a different hopeless.

Well, and, and in this case, by.

The way, for a general box data cell. Right?

Yeah, that's the expression diff problem which we have.

In this case, it's somewhere between easy and hopeless because notice there are embedded hyperlinks and box expressions.

I know, I think you can just literally compare them and if they're different, they're different. You know, consider them like word granularity things.

Yeah, yeah.

I mean, similarly here I would have thought you could consider having a top level, just plain list do a same or different, you know, expression, same cube type thing for the elements. You could map that down one level in the list.

Well, there's the expression that the cell represents and there's the expression that the cell is actually made of. The box expression that it's actually made of.

Yeah, I don't know what to do with that. Okay, so.

So, and if you open the report notebook for this, you can actually dig into. So notice the openers on the left there. Right. So this will show you a side by side rendering that does make some attempt to highlight the strings that have changed.

All right.

In addition to the options.

Okay, let's talk about this. Where was the other one? I had, I had another diff thing from that. Yeah, here we go. Okay, so. I think details should be at the bottom.

That's fine.

Possibly even under a delimiter.

Well, I don't know that's worthwhile to have. What does an array plot do?

So I wanted a way to sort of iconographically indicate, you know, the extent, the presence and extent of changes. And this example for alignment chart isn't so interesting, but there are others that are interesting. Notice that in the notebook, in the report notebooks, I use this array plot to give an indication, a visual indication of the extent of the diffs.

Where's the ray plot here?

It could be not every diff. I mean, not every diff supports every format.

I understand, but I don't see why this Couldn't have it could this.

Could have a rate plot. Yes, I agree.

And alignment chart.

Yes.

Okay, well, I've got complaints to make about the. What we're putting in these, in these report notebooks, but they're largely cosmetic construct complaints. Okay. So let's have a little look at a little bit more of the, of the content before we get into that. Okay. Differentiality. Yeah, that's a tendency.

So let me call out one thing which I've. Before we get to the options, let me call out one thing which isn't quite implemented yet, at least I don't think it is, but was something I was suggesting was the use of kind of the equivalence of either, you know, an expression, an object, or a file expression which points to a file of that type. So you know, that you could diff, you know, any combination of a notebook expression, a notebook object, or a file of a dot MB file again.

And that does work now. Yeah, that does work now.

Ah, okay, nice.

What doesn't yet work is diffing a string expression against a file. For instance, a file containing a string.

Right.

But what I like about the thing you just said is that that means I can diff an open notebook which hasn't been saved against one that's saved on disk.

Absolutely, yes.

That's going to be an important feature, I think. I think we'll want a menu item specifically to say, what have I done to this notebook since the last time I saved it?

Yeah, yeah, that's a great feature to have. Okay, so just so I understand, plugging this into git, what's involved in doing that?

So the, I mean, you know, it depends a bit on, you know, whether you want to run it from, you know, have command line git, you know, do this sort of thing through like a wolfrascript type interface, which I think is what we really want to do, but we're not there yet, or whether you want to flip it the other way around, which you will in some cases and be able to do something from within the Notebook interface. And the ideal thing that you want to do is you have a notebook that's in Git, you open that notebook, you make some changes and then you say, show me the changes I just made, or show me the history of this file and the changes between two particular revisions in the history. We're not there yet. It's just. But it, I'm not concerned about doing it, but it's just work. It's just work that needs to be done.

Right. That's thing one Thing two is rebasing workflows in Git. Notebook files with all the metadata changes don't behave well for textual rebasing. And I very much want to have, you know, a rebase that is aware of the notebook file format and can do, you know, can get all the conflicts that aren't really conflicts out of the way.

Yes.

Now, some of this stuff that we're talking about, not all of it, but some of it is going to require Gitlink and well, as you know, Gitlink is we have a thing that works that's good, but you know, we haven't really, you know, the design hasn't been pounded upon. It's certainly not in. Available in the, in the system context, you know, so there's questions about how we want to proceed with that. We've not tried to resolve those questions for this release.

Right, okay. So, I mean, because all the guiness here is obviously not so relevant to.

Well, git. Git has the ability to. So Git has a way that you can plug something in called a diff tool, or there's another thing called a merge tool, which is same idea, but just for merges. And the thing that you plug in can be a gui. In fact, it often is. You know, people often use GUI diff programs as diff tools. So what's. But the, the diff tool is, I mean, there's a bit of nuance around that though, because the diff tool wants to own all of the diff transactions, whereas we really kind of like to just own the, you know, the, the notebook and Wolfram language oriented diff transactions. So that's.

I get it.

Yeah. So we're looking at other possibilities too.

Okay, where's diff apply? Okay, so if I want to turn a mouse into a human, for example, here I would take the diff results.

Well, you can turn a human into a mouse.

So you can see actually it's. What is it the third line down the identity there. So diff apply of diff of old.

Comma new comma O O returns new. Fine.

Okay, so it's important, it's important to understand that identity. But that's not the most interesting use of diffapply. The most interesting use of diff apply is to apply the diff object to something which is, you know, say old prime, you know, where the prime is. It got modified in some other way. And then you want to kind of, you know, pull the thing, stitch the things together.

And what happens if you have, if you want to edit the Diffs, so to speak. Or does that not make sense?

It does make sense and that's not something we've fully baked yet. It is absolutely possible that you could end up with merge conflicts. And that's one, that's two of the failure modes actually right now. And we're, you know, I think that's one of the highest priority things that we want to resolve. But that's not there today though.

Okay, so if I change this 6, 5 and I get rid of the 9 and I had an 11. Okay, I've got some diff here. Beautiful. Okay, now I want to take.

Now. Okay, so take your. Take your old. Yeah.

Change old. I'm going to say diff apply.

Diff apply of percent116 comma and then.

I want to make a change to old.

Oh, right now it's a second argument. Does diff apply work that way? Lou?

No, second argument only.

Yeah, yeah, you're right, it probably should. But yeah, change like you know, I just changed that. Yeah, now see that works fine because there was no conflict.

I get it. I get it. So now. Boy.

But, but if you change like the.

You guys. Oops. What the hell Did I just do that. You guys, this is so deeply physics project. You have no idea.

Good.

This is the story of. This is all about causal invariance, confluent rewrite systems, all these kinds of jolly things.

But Change like the 5, you know, 1, 1, 3, 4, 5. Change the 5 to something else and now it'll go awry. Oh, should it?

Well, what does it do when there's a merge conflict?

Should that have worked? That doesn't.

Well, diff applied you for two way diffs just does index replacement in sort of a stupid way.

For.

But hold on. Shouldn't it complain if it's being. If it's being asked to do something where it's not clear what to do?

Yeah, okay. Yes, I. I was starting to verge on the edge of this problem when I was playing with it last week and I didn't. Yeah, yeah, yes, I agree. I think it should have done a better job in this case and this is going to go even worse. Yes, yes.

What did that just do?

It did index based replacements.

What does that mean? You mean it looked at its list of where things are? I'm confused.

Okay, so Steven, here's Watch the failure mode. Same thing, but just with the first four elements. Delete all the other elements in the list. This will maybe give you an idea of.

Splafaroo.

Yeah. And this is a conversation that Lou and I already had. So yes, it basically it should do, it should do better confirmation.

Okay, hold on, hold on, hold on. What should it do? Should it generate something where there's failure in the output that doesn't seem like an obviously stupid thing to do in this case?

Yes.

Object as the output or the stuff it can do and then failure embedded in it?

I am inclined to think it should be a single failure object with detailed information about the nature of the conflict.

Yeah, I agree.

And, and if there's more than one conflict, then, you know, you should be able to see them all. Maybe, you know, it might be limited size, limited in some way. I mean if there's a thousand conflicts, we don't need to see all 1,000, but.

Right, but I mean your basic point is if it doesn't work, if there's a merge conflict, it just says, you know, failure, that's the output. So in the, in the previous cases, if there are overlapping changes, then it, then it just fails.

Yeah, I mean what I would want or at least have the option to have is something where it applies what it can and ignores, just ignores the rest.

Well, how can it ignore it? What does it do? It leaves them in the form.

It just doesn't apply. That piece of the diff. Right.

Stephen, when we were looking at the details, remember the left hand sides of the rules were the positions and the strings or the positions in the list. If those positions don't exist, then yes, that could, that particular hunk could just be skipped if either the original position or the target position don't exist.

And what I'm suggesting is that it would be a little more sensitive to that. Not only detecting position, but detecting a context window. You know that the, within the context window it actually matches what it needs.

Something like a merge conflict action option. Is that right?

Yes, yes, that's certainly what I'm arguing for.

Right, okay. All right, so. Okay, fair enough. All right, let's keep going here. So that's diff apply. Now how much. What does diff apply have? Where's. I haven't looked at.

So you might be interested to know, by the way, that git merge part of Gitlink has, this, has this notion of conflict functions and the conflict functions are a bit more nuanced because it's detecting, you know, oh, we're, we're dealing with a particular file type because of course git might be dealing with a bunch of file types. And in the case of this particular file type, then, you know, then look at doing merges in a particular way.

I see.

And we actually do that so that we can do, you know, like notebook files can be merged in, you know, using our notebook merger as a. For example.

But so you've already got some failure conditions there listed under diff apply here.

Yes, those are aspirational. Yes.

Yeah, they're not implemented yet. But this was. I, I was just writing this up last week.

Okay. All right, let's look at diff3. Do we have examples? Let's take a look at diff3 here. Okay, well, so naively. Let me just go ahead and do this. I'm just going to say range of 10 comma. Okay, let me start messing around with this. It's very wide. Wait a minute. Who's the they and the I and the you and the them? I don't know. It doesn't seem right to me.

This is the standard nomenclature or so.

The first, the first one is. The first one is the common. Right. The first arg. I believe it's in the order that you're providing them. Right. And Right. So the, the common ancestors, the first argument, quote, unquote, whatever. Our version goes in the second slot and you know, the. What other people did. Right.

Which.

So for example, in a git example, right. If you have your local version and somebody pushed a change to the server, then you want to. Right. Then there's obviously their version and then our version and whatever was the common ancestor of those.

So question, has graphic design had a look at any of these interfaces?

They, they have, they've commented on colors but not much else.

I, I don't like the fact that it's so loud. Common ancestor. Our changes. Their changes. Because it's kind of a weird, you know, I mean it just, we could just bring it down, I think.

Billy. We have Billy here.

Okay.

Yeah. Okay.

So I mean just. I don't know what you want to do to bring it down, but just make it, you know, something you can see and you can see it's clearly different than the data. But why are these columns so wide?

Well, I mean they, they only look wide because you have one digit elements in them. Imagine that you were diffing text, you know, lines of text. Then they would probably be too narrow. I mean this is a frequent problem with three way diffs though. Is that usually in order for them to be useful, like you're maximizing your window to the entire monitor.

Yes, I've seen that. Explain this to me. How does the array plot work?

So reading left to right, row one is you Know, first element, second element, third element, and then the next row is fourth element, fifth gray means no change.

Okay, so. So we wrapped it around with sc.

Yes, yes. And I chose a dimension for the. For the plot that seemed sensible for the number of elements.

And.

But is there any indication of ours versus theirs here?

No.

Okay.

Which will be good to do, I think. But, you know, another thing might help is by putting a scale on the side. Given that you do have a scale of, you know, positions. Right. You could indicate that this is going up by strides of three, so to speak. Do. Do you get what I'm saying?

So you want some sort of axis. Axis.

I want something. Some numbers here that indicate that this is, you know, one through three. I don't know exactly how to do it. Maybe there's a three there, maybe there's a six there, nine there. Something like that.

Okay.

To indicate that this is.

So my hope was to have an iconic view of the diff that didn't need explanation. I think if we get to the point where we need explanation, we may have. We may have missed the mark.

Okay, fair enough. Well, I'd like to see my. Can I see my.

Right.

And if. Right. And. And for a large diff. Right. There will be no room for numbers there. Right.

But I do feel like. I mean, the fact that there are no grid lines or axes or anything, it can feel like it can be a bit. It might be a bit difficult to even know what the size of a unit square is.

Okay, so look at this puppy.

Right?

So imagine that as sort of the icon in a summary box.

That.

That's sort of what I was aiming at.

Blue is an edit.

Correct.

So is that standard? Green? Blue, Red.

Well, I think green and red are standard.

Yeah. Blue, I think. I think this blue came from design. I don't remember for sure, but there isn't really a standard change color. Also. Also another thing that we've thought about is dealing with people who are colorblind. And there. There are, you know, programs out there that have colorblind schema, but I don't know if the. The choices for colorblind folks are very standardized either.

Okay, why is there extra space at the bottom there? Bug.

Buglet.

Yes, my speaker. Yeah.

All right, so this looks pretty reasonable to me, but I don't understand why I see there. All right, fine. It's a certain aspect ratio that you've picked. There isn't an aspect ratio based on the width.

I mean, it's. Yeah, it's. It's. It's Just one big long string. So there isn't.

But this is quite nice. I mean, this, this is useful to me.

I agree.

Okay.

It would be really nice if there was some indication that it's working as opposed to just sitting there dead as a stone.

I.

Okay. And that's a very big alignment chart that's not very happy with. Okay.

I mean, I can describe what alignment chart's trying to do.

Oh, I know what it's trying to do. Okay, Sensible. All right. What about a three way alignment chart?

I wonder if in this sort of case, instead of showing every single alignment, we want to, you know, do every nth alignment when it starts to get like this.

Yes, Okay, I agree. Okay, so let's just look on here. Okay, so I think I'm getting the picture here. It's. How does this relate? You know, like, like we've had this notebook history stuff, right. When we come to notebook diffs, we have a bunch of things in notebook history where it's kind of nice because you can click on something and you go back to that place in the notebook. Do we have anything like that here.

In the report view? Yes, if it. Now, now keep in mind, you know, this presumes a notebook that is, you know, that is either openable or already open. If you have a notebook expression.

Yeah.

Then it's not necessarily been instantiated anywhere.

Okay.

All right.

So I mean, a workflow would be generate the diff report, then I'm going to go modify my notebook to either apply those diffs or whatever else. I mean, is there any possibility of it? Is that sensible? I mean, in other words, I want to see what changes I've made. I'm trying to align some of my changes to some changes.

Well, so as a. For example, you know, what our DQA team will want to do is they'll want to see a review of all the diffs since the last time they looked at a notebook. And if, you know, there's something that's kind of not obvious that needs further review, then they'll want to go to that section so they can review it fully in context.

Right. Okay.

And yeah, these things. Sorry, remind me, Lou, these things are basically hyperlinks, right? The, the cell content changes.

I'm not sure exactly what.

Sorry. In the report, there's chevron at the end of.

Are you talking about the file names at the top?

No, I'm talking about individual changes. Individual cells. Okay. So yeah, the. It could be clear, but in, at the top there, where it says text cell changed and there's a chevron, which maybe we need to make. Maybe that needs to be a bigger story.

Well, it's also not.

Well, the notebooks hadn't been open, so it needed to open the notebooks first.

Yeah, but that's completely unclear where this selection is.

Yeah, you know, changed.

What does that mean?

Oh, that's tried to make it clear for you. Also, something that would help for percent modernization. Jason Harris added a self flashing mechanism and maybe we might want to use that here as well.

Sounds good to me.

As long as it's not too visually disruptive.

So long as it's not too flashy. He's trying to say.

Well, the interesting thing about it is that, like, if you miss the flash, you could click it again and it'll flash again.

I see. Okay, there's some questions here from Live Stream. Xander asks, what about sparse array? What does it do?

Nothing at the moment.

Differing between a sparse array. I mean, that's, that's like differing between a file and I mean, differing between different. You know, that, that's kind of the, the mathematically aware diff, which I think is a more complicated thing. This is very.

Yeah. And, and this is why I, you know, I put some thought into how to organize a documentation and this is why I started the documentation up front by, you know, structuring around what is it you can diff? Because we can't diff everything yet.

Right. Well, and some diffing is, you know, can you diffuse different representations of the same algebraic expression? By same, I mean same in value? You know, it's very complicated. So there's going to be, I think syntactic diffing is the only thing we can expect. And then the question is, how broad can the syntax go? And does a sparse array, is it equivalent to an array or not?

Well, and also I didn't, frankly, I didn't think about this. Well, so the, and this is something I didn't really drill down into, but there is a third argument format which, which I did not properly make sure was. I didn't put a table for that in. But. But yeah, I mean, you know, for two. Given things, especially in a world where we're supporting more types, one can easily imagine that, that, you know, the types are going to be different. Sorry. That you might want to, you know, take two expressions and get two different kinds of diffs out of them or more.

Yeah, that's complicated.

It's. It's certainly already the case for strings, because for strings it is already the case that when you're Diffing two strings. You might want to get a character diff, a word diff or a line diff.

Yeah, yeah. Okay. Ethan asks here, can you diff files on metadata that seems useful, like diffing GIFs on EXIF data, for example?

Yeah. So right now we don't have any understanding of beyond the formats that I outlined.

That's the thing for the future. And that needs to be.

But that having been said, there is already kind of a peek at what metadata diffing might look like in the diffing of cells and notebooks, where a cell or a note or cells in a notebook might only be different by option settings and not by content.

Indeed. All right, let's just look at this notebook changes. I'm not a fan so far. I think this needs to be much more graphically. Okay, so let's look at this case here. Look, if it's old and new, for goodness sake, say old and new, not first and second.

Sure.

See what I'm saying?

Yes.

I think this should be a little bit more designed, you know, this, this should not be a notebook changes. Doesn't have to be. I mean, that's kind of a header, but, you know, it's a notebook change thing. You don't need to say big blaring title notebook changes.

Well, if you save it and then open it some later point, then you might not know that.

All right, fair enough.

Well, I mean, actually, I, I think we do need some kind of announcement because you're potentially. Especially if you start clicking on things, you're going to have three windows on the screen at a time.

Fine. I, I want, I would like to see something more stylized here rather than something that looks like a type.

That's fine. That's fine. I'm. I'm. Yeah, I. We've not put as much effort into the design of this as we could have. And I. Okay, I agree. We should.

So old versus new. And when it's a notebook expression, it needs to be some graphical thing that indicates that, not something with textual angle brackets that looks like it came from 1988. What does that summary?

Well, I mean, it could be nothing more than an iconize of. Yeah, I think what an iconized of notebook expression would look like.

Yeah, that'd be beautiful.

It might even be. I don't know, we might even. If it's going to be iconized, we might even put the actual thing in there so people could copy and paste it later.

Well, that would be even more lovely.

Yeah.

And why don't you just Say old and new. Why don't we call this notebook diff report?

The. In the documentation I refer to original and new. But we can go. I, I'm. There's nothing here that I'm, I'm. You know.

Well, let's decide what it is with three way diffs as well. Original. Go ahead.

Users aren't prohibited from diffing in the other order notebook files. You know, their age is irrelevant.

Okay, but good point.

Well, in fact, in fact, sometimes people actually, even if there are things that have an age, people may. Might choose to diff in the opposite order.

Okay, so what would you like to call the two things?

Yeah, well, and you saw what I used for diff3, which kind of faced this problem as well, right there.

Common ancestor, ours, our changes, their changes.

Well, and in, in the, the usage I referred to it as ancestor edit one and edit two. But yeah, we can.

Well, version one, version two or something. Version that. That has an ordering again.

Yeah, well, I can certainly flip it to ours and theirs. I don't have a problem with that.

I hate that. I hate that. I hate that. That's incomprehensible. I mean, I'm not sure that old and new gives a temporal sequence. I mean, you can diff. I see what you're saying. Original and changed. No, but you might just want to diff two things that are both on unequal.

I mean, like in the, in the diff usage message, I referred to them as original and new.

Okay, but look, look. The mouse versus the human mitochondrion. Which one's original? Which one's new? Which came first, the mouse or the human? You know, I don't know. I don't actually know.

Historically, I think probably the mouse, but.

They could be wrong. But humans go back longer. I mean, depending on the strain. Okay, Okay. I mean this, this thing. By the way, the heading absolutely should be notebook diff report. Surely. Because that's what you asked for was the report.

Yeah, that's reasonable.

Okay, so now. Why not say instead of other changes, metadata changes. Metadata and option changes or something just to be clear what the heck it is. Otherwise you'd just be completely confused.

Okay, yeah, I agree.

And then content changes. I think you don't need to say notebook content. Yeah, and then, you know, those should be, I would say much more, you know, like they could be boxed or something like that. I'd be perfectly happy to see box, box, box, box going down there to indicate these with some button in the box, basically that goes to, you know, that does the hyperlink opening thing.

If.

You'Re pointing at things. My zoom is not.

I'm not, I'm not. No, I'm not. I meant these. I meant these here.

Okay.

I think this just needs a little bit of a, of a design effort and then I really want to understand the workflow of how I use these. So you're saying dqa, for example, would use them by just saying, you know, what changed, so to speak?

Yeah, I mean, in DQA's case, they're not going to be doing any diff apply sort of thing. What they're going to be doing is just using it as a farm of hyperlinks for them to go look at the changes.

Yeah, it's super useful, basically.

Yeah.

I have no idea what they're doing right now, but this is obviously super useful. All right, what else do we have to cover as we're a bit over time here? What else do we have to cover right now?

So, I mean, I think the main thing is we could go a bit through the options of diff if you wanted to and see how sensible or insensible you think they are. But if you want to postpone that to another time, we could schedule a follow up as well.

I think we're going to need a follow up for some of these design things. But let's just take a quick look at the diff options. Where did I put them here? Okay. I think we should call it the diff display. The overall size of the diff just seems crazy.

By the way, it's interesting that they're kind of. These options fit into two broad categories. One broad category is the, you know, how to display the diff and the other broad category is how to do the diff.

Sure, I agree, but I think you should be clear in the description.

Should we have two separate tables for that? Is. Is there. Yes.

Precedence for that?

There is. Good.

Yep.

Good, then I'll do that.

Okay, but I think you should say the diff display when you're talking about.

Okay.

A list of data types that should be ignored in doing the diff, presumably.

Yes. When calculating the diff.

Right. So align cell lists. I'm unclear what that is.

Okay, so align cell list is only relevant for doing notebooks. That's another kind of weirdness is some options, they're not relevant for many of the cases. But if you're doing notebooks, the question is how do you figure out kind of how the cells line up when the cells might be different from one another? Yep, there are two useful options. And one kind of useless option. At least I think it's useless. The two useful options. One is to use expression UUIDs. Okay. And so any just looking for where they're matching up. The second is to use heuristics based on edit distance to determine similarity.

I get it.

And then the useless option is to just, you know, do an index count from you know, of. Of the number.

Super confusing that alignment has to do with where things are positioned on the page and the line cell lists has to do with kind of.

Well, and so breaking it up into two tables will help that.

Yeah, I understand. But I think this name align cell list is too weird and I think.

Okay, should it be a method option? Is it. Can we relegate it to that level of.

No, I don't think so. Particularly I. I mean I think it's going to be something people care whether it's UUIDs or not. I could call it something like.

Uuids will definitely be the preferred case for people working in git but there will be cases where people will fight. It just gets in the way.

It's very charming. It's very physics projecty. You can either have two hypergraphs that have been, you know, have different origins but happen to be isomorphic, be merged or you can insist on having lineage.

Oh, okay. So the. One of the big cases, just to give you some context where you would want to line cells as line sellers, not to use expression UIDs is for notebooks that are generated.

Yes, yes. You have two generation notebooks, they got different UUIDs.

Exactly.

Okay, well I mean I think, I think this should be something like diff. It should have a name that makes it clear. Something like diff alignment method or something.

That's not terrible. I wouldn't be totally ashamed of that. Okay.

Base style specifications for the diff. Is that display based styles?

No, sorry Lou, you remember this better than I do. Can you explain this?

So it's a way to specify the display. You know, if you don't like green, you want to use your own green.

I get it.

But. But it's not standard notebook options which is why we didn't just call it base style.

I understand but I mean diff display style. And how does that relate to appearance? Why don't you just have one option and just, just.

Okay, so appearance is. And. Well, yes, it could be sub keys in appearance. Yes, we could do that. Appearance is basic right now is basically just, you know. Is that drop down selector.

Is which one it is?

Yes, like the named view but we could.

But we do. We do have precedence for having appearance, being able to specify kind of, you know, multiple vectors of appearance.

Well, I think this is going to need another meeting. I need to go to something else.

Okay.

All right.

But we've already got a decent amount of changes that we can work on and bring to the next meeting, so that's good.

Sounds good. And bring some more examples next time. Okay, sounds good. See you guys soon.

Thank you.

Bye.