Modern Notebook Diffing

Okay. Hello everyone. Good morning, good afternoon, good evening. My name is Lou d'. Andrea. I worked at Wolfram Research for a while now, and today I'd like to talk to you about some diffing tools that I've been working on. So in the beginning, when we first started thinking about diffing many years ago, we built something called sequence alignment, which gets us sort of the first step towards what we need for diff. It finds optimal alignments between lists and strings. So here we have a couple of strings, and sequence alignment sort of finds the common bits of the string and then the bits that are different, and here's more common bits and the bits that are different. And it did this for strings and for lists. So here we have list case, and this is slightly more difficult to visually parse. So we have bits that are different and the bits that sequence alignment thinks are aligned, and then things that were added aligned, things that were removed, and so on. This was a wonderful first step back in 2008. Alignment is a very important part of diffing, especially strings and lists. And sequence alignment has sort of a rich infrastructure of different methods and options that one can use to determine how to align things. However, as I mentioned, you know this is not the most obvious structure to either visually parse or to computationally parse. Also, sequence alignment only accepts only was able to compare lists or strings. And in fact, some sequence alignment outputs aren't even sufficient to tell whether the two things that were fed to it were were the same or not. I will leave this for you to ponder afterwards if you'd like to get this notebook. By the way, the handout is in the usual place for handouts at this conference, but. So enter Diff Tools the packlet. So we have just in the last week released this Diff tool, Wolfram Diff Tools, as a packlet on the packlet repository. It is a prototype. It is experimental, but it is fully functional and I encourage everybody to install it. All you need to do to install is run Pacman install of Wolfram Diff Tools. Once you have it installed and you call needs, then it exposes a bunch of new functions into your kernel session. The main such function is diff. So Diff takes two lists. The these are the same lists that we fed Sequence Alignment, but it displays the result in a way that's sort of much more clear what's going on. It's more of the kind of difference view that you might see in other diffing programs. And this also allows you to see different views of this data of this diff. So we can look at sort of a visual summary of the diff, or we can look at, we can look at a columnar view of the diff. This was the first list, this was the second list, and this is how diff thinks that they should be aligned. And likewise for strings, diff starts off where sequence alignment left off, with strings and lists, and adds a healthy dose of viewing these things nicely. What else can be said at a high level about diff? So it returns a thing that is a computable object. It looks like an interface, but this is actually a diff object. And it happens to give access to various views, but it can also be used in computation as an object. And I'll show that in just a second. Diff starts, as I said, where sequence alignment left off with lists and strings, but it then extends to diffing of files, diffing of cells and notebooks. Especially important for our purposes and lots more to come. And I'll talk about what the directions are that we care about towards the end of the talk. And of course, cells is an important case because once you have differencing of cells and visualizing differences of cells, well, cells can contain arbitrary expressions, so then you effectively have differencing of arbitrary expressions. And we'll see some examples of that. So let's start with diff objects. So the output of diff, even though it looks like an interface, is an object. So here I'm diffing the options of sequence alignment with the options of diff. And we see there's no red here. So all of the options of sequence alignment are in fact options of diff. And then the green bits are the things that are new in diff. And we assign this to an object. And what can we do with that object? Well, we can query its properties. So here are the properties that we currently make available for diff objects, and you access those properties in sort of the usual way. We can extract the summary string as one of the properties and we get the string that summarizes the diff. So that object can also be viewed directly in different ways by the diff view function. And the formats that are available for a particular object change from object to object. So we have the way to query what are the formats that this object supports being viewed. And here we see the same listing that we saw in the interfaces above. And we can, in this case, I'm going to grab the side by side view directly and display it outside of the chrome of the diff object. This is sort of a disembodied version of the diff. So that's useful, a way to use the diff object as computable thing. Another way to use it as a computable thing is to take that object and use it to transform other expressions. So in this case, we take the object and we take the starting point and we apply the changes from that starting point encoded in that object and we wind up with the result, the options of diff. And in fact we see that the result of applying that object does in fact give us the same things that options of diff would. So those are two way diffs. Three way diffs are somewhat more interesting. And this is a trick that sequence alignment cannot do. So this is very common in source control systems where you have a single version of content, a file, whatever, and you have two different people, or possibly more than two people, making different edits to that content. And you want to know, can all those edits be merged back together? And this is for those of you familiar with git, this is the merge operation or the rebase operation requires this kind of operation, this three way diff notion. So here we have, well, the function that does that in Our package is diff3 takes three arguments. The first argument is the common starting point. So here we have 1, 3, 5. Here somebody made an edit to this list by adding something between the one and the three. And someone else made an edit to this list by adding something between the three and the five. And of course those differences can be merged together. There's no conflict in those changes. But sometimes if somebody, if two people are adjusting the same piece of an expression, there can be conflict. So if we start with 1, 3, 5, here we have someone added something between the 1 and the 3 and they change the 5. And here someone else added something different between the one of the three and they changed the five in a different way. So both those additions and those changes are conflicts. So, and they're indicated in this conflict view with this yellow background and the warning triangle sign. So diff3 is the thing that will support all of the merge operations when we get there, right? So that's diff object and diffing of cells, diffing of lists and strings. Let's look now at diffing of cells. So some cells look a lot like strings. In fact, some cells really are just strings. Here's an example of some two cells that aren't quite just strings, because there's some styling in here. So what do we do when we're diffing cells. Well, if we had a cell object reference to these cells, we could diff the cell objects. I'm going to, here, I'm going to grab the cell object for this first guy and I'm going to grab the cell object for the second cell and we diff the cell objects. And the result is a view of the changes. Well, the result is a diff object which displays as a, a view of the changes between these mostly textual cells. So it shows that, you know, maybe the, the eagle eye among you have already noticed that, you know, you could see some of these things and, but, but here we lay bare what the changes were in the content of the cell. And of course, cells are more than just content. Cells have styles and they have options. And this view also lays bare the changes in, if any, in those sections of the cell expression. So for cells that are mainly text, you know, this is kind of a nice view of things. For cells that are mostly not text, what can we do? So here we have, you know, we want to effectively diff two expressions and we want to diff, you know, diff input function is just a convenient way of saying, you know, apply this function to the two inputs. And in this case that input, that function creates cells. So we're diffing the output cells of these two cases and we can do a pretty good job of highlighting the differences in exactly the same way as, as the text case did. And so we have, you know, the bits that have, so let's see, the differences in the input were, you know, the. How does the 2, how does it change from 1 to 2 affect the output of these integrals? And we see here that it affects it by, well, there's unsurprisingly a factor of two out front and maybe some sign changes. Right, so this is good as far as it goes. The we haven't yet cracked the highlighting of at deeper levels of an expression. So at sort of the top level of an expression we see this highlighting sort of gives nice tight markings. Here we have a fraction which is different from one to the next. And rather than highlighting the whole fraction, of course we would really prefer to highlight just the point difference which in this case looks like the one has a different sign in these two guys. But currently the highlighting sort of stops at the fraction box level where, you know, we're working towards getting it to rodentify into these guys to give a much more granular diff marking, but we're working on that. The other thing I wanted to point out is Once you install this packlet, not only do you get the kernel functionality, but you also get a bunch of palettes. Well, a couple of palettes for a start. And one of them gives you a way to compare cells without needing to grab their cell object beforehand or extract their cell expression. I'm going to point at this cell and say, I want to care about this. I'm going to point at this cell by clicking on it and say, I want to care about this. And as soon as I select that, we see that the diff appears right there in the palette and you can change the cell. Maybe we want to look at these input cells instead. And sure enough, now cell OBJ1, cell OBJ2. They're different. So for people who don't want to be writing code, just want to sort of get on with their lives and compare the cells, I'd encourage you to take a look at the palette. So that is cells. Now let's look at notebooks. I'm going to open a couple of notebooks in the background here. So in some sense, a notebook file. Any of you who have used notebooks have seen notebook files in source control, they're just a bunch of text, and when you diff them in source control, you just get a plain text. If the diff package can do. If you say, I care about these things as lines of text and not as notebooks, then it'll go through and give you the same kind of differences that you'd see in your git client or your CVS client or whatever. You have all this difference in metadata. You have changes in text lines, some of which refer to whole cells, some of which refer to pieces of cells. And it's hard to make sense of this. It's hard to visually parse this. I've developed another view of this diff of diffs, generally that I'm calling array plot, that sort of summarizes the changes as a grid of color. And so each point in this array plot, each pixel represents, in this case, represents a line. So first. So we Read across line 1, line 1 of the array plot and indicates the first, I don't know, 30 or so lines of the file. Looks like there were a couple of changes in the first few lines. There are some other changes. There are some additions. And here at the bottom of the file, we see the churn, that is the difference in the metadata, the file outline cache that gets changed every time the front end writes out a file. So that's the old world, right? That's diffing Notebooks as lines in the new world. We have deep knowledge that the notebook is actually a Wolfram language expression. And a Wolfram language expression is at its core it's a sequence of cells, it's some grouping operations, it's some options. And so when we diff two notebook files by default, what we get is a diff object that knows about what these differences are. We look at the array plot. In this case we see, ah, well it noticed that there were three cells that were edited and two cells that were added. How do we view those things? Well, let's see, there is a report form for the notebook diff that shows and you don't really need to look at too much at the details of this, but just know that it exists. We have sort of a replication of the array here, of the array plot and then we have the itemized differences so we can look at them in the report. We can say, oh, this cell changed. How did it change? It looks like somebody edited it. That's fine. If we want to see it in the original notebooks, we have the ability to view these notebooks side by side and we see, ok, those are the cells that changed. If you cared about looking at them, you know where they live. And here were some cells that were added. Where were they added? Well, they were added right after this dynamic module example. Okay, that looks fine. So you know, we have sort of deep knowledge of, of where things happened, went wrong. We have the ability to drill down into individual changes to view them in detail. And we see that this notebook in fact didn't have any changes to grouping or option or notebook options. So this is the kind of report that is going to be very useful for notebook differencing going forward. And we have lots of options to adjust the kind of differences that happen. So for instance, we have the ability to ignore certain options if you don't care about whether your font size has changed or what the version of the front end was that saved the notebook. You can ignore those automatically. And you can also of course take that object and use it as a computable thing to to patch existing notebooks. That functionality isn't quite there in Diff Apply yet, but it will be there soon. And finally, I wanted to also point out that notebook differencing, like Cell differencing, is available through a palette provided by Diff Tools. So here we see we can create the same diff report by comparing these two open notebooks. And sure enough, there's the diffuser, or we can choose to compare with notebook files which turns out to be. We can compare notebook objects with notebook files. Turns out to be very useful. I'll talk about that in just a second. So that's Notebook Diff. One of the biggest problems we encountered when developing Notebook Diff was how to determine whether cells were aligned with one another. So in other words, if you have a changed cell, you have an added cell. How do you know when whether a cell has been added or changed? Here is a very simple academic example, just to demonstrate the point. We have a title. This is before edits. We have a title and we have a subtitle which was in the wrong style cell. But that's fine. Okay, you can do that. And then somebody who came along and edited that, they didn't touch the title cell. They added an abstract and put it in the proper style. And they noticed that the this subtitle was in the wrong style cell. And so they changed that and they also made an edit to the cell. So how is notebookdiff supposed to know which cell corresponds if either of these change either of these cells, which were not in the original notebook, which of them should correspond to that cell? And in fact, the front end has been tracking such information for a while. Back in version 11, we started adding expression UID to every cell. Every cell, when it's saved to a file, has an expression UID associated with it. In fact, this isn't just when it saved the files, it's when the cell is originally created. It's there, but you can't see it by default with most interactive gestures in the front end when you're working on the file. What really is happening here is we have a cell with an expression uid, a subtitle cell with that wrong style that got an expression UID and the UIDs are really longer strings. I'm using an abbreviation here, just so that's visually something that you can parse. And then afterwards we see that the title cell is the same. The abstract cell is a new cell with a new expression UID that didn't appear before. And the subtitle cell is an edited version of this abstract cell. So in this way we can align, we can carefully align cells with one another. And this really helps. There's a little bit about this in the documentation for Diff Tools, which is available on the Packlet repository. This really helps with figuring out notebook diffing. Ok, let's look at three way diffing in Notebook files. Because notebook merge is one of the biggest problems that people have in source control. In my experience so here we have three versions of a notebook, an original notebook, a notebook that Alice made some edits to, and a notebook that Bob made some edits to. And the question is, can we pick up all those edits? And in the old world, again, just to drive home the point, I have diff three of these notebook files and we look at them as text lines, as files that contain just a bunch of lines of text. And we see conflicts immediately. Let's look at the array plot of this. We see conflicts right in the first few lines and we see conflicts all over the place. When the file outline caches differ, the content looks fairly mergeable. But it's impossible to know from a textual diff whether this conflicts are real or not. So by default, without that option, we diff three notebook files. And can we get rid of the conflicts? Well, in this case, no one of them was a real conflict. What was the real conflict? Let's look at the array plot. And we see it was that very first cell is in conflict. And if we look at the cells, we see the original notebook had intro to dynamic. Alice changed it to getting started. Bob changed it to introduction. So they made conflicting edits to the same cell. Something has to give. A human could be in the loop to resolve this conflict. Or maybe we have some auto resolution that says, well, Alice's changes always trump Bob's. So we could develop functionality to do automatic merging and conflict resolution, even in this case. In fact, we've already built such technology in Gitlink and it's just a matter of retrofitting it to work with the Diff Tools model. And that's something we absolutely plan to do, but it's not in place yet. So. Right, that's three way notebook diffing. I wanted to show a couple of other examples. We talked about diffing notebook files versus notebook objects and immediately that gets you to a place where you can say, you can answer the question of what have I edited so far in this session? So I just saved this notebook and I'm going to add, I don't know, I'll add an ellipsis to that section cell and it's not saved. But I'm going to compare what was in the file with what is now in the notebook object and it says there's exactly one edit a section cell has changed. And there is the change. Well, isn't that neat? And I think I want to drive this point home by saying I'm going to switch over to the working mode in my session in my in My presentation not save it and rerun this diff. And now we have the section cell changed. There was an output cell that wasn't there before and some notebook options have changed. Look at that. The screen style environment is different, so very useful. I can easily see a world in which this functionality makes its way into the main menu bar somewhere. I also wanted to show sort of a more application Y example. In our Source Control internal to Wolfram Research, we have access to the legacy versions of lots of notebooks. We've been using Git to store notebooks for a long time. This is an example of one of our style sheets that's been in git. And this notebook history function is just a utility I wrote that extracts the value, you know, extracts these files from Source Control and then compares them pairwise with one another. Default is about 250 cells, and I've chosen to go back about 30 odd releases or at least waypoints in our Source Control system. All those diffs took about two seconds in total to run. And giving us a visual indication of what all the changes were. It looks like in 12.0 we added some stuff. 12.2, we added some stuff. When I was showing this to a colleague, his question was, well, what are those changes? I added this button that actually goes off and opens the report for each of these guys. And it looks like the 12 cells that were added for 12.2 were something about Echo before and Echo after. Fine. And for larger notebooks, here we have core NB, which is about 1500 cells over that same number of releases we have. It takes less than 30 seconds to get all these guys, and we can see sort of a visual history of Core over that time period. Okay, so all of these things are available right now in the Diff Tools packlet. The Diff Tools, I'm running this in version 13.3. This is the shipping version that's been out for the last, I don't know, six, eight months. This will also work even better in 14, but all this stuff works in 13. I'll remind you that you can just pack and install it, and once you've done so, you'll have access to the docs that are displayed in the Packlet repository locally in your system. So you can go to the. All right, you can go work through the introduction and see all of the things that I don't have time to talk about here today. Where are we going with this? Well, lots to do. One of the things we want to do is better integration with Control system. With Source Control systems. And so the main route to that is to package up a lot of these utilities as well from language script files so that you can automatically access them from your git clients, your CVS clients, etc. We're going to be adding native diffing for more and more kinds of objects, including associations, bitmaps, et cetera. These things are going to have different views, they're going to have different kinds of data stored in the diff object, but they're no less important to have sort of native comparisons of. I mentioned planned improved visualizations for cell changes. This is especially true for things that the cell wouldn't otherwise be showing. Of course, once you have the diffs, the question is you might not want to apply a whole diff, you might want to apply just a piece of a diff or decide, you know, at application time. Yes, yes, yes, no, no, no. And that's definitely something on our radar. I've already talked about automating, conflict resolution, etc. Okay, so I will repeat one last time. Diff Tools is available now on the packlet repository. Here is the link, here is the way to install it, and I hope that all of you go grab it and take a look at it and see what you think do let us know what you think. This packlet is marked as experimental. The hope is that this functionality will be folded back into the Wolfram kernel at some point in the future. But for now we wanted to get this out to you sooner rather than later. Thank you for your attention and I'll turn my attention to the Q and A to see if there was anything that I can address in the last few minutes. And if I run out of time, I'd encourage all of you to go to the front end office hour, which is starting immediately following this talk. And if you have questions on diff or on other things related to front end functionality, we can pick up the conversation there. So let me see, what is your imagined workflow for using this with a notebook stored in a git repo where you don't have different versions at the same time? Well, if you have a git repo, then you actually do have the different versions, or rather the git repo has the different versions all at the same time. And that's how the example that I showed works. It goes to my local repo that was a fully local operation. It went to my clone of the repo and extracted out of the repo information the bits that correspond to that file at various points in history. That's I think something that will be straightforward. We can do inner token diffing difference between foobar bud and foobar bunch. Just highlight the n if yes, I think the question is the difference between the granularity of a string I'm sorry, granularity of characters of a string versus words of a string or tokens. And we definitely have the infrastructure to do that. String differencing already has the features to do that. We haven't yet expanded those features to cell difference to the view used by cell differencing. But you definitely will be able to say I really care about the character by character highlighting of differences and not the and not anything else. And then John mentions extending gif get diff and merge with drivers. That's an interesting thought. Possible to export diffs from a notebook as a wls so I can pass them to git. Sure. I mean the diff object is maybe I haven't driven this home solidly enough, the result of a diff, even for notebooks, but I'll this is easier to type is a diff object. If I start typing around here, you'll see it actually converts to the display of a diff object and it stores the changes in a way you should treat this as a black box, but it is an object that this is the thing that gets applied when you do gif apply. And sure, this can be stored in source control if you'd like. Absolutely. Okay, that does it for the questions, and I hope to see many of you at the front end office hour coming up next. Thanks again for your attention and I'll talk to you soon.