Notebook Assets

Hello, my name is Ian Hutchnicki, I'm a user interface architect in the user Interfaces group. As John mentioned in his talk, which is probably just ending, I'm going to talk a little bit about some preview functionality that we started putting into the product. It is an early preview state, but we're looking to get some feedback, see what people are interested in doing with it. Some problem points they might have with is to solve a very naggling problem that we've had for a number of years, which is I'm building this application document and it has some dependencies on some external data and moving that with my notebook has always been problematic. It dates back many, many years. It came up a lot during the web browser plugin days and typically redistributing it meant bundling, packaging it up, bundling it into something, sending it off to users and then explaining to them how to undo what you did and exactly how to do that, which is always a bit of a pain. It's not just click and go. We'd like something a bit more plug and play. So to do that, a lot of the times you'd end up burning your data into a notebook directly, which is a bit problematic because it is not as efficient as just including the raw data off to the side. Primarily because a lot of the times your data is binary, which is arguably the most efficient format that you could use versus a notebook's inherent plaintext format. So it incurs an encoding overhead penalty when converting that into text. And then specifically in the case of images, which images are burnt into notebooks, you have format differences. So an image would on disk be compressed via some codec and binary and in a notebook is going to be uncompressed raw RGB quads. So you have format differences there as well. And that is a non trivial difference in the case of images. And then of course on top of that you have computation time, the time that it takes when you read a notebook in or, or serialize it back out. How much time is spent doing that conversion between text and then binary. So notebook assets are our solution to this particular problem, which the goals of it are to simplify redistribution of notebooks with external data. We would like to keep binary data as binary because it's more efficient and then it can actually provide some functionality so we can start doing data reuse inside the notebook itself. That's more of a long term stretch goal, but nonetheless it is enabled once we start doing this. So for 14, what do we get? 14 is basically going to revolve around programmatic access. So how can I modify the asset collection for a notebook? How can I enumerate them and how can I work with them? And this talk is effectively the documentation for 14. All of this is of course in the experimental context. It is alpha like at the moment. So some things will change. I already know some things will change and I can't guarantee that everything that you do today will continue to work going forward. But it is in a working state to start off with, the first thing you have to do is add to create an asset. We have asset add, which takes a name and then the data. Asset add is always going to target a notebook. It's just a question of whether or not you are actually adding specifying a notebook or is it implied. The the important bits are here is it asset add is hold rest. So here's a quick little example. I have some placeholder text here which I would like to add as an asset. I have given it a name and when I evaluate I will give back this asset object. Now the key bits to know about asset object. It's mostly an opaque type. The keys in it will change at some point I'm sure over time, but is effectively enough to be able to use as a handle to some arbitrary data. Think of it as like a promise, I have data somewhere, I would like to get it back at some point in time. For a collection, asset names are required to be unique. That doesn't and by collection I mean it's per notebook. If you have multiple notebooks, you can have the same name appear in multiple notebooks. It's just in a collection it has to be unique and any attempt to add a new asset on top of a previously existing one will return failed. Now that we have this asset object, what can we do with it? This is the meat of assets, which is asset value, which given an asset object will. There we go. Given an asset object, I can ask for a particular property of it and it will return its value. This assets are. I don't want to call them strongly typed, but they are not loosely typed. And I'll get to that in a second. The key bits are is that if you're curious to know what properties this particular asset supports, you can just ask for its properties, which you'll notice from asset objects. Some of these are repeated. Some interesting ones are some metadata created and modified. These are timestamps you can ask for what formats are available to be able to grab the content of the asset with. Then the default is basically I don't care Just give me the one that you prefer. As you can see here, this is a really simple example because strings are always where you start. It only supports strings. And then of course the default format for strings is just string. You get that back. Dave has asked what's the largest asset side that you support? Great question. Maximum size right now is unlimited. But experimental notebooks have some limitations itself. And so at the moment, in its current form, I wouldn't try any do and try doing anything of real substantial size. I'm not going to give a hard number on it. There's some things that will happen once this thing evolves over time that will lift that limit up to effectively whatever you feel like. Next. But obviously assets are more useful than just strings. Obviously a lot of the times in a language you want to work with expressions. Here's a quick example. I just have 34 to the 78th power. I'm going to add that as an asset. As you can see here, we know that this is an expression. You can see that it's supported formats as expression. You can get it back and you get back the evaluated result. But that's not actually what was stored because it's hold rest. You can actually ask for it, supplying a second argument for expression which will allow you to get it back held. So you can actually see the original expression that was added. More interestingly, for notebook assets, for a lot of times I think people will use it is I have a bunch of binary data. I want this to be locked into my notebook. For me, just borrowing this waveform from the layout, I'm going to add it. Because string is a type, file paths are strings. You don't really want to add a file path as an asset. You wrap it in file and you can add that. Here's my audio. It knows that it's a waveform. As you can see, my supported formats are bytes and file. In this case, I'm going to just ask for the bytes which I can then pass into import byte array. It will figure out what the content type is, which will be a waveform. And I'm going to throw it directly into audio plot and then I get back my nice fancy plot of the data. File is also one of those. I'm going to do that with an SQLite database. As you see, the front end has no implicit knowledge of what a SQLite database is. So it'll just report that it's an octet stream. But once again, bytes and file are the two choices that you get. Database reference. As far as I recall Only works with files. When you ask for the file for an asset, it will spin it off into a temporary file inside your temporary directory and the front end will clean it up when the process terminates. So you don't need to worry about going through and deleting it afterwards. But an important thing to note is that it's read only from the front end perspective. It's read only. It will not take any changes that you have made and automagically percolate it back in. And then there's another specialized accessor called image, which given a file we can tell that it's of type image. On Windows I can import this directly. I get bytes file and image. Image is probably one of the tags that will change. But nonetheless, because the front end supports it natively, I can just ask for the raw boxes and I will get back to an image directly in the front end without having to go off through the kernel to do it. Is there a way to share assets between multiple notebooks? Yes, Once you have an asset object, you can reference it anywhere. It's just that the moment that you close the notebook, you'll no longer be able to access it. Mention that one. Storing an asset in compressed form be helpful to avoid size issues. I mean, it does. It is compressed, but the problem is you still have to write as text. You still have some native overhead anyways. Can I add a font? George? Wait for the end. That's what I would say on that one. File system show these assets are stored. They're not actually stored in the file system, they are stored in the notebook proper. So you don't have to right now you can't manually drag them into a notebook. This is all programmatic, this is all alpha preview stuff. The key thing that I want to point out with asset object is its lifetime. Its design is very much based around being able to add an asset and then use it inside of interfaces without having to do a lot of gymnastics. It's important to note that because I want assets to be usable inside of dynamic module as instance variables. That means they have to live beyond kernel, session or even front end sessions. Asset object is a guarantee. It's basically a handle that you can grab and use. A promise to get the data at any point in the future as long as the notebook is open and the asset exists within the notebook. So here's an example that I put together. I did this last week. The asset was created on last Thursday in the afternoon. And it's just really. It's just a simple string that says hello world. The important bit is that I didn't have to do Asset Find in order to be able to find it and get an updated object. You can store it and then use it as long as it's still around. Looping back to the rest of the API surface Asset Find it basically does what it says, which is find all the assets for a notebook or given the name. Give me back the asset. Looking back at my demo example here, here's the string from earlier. And then if you were to do, you know, always the same cue to the thing that I had previously, that had better be true. And then you can go through and enumerate all of the assets that are available in this particular notebook so you can see all the ones I've added to it. Since Asset Find, it's pretty, it's pretty straightforward. And then next is Asset Remove, last of the pillars that are available in 14. It does exactly what it says. Given an asset object, you can call remove and it will remove it from the notebooks collection. Once you remove it, you cannot go back and then any existing reference that you had now no longer works preferred. If you were to save it right now, no longer, it wouldn't be serialized back to disk. A demo, a quick demo of a simple image preview or simple only in appearance. The code is actually at the end of the presentation. You can see the presentation is in the handouts by the way, so you can play along. Here is this little image previewer I have. What this is doing is it's a dynamic module. When it initializes, it goes through, reads in the list and then it filters all the assets based on MIME type to filter out all of the images. This is the image that I added earlier. Then using Asset Add I can just go through and add. Here's a picture of me. Well, not me, but it's me at Save St. Andrews a handful of years ago. And so you can add a bunch of images using Asset Add. And then the labels on the left are a property of file based assets. So you can actually get the original name that was added with. And then the preview on the right is using the Image tab. So that way it just gives you back the raw boxes. If you were to save this in its. If I were to save this in its current state and reopen it, all of those pictures would be slurped into the notebook and then you can send it off to others and they can just open and use and preview all of these exactly like I am right now. Code's all at the bottom you can check it out after after the talk future because like I said this is very much a preview things that I would like to see done to it right now. Working with a temporary file to insert other types of data beyond just a generic expression of byte array which you can do but to get so it acts like files is something that I wish to do that is optional but it would be nice asset update for in place modifications. I mean you could do it now. It's a bit ham handed. You take an asset, remove it and then re add it again. It'd be nice to be able to do it without updating the UIDs along the way. The biggest thing in order for notebook assets to truly fly, it needs a different notebook format. So we are planning on building a zip based one similar to other compound document types that are out there like all the Office documents and yeah the open type, the open document. So all the stuff from LibreOffice and others are based on this this pattern and we will do the same which basically gives us way more efficient storage for binary data and will allow larger usage, larger assets to be stored within the document. Effectively you can do it now with some you probably end up reaching problematic points then of course in order to ship this for real, in my opinion, we need an interface for managing these because everyone doesn't want to drop into the language. In order to be able to manage these things, you want to be able to view them really easily, manipulate them, add, remove, rename, etc. Some potential future things which I would like to hear some input on from our users is there's the idea of having additional collections beyond Notebooks, particularly to be able to reference assets within packlets. That would probably be a read only kind of thing, but I'm not opposed to maybe making it so it's updatable. It depends on how that one flies. There's stakeholders that will have a say in that. And then there's been internal talk of doing a collection for the session so you can have just instead of having it stashed away in the notebook, you can actually just have it available in the session. I don't know if that will actually hold out or not, but it's certainly interesting if you think you might have a use case for that. That would be good to know. And then some other stuff that I've been kicking around as I've been working with. This is adding more formats to the front end's native support structure, much like Image, which gives you back raster boxes to be able to embed fonts into it as an asset and then being able to use that in the document and have it port around shaders. There's a talk YouTube is going to be giving. I forget the day, I think it's Friday, I think. Or no, tomorrow might be tomorrow where he's going to talk about shaders. Shaders can also be external and a lot of the times those are reused a number of times in a scene, sometimes even across multiple graphics. Having that for reuse would be good then. Another idea that I've had kicking around is style sheets. I've never tried it, but I think in theory it's possible that you can do nested private style sheets within a notebook. I think it's possible. I don't think anyone's ever actually legitimately tried it because I imagine the maintenance for that and keeping that working might be a bit problematic. But certainly to be able to have a complicated style sheet structure which has style sheets that inherit from other style sheets and you want to just basically lock them into the document and send them around because they're custom, that would definitely be a nice thing to have. That's some thoughts. If you have more that would be good to know. Certainly feel free to send us feedback on that. Like I said, this is all in 14. It should be in the conference pre release build, so you should be able to get your hands on it and start playing with it today. I think I don't know exactly what the, what the, the procedure is for getting hands on it, but it's certainly there. The conference beta will not be as good as the release version of 14, I can guarantee you that. I missed the deadline on that. Sorry. But it certainly is there and is usable in that build. And then certainly if you have questions and or comments as you start playing around with this, we have our office hour on Friday, which is at 11 Central, which you can certainly I will be at. John will be at. You can certainly come by and let us know what your thoughts are on it and the problems that you've been having with it. Or maybe just to say, hey, this is really cool, thanks for doing this. It would be great if this was available sooner rather than later. So yeah, so that's the contents of this. So I'm going to catch up on some comments they're sending. Do pattern names work? No, you cannot do Asset. Find of Star is literally just the name. Something that's interesting, certainly. When does the serpent happen? When does the serpent. As soon as you call add the moment that you call asset, add it Goes off to the file system, it pulls in all the bytes and then adds the asset right then and there. So it's already in memory before you even get the asset object back. What having. Is it importing as plain text? Will it break it? Well, no, because we would make import actually understand how this format works and so it would give you the plain text back. The problem is that if you have assets that are referencing other things that are placeholders that are dynamic, unless you're resolving all the dynamics, you might not actually know what all the content is. But yeah, no, I mean we'll. I mean there's a lot to change. Notebook formats is a lot of work, you know, because it's not just teaching the front end on how to do it, but then there's import that would also want to know how to do it. Then there's things like on Mac Spotlight, you know, the indexer. The indexer on Windows, you know, all of those things would need to be updated as well. You have to teach the cloud about it. I mean there's a lot of moving parts to make it happen in order to fully flesh it out. But in order for it to be a usable format, it has to be fleshed out. Let's see, is there anything, Is there any special treatment in packlets? I don't know what you mean by in packlets at the moment. No, because packlets aren't packets, aren't a collection, they're not targeting. But certainly packlets pose an interesting dilemma because of the fact that packlets are versioned and it happens outside the cadence. So dealing with asset objects, references inside of a version thing can be a bit interesting. I'm not sure how that will play out in the end. That's why I think there's stakeholders that will have very strong opinions on how this does. Let's see, is there any bikes? Ah, so there's a question about version control and version control for any zip based. No, you know, any zip based file format. I mean git doesn't have native support for any of them. Word documents in git. You don't see diffs out of diffing though is something that we're actively working on. And Lou d' Andrea has a talk, I think that's Friday, where he's going to talk about diffing notebooks and that's diffing notebooks today in the plain text form. But it, you know, diff just would get extended to be able to support this new format as well. Let's see, textures for materials. Yeah, textures definitely are one of those data deduplication, data reuse things where textures will be tossed at multiple surfaces. And if you look at it in situ, a lot of the times the texture is inlined. So if you ended up using the same textures over and over and over again, there's a certain amount of reuse that can come from that. Let's see. I think I've run the gamut of all the questions. Certainly, if you have any other questions, you can come up, come find us on Friday. But otherwise, thank you very much.