Paclet Repository: One Year In

All right, I'll go ahead and get started. I'm Bob St. Heinrich. I'm a developer here at Wolfram. And this talk, I'm talking about the packlet repository. If you attended the conference last year, there were several talks about packlets in the packlet repository. It was the official announcement of the packlet repository was at this conference last year. This is the first birthday for it. So we get this nice little cake. This serves as a refresher and update on the packlet repository and packlet development in general. I want to note off the bat that we have an office hour today at 1 o' clock where I will be there and several of my colleagues and we will be able to, you know, take any questions. So bring your, bring your questions, bring your code. We can, we can, you know, take time individually to, to look at things with you there. All right, so, so quick refresher and I'll try not to waste too much of my time on this. What are packlets? For those of you who are coming to this naive, packlets are the main tool for bundling up Wolfram language content and distributing it. The most common thing that lives in a packlet is Wolfram language code, but it can be a lot more. Documentation is standard. Any good packlet will have documentation. But you can also distribute style sheets, you can distribute palettes, you can distribute entire apps that run in Wolfram language via packlets. It's important to note that they're named. The name of a packlet is its identifier and their version. You can update packlets and it's important to keep track of what versions of a packlet you are working with sometimes. And then more about the Paklet repository. As I said just a minute ago, it was released last year at this conference, sort of. It's the official packlet distribution mechanism for Wolfram. You know, we've had. Packlets have been around for a long time, for decades, but really the only official distribution mechanism was not open to people outside of the company to contribute to. So now, so starting last year, we have a packlet repository or packlit distribution system that's open and it's the first time we've had that, which is great because it's sort of something that people expect for modern software systems. And like other Wolfram repositories, it uses a Publisher ID system and a definition notebook if you want to contribute. So if you've submitted a function to the function repository, then you're probably familiar with the Publisher ID system in the definition notebook. I should Also note here, the packlet repository, let me add a new button because this is important. Not reviewed like function repository. In the function repository when you submit a function, it goes to a review team that checks it out. Make sure it's safe, make sure that the name is right for the function, that the usage is reasonable, that the documentation looks good. They don't really make sure that the code is efficient or anything like that before they would publish it. There's usually back and forth with comments, but the packlet repository is not like that at all. You push a button and a few minutes later your packlet is live. And it's only, you know, only computers between you and being published. No humans. All right, so first birthday. What's in there? Sorry, I saw that another one came in yesterday and several more versions. So this is out of date. 132 packlets have been published and many, you know, the average is something like, you know, six, five versions for each packlet. But you know, some of them have had like 30 updates and some of them are still the original one. So the mean doesn't mean too much there. Some of the leading publishers with many packlets are Anton who is attending this conference. So if you want an honest opinion of working with the packlet repository from independent source, go by the around town. We've got Peter Burberry has submitted a lot of packlets. He's one of our avid users. Kiro Bellov has put some great content in there and then the chemistry team, which is mostly Wolfram employees, but I think there might be some external contributions to that as well. They've submitted several packlets and you know, the pack repository hasn't really fallen into any domain niche. Right. So recently, recently you know, there's been a lot of things for AI hooking up to LLMs has been really popular. There have been multiple people who've sort of put their effort, their ideas of the right way to integrate LLMs have come through the packlet repository, which is great. There's a lot of pure math packlets, there's a few art things, just beautiful visualizations, packages of code for that. Then a lot of development tools. If you want to get some neat interfaces or programming utilities to help you become better Wolfram language developer. Look at the packet repository. That is a great source for that. Also if you explore on the packlet you see that the state of these packages varies a lot. Some of them are really polished. These are really production ready things with polished documentation. There aren't too many bugs in the code and this really moved into the phase of updates and maintenance and some of them are really raw, very prototypey. You know, maybe there's one well established function and then sort of just a loose idea of a framework around it and not very good documentation. Right. And all of those are great. Like, like we hope that the prototype things get built out more, but we love people to just get it out there in an early stage to share ideas. It's really useful. And one thing that I didn't necessarily expect when we launched a year ago is that developers within Wolfram Research are increasingly putting early versions of their work on the packlet repository, making them publicly available. Like the chemistry team that I mentioned earlier is doing this a lot. And this is really useful for everybody because external people can see their work before, you know, maybe a year before they would have otherwise. And the developers can get feedback on their ideas and try things out with a wider audience than they would have. Normally. It works as a collaborative environment. And then sometimes the things that you see early in the packlet repository will show up in the released Wolfram language in a Mathematica build, you know, a month, a few months later, which is kind of a fun way to see it. So I'm going to take a few minutes to just look at a couple, highlight a couple of packlets to show sort of the diversity and also show sort of different ways to use code in packlets. So this data reshapers one is pretty nice here. Hold on, let me leave my presentation. Here's the, the website. Let me see, let me increase magnification there a little bit. Here's the website for this packlet. You can see that there's lots of documented symbols. There's this main example section that gives a high level overview and all of the stuff here you can click and copy and just start using right away. I've already copied some things out. Here's a dataset from the data repository. It's not part of this packlet. This is just in the standard data repository. You see, it's just two columns about different cars that are made. Now I'm going to install the data reshapers packlet. I'm going to load the code with needs. Looks like there's some bug maybe in that packlet, but I think we can ignore that one. Now cross tabulate this function that exists if I say context of that symbol. This is in Anton's data reshapers PackletContext. But since I called needs, that context is on my context path. So I can use this symbol right away without giving a long context. So this is sort of the traditional normal way to use a packlet is to call needs and then the symbols are available. And this cross tabulate does this nice thing where it turns this two column table into this two dimensional data set where the values are tallies or counts of the original data. I don't know a one line way to do that in Wolfram language. I've used this a few times. It's pretty nice. Another nice packlet is this Open Chemical Systems. I'm not a chemist, so I don't really. I can't say that I really use this regularly, but I definitely looked into it when it came out and thought it was pretty interesting. So it's modeling chemical processes as graphs as you sort of discrete systems in what's called a petri net. And it has code for doing simulations and visualizations. So for example, here's is it runs a simulation of this reaction and it creates a visualization of some chemical concentrations throughout the process. And if you'll notice here, I didn't call needs at all. So the way this packlet is being used is with these little lozenges or blobs, whatever you want to call them, that are packlet symbol If I do, sorry, before I evaluate it, let me look at the input form of it. Well, that's evaluating place. Let me just do command shift I on this thing. Let me paste it here. Command shift I and this is packlet symbol of the packlet with the symbol name. And when I evaluate becomes a symbol from the packlet. But it does more than that. It ensures that the packlet is downloaded and extracted and that primary context of the packlet is loaded so that the symbol is available. And this, this is really useful. It does not permanently install the packlet, so it doesn't use packlet install. So it's really great for like trying out a packet from the packlet repository. If I were to quit my session, it would be like I'd never use this packlet next time. So this is a great trial tool for packlets from the packlet repository Automates all of the initial loading. If the packlet is already installed, it'll just use what's already installed. Another packlet that I played around with and enjoyed was Peter Burberry's Recreational Math. So this is a collection of just sort of fun ways to waste an hour exploring math. Sometimes you need to remember that math can be fun, right? So for this one I'm calling needs. But instead of just having the Primary context of the packlet. I'm saying none here. So this will load. I've already installed it. I left out the packlet install line because I'd done that previously. I should have had packlet install here as well. This will load the context, but it will not put it on the context path. I can't use if I take this diagonal walk plot and I run doesn't know what that is because this context is not on my context path. Now I need to use the full context. This can be important to use. If, say, there was another packlet in the packlet repository with a diagonal walk plot, there'd be a name collision and I want to avoid that. So I don't want to put this on my context path necessarily. This is very common to do. If you're using a packlet that calls recreational math and internal to your packlet, you want to load this. It's really smart to use this and then have the full context because that way the end user of your packlet won't end up with Peter's context on the context path. But as you can see, like exploring these different rant, these different walks along a path is kind of interesting and fun and a good way to spend an hour maybe. And the last path that I wanted to highlight is this max rd. It's another chemistry one. For this one I'm going to call needs and instead of none, I'm going to assign it a context alias. Oh shoot, what am I doing? What happened here? Do I need to do a packlet install? So when you do this instead of oh shoot, Do I have a typo? Okay, there it is. I just hadn't installed the packlet right. So instead of. So instead of not putting this on, not putting anything on the context path or like needs with none, or putting this on the context path like needs with just the name where this would be on the context path, this renames it with an alias. Now all the symbols in there are available in this crystal context. You can control the context yourself when you load it this way, which is really useful. This packlet has all this crystallography stuff in it. It's built on top of the chemistry stuff that's in. There's an interesting one that's in Wolfram language, but it has these dedicated crystallography functions that are really useful, including these interfaces and visualizations. So this is another cool one to check out. So this is just like a quick run through some of the. How am I doing on time? Oh, ok. Use more time. There than I thought. So sort of a quick run through of the sort of the breadth of domains that are in the packlet repository. So I'm going to try to run real quickly through contributing to the packlet repository. And if you're starting fresh, you can use Create Notebook of Packlet Resource or there's a file new repository item packlet. It'll launch this dialogue. You can choose a directory where you already have a packlet if you have one, or you can make a new directory and it'll create the packlet layout for you. If you're interested in seeing that, look at my talk from last year. Because from this tech conference last year I go through that whole thing. So this year I'm going to do something a little different. Hold on, let me exit full screen here. And I'm going to update a packlet that I've already published. So you can see this is the definition notebook that I created previously. Let me magnify this. And this didn't magnify but you can see in February I published version 1.2.3 of my games packlet. I want to add some code to this, some new functions. I could click on one of these and just modify in the package editor, which has all these great new features. But I find when you're working with multiple files, like for packet development, using an IDE is really helpful. I have VS code here. Let me see. Does this magnify the way I expect? Yeah. Here we go. I'm going to. When I want to add a new file, I copy an existing one. I rename it. I'm going to call it wordle. Hopefully there's some people who haven't quit playing wordle yet. I'm sure there's lots of people who played it a year ago. Now I'm just piling everything into one context in this packlet, which isn't great practice for big development projects. But this is a small, maybe medium sized one and I'm. I wrote this code earlier because I figured I might be short of time and I'm just going to paste it here. So here's four new symbols that I want to add to my packlet. So I've $wordle words and I want wordle free and wordle. So here I'm putting them up here so that they'll be declared in BOB Games instead of BOB Games. Private BOB Games is the primary context of my packlet. And then what's the last one? Wordle match. Here's four new symbols in my packlet that I want people to know about. Then I want to declare these in my packlet info file. So this means that if the packlet is installed, the Wolfram language will immediately know about these symbols. They'll be black at startup without even having to load the packlet. And there's just other conveniences that they'll be in your autocompletes. There's lots of nice things that will happen because I've declared them in my packlet info file. There. I've declared my packlet info file. This packlet. This is like my loader file. This games wl is when somebody calls needs on the context. This is the file that gets loaded and then it loads all the other files. I want to add wordle to there. I think I've added all of this code to my packlet. Now I'm going to come back here and I'm going to run check. It tells me my definition notebook is out of date. It's an old one. Do I want to update it? I'm going to say yes. We're regularly updating the definition notebook framework that reset it to 100% magnification too. We should probably fix that. All right, let me finish that before I. Okay, so it found three issues. Let me magnify this. It's telling me that there's three issues. One is that the packlet version is lower than the published resource. I click on this and it has helpers. I'm going to say, specify new version. I get this dialog that helps me pick out versions. I'm going to say this is a minor update. And that opens my packlet info file and asks me if I and it makes the change automatically. And it asked me if I want to save it. I say yes. All right, and then I'll close that and let me see, what are the other issues? This contains very large images. All right, I'll let it sort of. There was a helper just to reduce the resolution to reduced file size. And then it's telling me that this is just big. And I think I'm just going to let that one be a problem. I'm going to say don't show again for this cell because it's just inherently a big output from that function. Now I want to write some documentation. I open my toolbar. I say, documentation tools, open palette. That this says games. I'm sure this is really small. Okay, it looks like I can magnify. Here we go. So it already knows about my packlet. I'm going to say I want a new function page I'm going to document $wordle words and now I have this documentation thing. This is a dollar symbol so I want to get rid of those brackets and I want to say lists the five letter words. Available. It's not the best verb for wordle appropriate and now it would. And then within this notebook this needs statement is automatically evaluated. So I can just say $wordle words and if I evaluate that it's going to run and it's going to give me this. That's kind of a huge output for a documentation and then I'll say list the eligible words. All right. Eligible. All right. So this is sort of a minimal documentation page. Could be much better but I'm going to save it because I'm running out of time and I'm going to come here, I'm going to hit refresh on this and you should see here's my new documentation page and I'm going to build and this will take a few minutes. I'm going to where my presentation go come back here for a minute and talk briefly about what's coming up. A big initiative coming soon ish is dependencies. If you have a packlet and it depends on another packlet which then depends on another packlet. It's a really complicated problem. Package managers and software in general struggle with this and we're going to try to create a a decent solution for our system coming soon and then better administration for managing your published packlets for publishers, including a pre release version sort of specification so you can publish something that people can use if they want to, but it won't get downloaded by default. All right, so that's sort of the end of the talk. The build is completed. It didn't give me any warnings so I'm going to go ahead and say submit to repository and while that runs I can look at the questions and if it doesn't finish by the end, so be it. All right, so let's see. Seth is asking, could you add a recently added button to the packlet repository? Oh yes, there's. That should exist packlets.com if I go to the all page there's. You can sort by date but if the function repository I think makes this much more accessible, like there's a button down here at the homepage. Maybe that's what Seth is asking for. That's a good idea. We can. Yeah, we should do that. I agree. Yeah. And then email that gets pushed out weekly. We've talked about that. You know this doesn't have quite as much throughput as the function repository. So it might be more like monthly. We're a little hesitant because, you know, we don't review this content, so we don't really want to send out emails blindly about content. But maybe that doesn't really matter. Maybe people, maybe we can communicate. This was done, but we didn't do it. All right. Oh, Seth has a simple one liner to do across town. Okay. Okay. I said I couldn't come up with a one liner, but Seth, Seth, the master of query, can. So. So yeah. Okay. I. Good. Thank you. Let's see. It looks like Todd was hoping to answer some questions and. Oh, and Anton saying the packlit system can be used to publish books. Absolutely it can. Yeah. You should, you should check in on my colleague Bradley who has ideas about that. And maybe at some point having a sort of book branded repository that might just be packlets and as an implementation, Anton says he's not really independent. I know, I know, but pseudo independent and. Oh, it's not a bug. Okay, okay. I'm sorry I defamed you. Maybe, maybe. I don't know. I don't know why. It gave a couple messages when I loaded the packlet. Maybe it's expecting to auto load or something. Okay, let's see. There's a lot of comments. Anton uses IntelliJ instead of VS code. I've used Intellij in the past. It's good. I slightly prefer VS code mostly because I think our plugin works a little bit better in VS code. Right. I think that's the question. Let's. Let's check back in on my publication, see what it's. It says it is being published, so usually that takes a few minutes and then I'll get an email telling me, telling me it's published. We can, we can check on, Let's see, games. See if I have 1.3 here yet. Probably not. Oh, it's already done. Great. So it's. It's published. I probably have an email waiting for my inbox and you can see $wordle words is here. So this is all updated and it's. It's as easy as that. All right, so we're right on time. Go check out another talk. I know there's one about robots and one about graphs. I plan on watching both. I don't know which one I'll do live, thank you very much.