Continuous Integration & Deployment of Paclets

Hi, I'm Rick Hennigan, I'm a developer here at Wolfram. Today I'm giving a talk on continuous integration and continuous deployment of Wolfram language packlets involving mainly GitHub and how you can automatically test, check and deploy to new Wolfram packlore repository, all from within GitHub. I'm Rick Hannigan, developer at. Well, from. I work with Bob. I've mostly been working on Packlet repository stuff lately, particularly the client side implementation of the Definition notebook that he was showing you. A lot of hint system that's built into that and this is kind of an extension of that. So continuous integration and continuous deployment or continuous delivery, depending on which D you're going for here, is basically just a system of automation set up on top of source control. In this case, I'm going to be focusing on GitHub because that's kind of the platform that I've targeted first for support, mainly just because it's popular and it was all very well documented. But it doesn't have to be GitHub. There's not a lot of time, so I can't show you all the stuff I'd like to. So this is just kind of a very small fraction of what this stuff can do. So I'm going to assume you've all been here and you paid attention to the last two talks and you know, packlets are, you know about the definition notebook, you know the hints that pop up. It helps if you're familiar with the testing framework, like verification test, not necessary. But I'm also going to be using a lot of source control terms like commit, pushing branches, things like that. So I'm going to be going pretty quick. I encourage you, if you have questions or you're confused, to definitely either find me after. And you should either way definitely come to the workshop tomorrow and I will be there and I can answer my questions. So here we go. So, yeah, so for CI CD workflows, some people call them pipelines, GitHub calls them workflows, so that's what I'm calling them here. For packlets, some of the types of things you might want to do are when you push changes to your source files to GitHub, you might want to run a check similar to what the Definition notebook does when it checks for potential errors. If somebody opens a pull request for your repository, you might want to know do all the tests pass before you even look at it. Otherwise you would have to check out that branch and then run all those tests manually and it would be Kind of tedious. So a few other scenarios here. Another one of the things you might want to do is release to the packlet repository directly from GitHub. So you've seen the definition notebook. This is an example of one. I have a sample packlet here, probably zoom in. Oh yeah, I forgot these hints don't zoom. But you can see there's two hints. So the description shouldn't have punctuation. Usually we kind of want to make sure everything's consistent. And this also hooks up to Code Inspector. So if you were here for Brenton's talks about VS code and how that all integrates, all those types of lints are also covered. So for your package files. So it might not always be practical to always have to open the notebook and, you know, run the check. If you're just working with package files, let's say using some other IDE like VS code, for example, it might not be that convenient to keep jumping back to the notebook and running the check. So the motivation for this is automating that process without needing to jump to the notebook every time and being able to see those results in a way that makes sense and not tying up your local machine. So here's a quick little example of what that output looks like. So this is something that ran in GitHub on GitHub machines. It's a job called check. And you can see here these annotations, the description should you not end with punctuation and unexpected predicate after blank. So if we look back here, those were the same hints we got here. So this, this packlet, the sample pack that I have right here is this very same repository here. So that, that notebook is right in the root of that. That directory. So how do we do that? So it's, it is actually really easy. So GitHub fully documents how to do it. You create a YAML file, everyone loves YAML and you just kind of describe what it should do. So here's an example of one, super easy. So you give it a name, you tell it what kinds of things should trigger the workflow to run. Here's a daily schedule. Or it will also trigger if I push to the main branch and this lets me run it manually, then there's environment variables and then you really just the rest of the workflow, which is totally easy. Okay, so it's not easy to do that, depending on who you ask. I wrote some initial versions of them and it wasn't fun and I wanted to spare other people the pain of that. So this packlet that I've created Packlet cicd. It's available on the packet repository. You can get it now. And this contains tools for generating these types of workflows so you don't have to write them manually. So I'm going to do this the easy way now. So this is the packlet corresponding to the definition notebook we just looked at. And this repo on GitHub so that this resource definition nb is what we were just looking at. And now this, that check workflow, a symbolic representation of it. And that corresponds to hitting that check button in the definition notebook. And this workflow export takes that symbolic representation and puts it in the right place in your packlet that GitHub would expect. So we can look at it here, let me zoom this a bit. So it, it has all these similar things and you know, really write the rest of the workflow. It is actually easy. In this case I didn't have to write it. So now just, just pushing these changes with that file now in the repository to GitHub will cause that workflow to run automatically. There is one thing you need to do to set up just one time per repository though. So you need to set up a license entitlement for the Wolfram engine just so it's activated and will run in GitHub. If you don't do that, your workflow will run, but everything will kind of immediately fail because, you know, Wolfram script will say it's not activated. So there's a tutorial page in the documentation for this pack with CI CD packlet that explains how to do that. And it's really simple. So. And I actually mean that this time. So there's Create License entitlement, that's a system symbol documented. You just evaluate something like this. You get out this license entitlement object, you click to copy this value here and then you go back to your repository. This is in settings. You go down secrets actions and then you do new repository secret. I've already done that here. You give it the name Wolfram Script entitlement and id and you just paste in that value. If you're planning on also submitting to the packlet repository, you also need one called Resource Publisher token and directions for that are also on this page. And it's the same idea. You evaluate one thing, copy the result, paste it in there. So now I have that file that I exported to my repository. I'm going to commit that and then push it to GitHub. And now if I visit finish first. Now if I visit this Page you can see this added a check workflow. That's the commit that I just made. It's starting up or it's queued rather. Ok, now it's running. They can take a while to get started. A lot of times you were just waiting around for GitHub machines to come online. So I'm not going to sit here and stare at it with you guys to watch it. So here's one I did earlier so you can see the output. In fact, this is the one I already showed. But yeah, so you get these hints here. Depending on the severity of these hints like suggestion and warning does not cause this job to fail. If you have an error here, this will come up with the red X instead of a green check. So that'll be important later. So another workflow, that one is for checking. Here's something that doesn't have a corresponding button in the definition notebook is running your test files. So it's just called test. Pretty simple. Going to export that to my packlet and I'm going to commit, call it adding added a testing workflow and push it just like before, verify that it's running. You'll also see that also started another check job. That's just because I left that workflow file in there. And that's fine. These will run concurrently and you can see my test workflow is running. And again, we're not going to wait for it. Here's an example now this one has a red X and it failed. Here's a nice report we get out of it. Shows me that there are three test files. Sixteen tests that were run. The proportion that failed tells me which files had failures. And if I open these details here, it'll show me exactly which tests fail failed and why. So this, this was undefined. It expected two, but it stayed undefined. And another neat thing here is these are all click to copy, which is nice for, you know, going back and testing things. If I want to see the exact test that failed, click that link to the source code for that particular commit the one that it was run on and highlights the actual test that caused that failure. Okay, another one. This one I actually didn't test while practicing my talk because I wanted to make sure it was a fresh submission to the packet processors. Hopefully I did everything right. Oh wait, no, I forgot. One step this would have failed. I want to fix those two problems that I artificially introduced. Okay, so publish packlet, export it. Now, like I said, I haven't actually run this one yet. You know what this one required I didn't, I didn't put a push trigger on this one. This one asks you to run it manually, which for these you'll. You'll have a button over here to run the workflow. So I didn't set this one up so it would automatically run every time you push because you'd also have to be incrementing version numbers. So I made it manual. So I'm going to start it. And there it goes. Unfortunately, we're not going to have time to see whether or not that succeeded, but it's called Sample Packlet under our Hennigan on GitHub, you can check and make sure that it worked later and then just see if it actually showed up on the packlet repository later. If it doesn't, something went wrong, which it might. I'll fix it, then commit it and use that same workflow again to make sure it gets on there so you guys can check it out. So how do we kind of customize these? So I have these named workflows as sort of presets. How do I make one that does something that I want? So I've separated these into three real building blocks, kind of going from top down. This workflow, which is kind of the outer container, it specifies, you know, the triggers that cause it to run. And then it has a list of workflow jobs. These say what type of machine it should run on and things like that. And there's a bunch of other metadata that's included as well. And it has a list of workflow steps. Each step is like one command that's run. So this workflow step here is. And look at it. It's called Build Packlet and it's actually like a pre made step that, that's available. It's on, you know, the GitHub Marketplace. And if you're familiar with GitHub Action Syntax, you might, you might know how to already use those. But you can also customize these as well by giving it shell commands script files. Okay. So we can also pull these apart as well. So here's a named workflow. I can extract the jobs from it. You can see that there's four of them here. By default they'll run concurrently unless there are dependencies. So this build package job depends on these three compile jobs. And yes, you can compile your code on multiple different platforms and then combine those results into your build packlet. That's something that is actually very difficult to do on your regular machine. So if you don't have access to a Mac and you're on Windows or whatever. You're kind of out of luck with figuring out how to build binaries. This solves that problem for you. So I'm going to take that last build pack of the job and just look at the steps that it has. So most jobs are going to have this checkout step. It's kind of necessary that you know gets your code from GitHub, otherwise you're just starting with an empty directory. It downloads the the compiled libraries from those three previous jobs. It builds the packlet. This is equivalent to that build on the Def notebook and then it uploads. The. Zoom is getting in my way. So there's syntax for how to make these manually. The fully specified way to do a workflow step is with an association like this. And so these are the two main ways you can. You can specify a pre made action that's on GitHub already, or you can give it a command. But there are also these pre made named steps like I've shown before. You can also specify shell commands using this workflow evaluate of a string. You can also have arbitrary Wolfram language code, but these end up getting just converted to one of these. Run commands like that and the same thing with a file, the script wrapper. So we're going from up here. The next level is Workflow Job, which combines steps. Also I should point out, you don't have to keep wrapping all these in Workflow step because it's unambiguous here. So anything that could be an argument to Workflow step, you can just use that way and build them here. So this is a job. I give it a name. This depends on another job called check, which we don't see here because this is kind of its own thing. And then it executes these steps once that check job succeeds. So just like Workflow Step, these also have a lot of the same named presets. But you can see what this does as a job. It's doing that named build Workflow step, but it's also surrounding it with steps that make that actually useful and usable. Because build by itself would totally fail if it didn't check out your code first. It would have nothing to build. And then you obviously want to upload the results, otherwise it just disappears into the void. So you can also specify with a list of steps, you can specify a file as a script and again that just wraps it in useful steps. You can also specify the operating system at the workflow job level, which just says this script should be run on a Windows machine. So here's a big complicated workflow that I put together that has some triggers here. Schedule it runs on pull requests as a name and these jobs here for building files. This is a real workflow that I've been using on this code equivalence utilities packlet and that's what the output looks like as a check and a test job. If you want to look at the workflow file that it generated, there it is. So that's an example of. So now here's an example. This was originally going to be the one that I submitted to the repository, but I ran into a problem. There was a bug and this actually turned out to be more interesting and useful because something went wrong that I didn't expect. So this one compiles code like I was mentioning before, it has some libraries. I'm just going to skip over these quickly because the thing I wanted to show, I put other examples earlier for them. So this is basically what this big workflow constructing does. There isn't a whole lot of time to go into all the details of how I made that, but it has these three compile jobs on three different platforms, merges them into a build package of the job, and then once that done, it triggers these test jobs to make sure that my compiled files are actually good. And then eventually it'll submit, assuming that all those tests pass and it's going to run a check job to make sure the definition notebook check works. So did I already export that? Yes, I did. Yeah. So that that's running and I think it's. Here's the result from before because that one will take a very long time. So here's what went wrong. I thought, I thought things were working because I checked a smaller version of it that just built the packlet and generated the files. You know, I looked in my library resources folder. I had a compiled library for Linux, Windows and Mac. I'm like, great, it's working. I tested it on Windows on my machine. It worked great. Like, okay, I'm just going to finish this example, put tests and submit and it'll just work. And here this is the interesting thing that happened. Something I didn't expect. It failed on a Linux machine. And this laptop I have is a Windows machine. I didn't, I didn't have a way to test on Linux. So I, this surprised me. If I didn't run this workflow, if I didn't set this up, I wouldn't have known that this was failing on Linux machines and I would have delivered something to the repository that was broken for a lot of users. So this is exactly the reason why you want to set up these types of things for your packlet. I did figure out what the bug was and I'm going to fix that and then redo this afterwards. But I thought this a better example of why you want these things. So how much does all this cost? Like, you know, GitHub must be charging a fortune to run on all these machines. Right. And we can find that out here. So if we look at the billable time column and add all these up, we get to 0 seconds of billable time. So yep, it's free. If it's a public repository, it's free. And they give you absurd amounts of computation time that you probably almost never need. So we're talking like hours and hours per job. Yeah. So it's, it's pretty cool. You should take advantage of. Doesn't take cloud credits or any to run the wolf to create the license entitlement that does take service credits, but it's like pennies. Is it a one time thing or for each run? I. I don't remember what it is exactly off top my head, but it's such a tiny amount. Yeah, I, yeah, I think it's like one or two credits per hour or something like that. And each of these jobs is just like a few minutes. Yeah, I've been doing hundreds and hundreds and hundreds of these workflows developing and testing this stuff and I think I only put like $5 of service credits in at the start or something like that. So. Yeah. So I just want to point out these other talks that are relevant. That's the end.