Beyond the Computational Essay

Hi, everyone. Let me introduce myself. My name is Ron Monson and I'm the founder of OnSoc, which is a startup company now that is looking to create some educational and statistical applications. My background is in academia. I worked at Victoria University and Edith Cowan University and decided to see if I could build some applications in the Wolfram language. And this talk describes some of that journey and some of the challenges, but also some of the opportunities. So what I want to start with is just mentioning the terrific computational stories that start out as notebooks. So we're all familiar with computational essay how useful it is, this combination of text, code, graphics and the story that it can tell. But what often happens is that you reach a certain level in exploration and you want to take it further. So maybe you want to create a larger package, maybe you want to create an application that you can sell, maybe you want to promulgate your ideas. And there's often a next level challenge to go from that interaction with a computation, with a notebook into into building a larger system. You get into software engineering issues, software design. And so the interesting question is how hard is it to transit from that initial exploration, experimentation into this building of this system? It's pretty hard, but I think there's a few lessons that can be learned in that process and hopefully I can communicate some of them today. So the overview of this talk is we first want to motivate and investigate the use of the Wolfram language for building sizable computational systems. We'll argue that even the possibility of this being readily available is important for all users. I think it's no secret in the last 10 years that Python has started to become used more and more often in education, in universities. And my view is that the reason for that is that it's got a dual capability. It can be used for research, but it can also be used to build systems. And to have that possibility available for the Wolfram language, I think is important because then it provides some sense of justification for using it in an educational setting, given that it can improve the job prospects of students, not only learning the domain that they're studying in rich ways, but also prepare them for the workforce. So we'll also introduce an upcoming online course covering each step in this process. So this is still a preliminary course, as I said, been sort of capturing each stage in this process, because my hunch is, is that there are many other people in the same boat that would also like to see if they can take their notebooks and produce larger packages and systems. And so part of this online course is just to test, test that thesis to see what sort of interest is out there in taking that step. Now we'll also announce and demonstrate Paklet Assurance, which is a new package expanding Wolfram Runger's units testing and Packlet deployment capabilities. So talking about how important it is that we can transit to building these larger systems, it would be good to have a case study or an example of proof of concept. So some common themes in the talk will be that the Wolfram language is intriguingly placed to become a feasible next generation system building tool. I mean, it's not immediately obvious that a language that's optimal for computational exploration is also good for system building. In fact, conventional wisdom would say otherwise. That you have this stack of technology, it's already out there with these other languages and that you'll be reinventing the wheel. And you know, it's a much harder road, it's often productive to viewing system building through the lens of long term, long form linguistic expression. So there's lots of analogies between language, I think, and the building of these larger systems that sometimes worth bringing out. ChatGPT. No talk would be complete without mentioning ChatGPT and how it impacts in this case the system building process. And I'll just mention this uncanny link between software design and foundational research that I found in, in designing some of these applications and also some stories and anecdotes from the trenches. So a rich sustainable language needs ethics. I think that even if you decide when we use language, not every proficient language user ends up being a novelist. That's true. But the mere possibility of that being being a pathway, I think is very important. It enables a critical mass of users, of speakers, of the language. And so I think it's very important that this is always a possibility for the Wolfram language. And as I mentioned earlier, educators are increasingly choosing programming language for future employment prospects over their inherent pedagogical values, their inherent pedagogical value. All WL users benefit, I think, when there's a pathway from experimentation through to deployment, when Wolfram language expertise can offer a reliable career trajectory. So in the course, this is a snippet from the course, there's a bit of a polemic saying why it is important that I think this skill set of being able to write packages, of being able to write systems is not just the province of the software engineers and developers, its everyday users. And I think, you know, this is a possibility. The language is already at a high level when it comes to exploration. And so the intriguing question is to what extent can that extend over to building these systems? And so I put these two polemic, sort of provocative questions that any sustainable language fit for computation, computational exploration must also be fit for computational system building. And any WL coder worth hiring must possess expertise in Orphram language system building. So for educators, I think it's important that not only they teach the domain area using the Wolfram language, but also how that can then translate into these larger projects that can then inject that computational understanding into the world. If you're a consultant at a company, it's all very well to do fantastic data analysis, but in order for that to be sustainable, it needs to be embedded and invariably a computational system created. So I leave this long winded explanation in the notebook itself, but you can read that in your own time. So what are the 10 steps to building computational systems in the Warframe language? Well, broken it down into these, these three, these 10 steps. And this forms part of the course that ONSOC is going to be running next year. So it needs to be a rationale initially, a core reason of why you want to go from a computational essay to building a larger system. It's not always the case that one will want to do that, but one needs a very simple core reason why that's important. One needs to get the foundational architecture right. That's, that's a big challenge. How do you go about designing the pieces? Next generation packaging so most users will be familiar with the current Begin package end package framework has some limitations however, and I think the time is right for, for a next generation packaging. I mean that's, that's a talk on its own, but that's also covered in the course. The art of function naming is very important. I think one of the characteristics of language that distinguishes it from other languages is how well its functions are named. And it's not easy to do. It's very easy to do moderately well, but it's extremely difficult to do very well. And of course it's done very well. You get that sustainability, the documentary dance. So one of the processes I found in the design that going from the documentation to the implementation, working in parallel back and forth, the new tools, the documentation tools really help in that respect. There's lots to be drawn out there. Unit testing reimagined is probably an underestimated area. So unit testing is typically considered a kind of boring software engineering exercise that one has to do. But I think it's a lot more interesting than that. In fact, we're going to focus on that for In a few minutes. Exception handling is important. I think it's. It's. But it. But it's messy. It can interrupt the flow of your code. And so being able to implement unobtrusive exception handling is important. Systemic parameterization is critical for building these larger systems to have lots of parameters and variables and organizing them I think is a. Is an important task. And then finally the promulgation party. How do you go about putting your findings and your computational discoveries out into the world? So as I said, I'm going to focus on one of those 10 today. The unit testing reimagined and claim that the power of the importance of unit testing is both wildly underestimated. I would argue that it arguably surpasses even mathematical theorems as a railroad to incontrovertible truth. It's also a ChatGPT guardrail and 13.3 the unit testing framework received a significant makeover, but my view it still lacks a certain scalability. So that was the motivation for developing Packlet Insurance which augments and complements the built in framework and adds this scalability. Okay, so for this conference made this as a. As a pre release. So if anyone wants to try this out, they can surf along to this address here and download the packlet and there's instructions for how you go about doing that. So this is the page that you go to. As I said, there's a DRM added to it. That's another interesting experiment and journey. But at the moment, as I said, this is pre release and it's free for all participants. And these are the steps that you need to follow to set this up. It's fairly simple. This can be improved and will over time. But at the moment you download these two packets, Paklet Assurance and the OnSoc License Manager, you evaluate these two Paklet install commands and then you evaluate Packlet Insurance and put in the key that's generated and then away you go. So when you load Impact Assurance for the first time, you get this screen here and so you put in your license key and then you're away. Okay, so let's so yes, this is a process. You packlet install those two downloads and the first time that you load it, as I said, you get that screen, you put in your license key and away you go. And then it can be thereafter loaded just like a normal package. So the basic usage of this here's an example is most of you will be familiar with the test report framework. So here we can generate 200 tests and you get a a test report object back telling you how many of those tests pass the success rate and some various other parameters. And we can. What Packlet Assurance introduces is a test summary of that report. So. At the moment, nothing exciting, you get the same thing back. The big difference is, is that with a test report, all the tests are actually included in this object, whereas at the moment, this contains statistical summaries of it. So a test summary has the same properties as a report, and so we can confirm that that's true, but it can also generate some human readable output for some other properties. So, for example, we can look at the time elapsed, the memory used, the mean time elapsed, and the mean memory used for all those tests. So the mean memory used is probably a good parameter of the size of the test, because the memory used gives you the entire memory for that whole test set. Now, the significant point here is the report summary has a much smaller memory footprint than our test report due to not storing all the original tests and their specifications. And so you can see a difference here already. 400 kilobytes compared to 7 kilobytes, which is not terribly consequential in this trivial toy example, but it can, as a system scales. So one of the really nice design features of the test report framework is that you can combine different reports. So you can generate 200 reports, tests in two reports, and you can combine them. And so each one of those was 100%, and so we can combine them into a new test suite of 200. Similarly, test summaries can be combined. So the syntax is exactly the same. I should point out that all of this is documented in the normal way. So if I go to. Packlet Assurance here, I'm not sure why that didn't appear in the. In the documentation there, but at any rate, I have another page that was generated earlier. So this is the suite of functions that you get with this package. You get test summary, test summary, object test files, and then you've got a few packlet development functions as well that generally streamline the process. So we've got this. So this is the documentation page that you get for Paklet Insurance. So you get these suite of functions test that I was talking to last time, test summary, test summary, object test files, packlet build installs. So just going to demonstrate some of these functions. So if we combine the tests here. Okay, so the big advantage for using this test summary is that you can combine all the tests, but you don't drag across the test inside the object. And so you get this large difference in the memory footprint as a result of that. So the difference here between the memory footprint of summaries and reports progressively grows. And so already it's starting to have a material effect. And if you want to build some downstream unit test applications on top of this, then, you know, if you had on the order of 100k tests, then this becomes significant with this test report, you know, taking up 1 or 2% of the total memory. And so it's prohibitive in the end. So that's where test summaries are a nice adjunct. We also want to make make unit tests a first class citizen. So the arrangement that I have in the structure is that every packlet has a test files directory. And in order that the packaging packlet object can conceive, packlet system can see that you need to specify this in the packlet object. Once you've done that, then though the contents of your test files contains all these objects, all these functions here. So the idea is that every function that you ship should in my view, carry with it the unit test that have applied, broken down by theme, so that users can verify the functionality that you're offering them. So you can see here that packlet build, install test files, test summary, object test summary and test file are all part of the functions of the package and they all have their own unit tests. And so we can see here the contents of the tests from the test files, broken them down into those tests that handle exception handling, documentation, core test and test files. And so you can. The advantage of doing this is that you can actually test to a much finer granularity. So for example, now we want to find a test summary of the exceptional handling. And so that's all good, 100% if we want to then step up a level. In generality, we want to test all the unit tests of the function test file, then we can do this. So that's all good at the moment. And if we want to move up a level further, we can test the entire packlet and we can get a. So clearly here there's not 100. Yes, it is 100% right. So we're getting close to release with this, with this package. What about deploying packlets? Well, there's a couple of handy tools that you can use rather than using this shorthand, these couple of statements, we can just click, we can just evaluate developer on and then that tells us to look in the directory that we're developing and we can load the packlet and. And then deploy it. Deploy is an interesting function. It's A it's an extension of the packlet build. It obfuscates the code and also removes the NB files and just makes it nice, nice and ready for deployment. So I'll leave you to if you're interested to play around with that packload assurance. I think we're going to be running out of time, so I'll just speak to these. Some of this example, some of the importance of the unit testing. So one of the perils of ignoring unit testing is I was going to show you several examples from the documentation where even though it is claimed that the documentation tests all its functionality in the documentation, there's quite a few errors in that that can be corrected. So unit test weren't clearly constructed for those. And so to me this is a. This is just an indication of the importance of the system. Mathematica is one of the pound for pound, probably the most correct rigorously produced system on the planet. And yet without these unit tests for e.g. the general shadow. Error, then it just doesn't get picked up. And so no single human can be expected to detect these, these downstream effects of the subtle changes. But the, the unit tests can do that. So let me just conclude then, look, I might stop now and if there's any. Any comments or questions, I'm happy to answer. Okay, so I'll just finish with as I said, this course, it has these 10 parts of building larger systems. And this is on the website onsoc.com and so there's some further detail about the rationale from the foundational architecture, various elements of that the art of function naming. You know, it's quite remarkable in some ways. One of the most basic parts of writing WL code is deciding what to call a function and yet seems to be floating in folklore. I think it'd be good to nail down what actually the process of naming. And so we go into that in the course Functional versatility. One of the great pedagogical tricks hit upon by natural language is using making the same word applicable in different circumstances. And so we want to really exploit that when you're designing the functions. As I mentioned earlier, the documentary dance is important. This idea of not knowing exactly what a function does when you start out, but it tends to evolve in concert with the documentation that you're building around it. This is a very powerful dialectic unit test reimagine. We showed some of the functionality of this packlet assurance package diamantine exception handling. As I said, it's a quintessential coding. Sure. No one wants to really do it, despite its indispensability as part of any software release. Worse, you carefully craft this code and then by the time you add all this exception handling, the gist of it's gone. And so what we want to do is retain that gist, but also have this nice bulletproof packages for users. As I mentioned, systemic parameter rejigged is what should be loaded at startup, should a system's parameters be continuously set, how should experiments configurations be stored, how should we even assign values to variables in a computational language? So we cover that. And then the final part is the promulgation party, which is you've got all these packages, the system, how do you get it out into the world? And so this talks about encryption, relative relevant repositories, cloud APIs and DRMs, etc. And so as I said, this is still a work in progress with this course, but I think it's a very important part of the language to be able to go from that computational essay notebook into building packages and in larger computational systems. All right, so the question says, if a problem requires many weeks of computer time to solve in a high end workstation, is it still suitable to be presented as a computational essay in a notebook? What would you think of just including the results of the simulation in the notebook for the user to manipulate, not the simulation code itself? Well, I mean, I guess the point of the notebook is that the code there is available. So it's probably, you know, if you can do it, I would think it's a good thing to do. You're suggesting that if, if that's too unwieldy, if the size of the, the code, then then obviously you have to do the results. But I, I think that's the, I guess the question then is of what types of problems. You can always strip down this, this problem that took many weeks of computer time and so you could provide a stripped down version of that with its code and then you could perhaps say, and if you want to see this in its full expression, then you know, you need to put it onto HPC, in a workstation, etc. But I think it's important to maintain this idea that the code is in the notebook, stays in the notebook, you know, it's part of reproducibility, people can still. It does raise an interesting point though, because there is this tension, and perhaps this is what you were getting at, is that there's this tension between packages, packages, things up. It's sort of like a black box. You don't have to think about what's underneath. Whereas you know, as an academic or as a researcher, you often want to know what are the innards, what are the details. And so there is a tension there. So I guess that was part of the whole idea about the unit testing, is that you provide that transparency. Yes. You package it up into this black box, which you kind of need to do, so you can go away and use it without thinking about all the underlying details. But then if something goes wrong or you want to investigate further, you want to be able to drill down. And that's part of the nice thing, I think, about having this, this framework where you can dissect all the unit tests into areas of functionality that allows you to doing that drilling down. Any other questions? No. I'd like to know if anyone is also considering that path. Have they ever, I mean I'm interested in the communities levels of interest in going from that computational notebook to building these last larger systems or packages. Has anyone tried that and then found it was just too difficult or it's not something that they even set out to do? I think that's an interesting question. Well, if that's. If there's no other questions. All right, so this example of the perils of ignoring the unit test, you can see that that's up there from the documentation. It states that the standards of correctness of the Wolfson system are certainly much higher than for typical mathematical proofs. But just as long proofs will inevitably contain errors that go undetected for many years. So also complex software systems such as the Wolfram system will contain errors that go undetected even after millions of people have used it. Nevertheless, partly after all the testing that has been done on and the probability that you'll actually discover an error is extremely low. Perhaps I think for consistent users it's a little higher than that. But any rate, here are the tests that are carried out for every Wolfram language function inputs a device that exercise both common value and extreme cases. Yes, coverage is important when we construct these unit tests. Units inputs are devised to exercise each feature of the internal code that's similar to the coverage. All examples in the Wolfram language documentation systems are used as well as those for many text many books about the Wolfram system and I think probably by and large that's true. But I want to show you an interesting example that shows that that's not always the case. Tests are constructed also from mathematical benchmarks, numerical tables, etc. So here's a shadowing error message that many of you experienced users no doubt have experienced before where, you know, the most common example of this is when you enter a function, that package function prior to actually loading the package. And so then when you subsequently load the package to provide the definitions, you get this conflict and you get this general shadowing message. And so this is a, a narrative that has been around since maybe 2009. So we define a new function, basically putting new function in the global context, and then defines a package here, writes the package to file, and then loads in the test. And so there's a conflict here between the new function which is in the global context, and this new function that's now been introduced in the test context. And it says the symbol from the package context can be accessed by including the context with a name of the symbol. So that's a curious comment because currently when we load in the package, we have the package from the function from the package available already because it goes to the front of the context path. So we don't actually need that test backslash there. But the documentation seems to indicate that you do. And this was from a prior age where the lookup looked at the global prior to the context path. And so that obviously hasn't been updated. But unfortunately what happens then is that this error sort of cascades. And then it says without the context name, the symbol from the package and the definition from the package will not be used, despite the opposite of what actually happens here shown in the documentation, it is used. You type in new function 2 and you get the package definition. So that's kind of completely opposite of what happens. And then it says the symbol from global context can be removed using remove. So, so, you know, it's a subtle error, nothing sort of obvious there. You could, you could perhaps run these tests or perhaps look at this and then decide that there's everything's in order. But there was a sub. The cause of this, I think, was a subtle change made over 15 years ago where in the following. After this used to say after and it was changed to before. Okay, so this says that in the context path is global, where it has listed the context of search before context, not after context. And so when you load a package, then you know, the function that you've loaded in becomes immediately available. And so it's a completely understandable error that this could have taken place. You know, what human could possibly make that change? And then have this awareness that this area in the documentation. So the only way of detecting this is through unit test. So it's an interesting question of whether or not GPT can Detect these narrative inconsistencies. One could envisage a GPT going through documentation or computational essay and making sure that the textual inputs dovetail with the actual outputs. I think that's an interesting pathway to improve the rigor. Now I find this a really interesting example because this is probably, you know, arguably the most common error message generated in the Wolfram language. Is this typical scenario that I mentioned earlier that you, you know, you, you enter a function that you're interested in and you get a straight return of that function, not realizing the definition has yet to be loaded. You go and you know this is coming, you go to load in the package, you know you're going to get this, this error message. And it's of course much worse if you're loading an entire evaluating large numbers of statements. When you load in, you get a blizzard of these because you know if you've got 10 functions in the, in this package, when you load it in, you get 10 error messages. And this has been around for 20 years. I have an abiding memory of when I was at Wolfram in Champagne Wolfram Search the visiting scholar 10, 15, 20 years ago and a senior developer there. I was really struck every he was good enough to show me around the area for a few days and senior developer that's now returned to the company and he'd be working on his laptop and every, every four hours or something it explode is it still haven't fixed this bug I think and it was exactly this error sort of nodding on it the injustice of it all. But I guess the bug that he was referring to is that if you have a sequence of these errors it might be good to have one. Or maybe you could actually have something a little more subtle when the error happens because you kind of know what's happened. You haven't actually defined the function here. So maybe a BIP or something. But certainly, you know, cascading series of errors is probably worth getting rid of. Now chatgpt I found a very powerful tool for, for developing these systems especially. Well the first example is for naming functions or packages. In some sense it's tailor made for it because when you want to name a function, yes, you've got to work out exactly what it does, but also what impressions people have of an English word, what it does, what the expectations are. And of course that's what GPT is optimized for. It's being able to trawl the entire net to find the perfect name. So I think this is a very powerful addition to naming functions it's also a very powerful tool for research and algorithm discovery. So we used it in the packlet assurance package that I mentioned earlier. If you recall, the test summary function actually holds a few statistics in its object. So it holds the mean, the variance, kurtosis, and the skewness in the object itself, as opposed to having all the tests. And so the question was, well, given that you have those four measures for two different n different sets of unit tests, how can you combine those two? If you know, the kurtosis, for example, for two different unit tests, suites of unit tests, then how do you combine them? And you know, it's not a, it's not a deep problem, but there's a lot of messing around. You know, haven't done statistics for a while. I've got it to pull out some moments, and I wasn't able to do it, you know, quickly using a Google search or going through some textbooks because there's a few little wrinkles. But I found using ChatGPT in refining what the formula is, you know, whether it's a raw moment doing some iteration there, within two or three hours, I had the algorithm that I needed for that. And it was something that I may have just bypassed and given up on it and said I could do it. It's going to take a few days. I'm not going to go ahead with it. But with this tool at your fingertips, you can. If it's two or three hours, you're happy to do that in an afternoon. So I think this is going to be very transformative. How we use it in structures in the Wolfram language, I think is also interesting too, because being a very high level already, in some sense it's optimized for the language itself. The obvious caveat that everyone seems to be mentioning, that it does need constant interaction and verification. One of the things that amazes me in the public discourse, however, is that this is often overlooked. When people talk about chatgpt stupidity, they don't factor in the actual interaction that, yes, it may come up with a hallucination initially, but that's not the end of the story. It's a chatbot. The whole point is that you interact with it and you, you, you slide towards truth through that interaction. And I think this is, this is the game changer that, that it's introducing. However, there is this, this by play between unit tests, which are very powerful method for ensuring reproducibility, which of course is what Chat GP does not have. That's part of its advantage is that stochastic is that it's imaginative, it can go off in different directions, but pulling that creativity into line, so to speak, with unit tests, I think is a, is a very important and potentially groundbreaking interaction. And so that's why I'm quite keen to continue development on this front with Paklet Assurance. So you can do things in GPT, for example, that you can have a unit test for a GPT output by pulling out some of the parameters or checking the sentiment that the unit test itself is not 100% but it's better than totally random. And so, I mean, they obviously have this at OpenAI because you need to ensure that every new model that they build in some sense is improving things. I mean, because the Astig machines, at some level it will go backwards in some area, but the idea is that overall it gets better and better. And I think unit tests have got an important role to play. And also for AI safety, I mean, they have a red team that then tries to, for every new release of a ChatGPT version, they try and see what it can do nefariously give us your worst. And unit tests can be a way of measuring that, a way of, of working about, of assessing the capabilities of the new AI model. This of course, is all assuming that it's not smart enough to realize that's what you're doing and to bypass that in some way. So there might be an arms race between sort of building these sufficiently large and rigorous sets of unit tests alongside the, the progress of, of these ChatGPT models. Okay, so there's a lot more that I could say talk about this. I don't know. As I said, this, this for some reason started an hour earlier and I'm happy to stay and ask any questions now if, if there's anyone still online or there was a request to share the notebooks that you presented earlier. Yes, I will share that definitely. And as I said, anyone who is interested in this Packlet Assurance wants to download it and give it a go. As I say, it's transformed my workflow and yeah, I'd be interested in any feedback to this address here, Ronald@onsoc.com and hopefully it can help some folk out in building these larger systems. But yes, I'll provide this notebook that I showed earlier the other questions experiences in kind of moving from a computational essay or. I mean, I'm not in any way shape or form denigrating the notion of computational essay. It's a fantastic innovative new. Well, it's not new to the community now, but it was definitely a step up from your static latex documents. But my strong sense is that if we want to really make use of our discoveries, we need to be able to somehow work out how to translate these discoveries, these narratives in a notebook, into this general larger system. And it's not easy to do as well. I think part of the issue is that this is one of the reasons why I wanted to start this sort of educational course, is that there's a lot of ongoing work on the technical side of building packlets, which is really neat, the way it's all coming together. But there's a whole other angle about actually designing a package and a packlet, which is kind of in folklore. It's out there in the ether. But noting down what that actually involves, I think it's really important. And what one area that I found really useful has been the online meetings that Stephen shares with the developers on Twitch. To see those, you see that design in action. And I remember you get a new version of Mathematica and you sort of wonder at all the new functionality, and then you go away and try and produce your own packages, that same level of intuitiveness, and you just didn't seem to be able to do it. So there seemed to be some sort of magic that went on. I think part of it was, is that I think you generally underestimate what is needed in good design. And so it was a bit of a revelation to see, you know, how hard it is to get the design right, how hard they have to work towards it. And it's kind of great to see that, because then it gives you an idea of the magnitude of it, but also the strategies and the variables that need to take into account into designing a package and how the pieces fit together. And so, you know, this is part of the idea is, I think once more and more, people want to take their ideas and put them out into the world. They want to know how to do it, they want to know how to design these things. And by no means am I claiming to be an expert here. I'm still learning on all of these different facets. And no doubt developers in the company will have had a lot more experience and perhaps they can contribute in some way as well. But I think this is going to be the sort of next level, beyond the computational essay, how can we create these larger systems? And the other thing also is in the language itself. So one of the big differences between the Wolfram language in Python is its level of literature. So it's like a natural language. But the question is, is, does that grate against the process of system building or is that an asset for system building? You know, do they work together? Now? I've got a few ideas about that and certainly my code style changes when I'm building a large system as compared to the one liner. And the question is, what types of new functions, what kind of new grammar are needed to build a larger system? If you've got, you know, a large bunch, if you look inside some of the packages in the distribution and you compare that to a computational essay, they're like chalk and cheese. You know, one, they're completely different. You know, the appearance of them, the way they fit together, I think that's an opportunity. I think that the underlying way packages set out and the way they're expressed, there's enormous upside in the language design to ensure that's more intuitive and more natural. In fact, it's going to be essential if you're going to lower the barrier so that people can go from a computational essay. So, you know, things in the language that involve state, involve how. I guess this comes back to computational thinking as well. How you, how you pick apart the different elements of a package, of the functionality and how that's rearranged is something that is hard and it's difficult, but it's teachable and it's learnable. But actually making that concrete, I think is going to be a really interesting challenge going forward. If there are any questions on each of these, I'm just sort of talking to the ether now. I don't know if this is continuing or not, but if there's any questions on discussion on each of these 1 to 10, I'm happy to talk about any of these areas that people might be interested in. They've tried themselves. You know, getting the documentation right, what's involved in exception handling, system parameterization, I'm happy to, to answer any questions on that.