Using the Query Function to Write Elegant and Efficient Programs

Yeah. Let's do the actual notebook as an attachment. That's right. And send that to Seth.

Yes. Okay, so is the link. Should be a double. Hang on.

Sent. Email the actual attachment. One sec.

All right, maybe I have it now. Let's see. Okay, I've got it.

Yay.

Cool. All right, so.

And Seth, let me share the screen, please.

You're gonna. How are you gonna share my screen?

No, I just want to share my screen while I'm introducing. Please.

Ah, okay, so I need to hide my share screen.

Yeah, I'll just take the first turn.

Okay.

Welcome to everyone who's joining us. We are now live. We're getting ready to get started in just. Just a few seconds here. There we go. And I'm sharing. I'm sharing Seth's query book on my screen right now. Hi, everybody. I'm Jamie Peterson. I'm with Wolfram U. And it is our pleasure to welcome everyone to our special webinar event today featuring an author talk with Seth Chandler, a foundation professor of law at the University of Houston Law Center. We also have online with us Paige Villarolo, our managing editor with Wolfram Media. So if you have any questions about publishing with Wolfram Media or even general Wolfram language questions, we have staff, staff online with us who will be monitoring the Q and A pane. Seth Chandler maintains a dual life as a law professor, teaching subjects such as constitutional law and insurance law. And he's an expert on Wolfram language, presenting at numerous Wolfram conferences and winning a Wolfram innovator award in 2011. Today, Seth will be sharing examples and lessons from a chapter in his new book, Getting Information from Data with the Wolfram Language. He'll show you how you can use query on a variety of data structures to make your programs more elegant and efficient. You'll want to try some of the examples yourself. And Seth has prepared a download, a handout that we've put in a download link. If you're in the webinar interface, you'll see that in the kind of blue, green, sticky area above the chat pane. So feel free to download that notebook so that you can try some of these examples yourself. You will need Mathematica in order to work with the notebook. So in case you need that, we're going to push out a link to a free trial where you can get Mathematica and then work with that notebook, even following the webinar today, I would like to ask a poll question at this time. This will help us understand who's online with us. Your Background, whether you're an academic, a research professional, business or commercial, a teacher, a student, or even just a hobbyist who's here interested in the topic. So I'll give a few seconds for everyone to answer that question for us. Thank you for doing so. You can see the results in the poll area in our pane on the right of your screen. And we'll have that. Let's see. Yep, I see those responses coming in, so that's great. We also are live streaming today and we welcome everyone on YouTube who's watching the live stream. You can type questions in whatever platform you're using today, and our staff will help to answer your questions as we go. We'll save any specific questions for Seth Chandler at the end. So let me take another look at those polls. That poll, let's see. I can, I can even share those responses, can't I? I'm able to do that. Cassidy, maybe you can share that for me. But we have, we have a really good mix. We have, about half are identifying as hobbyist. 18% academic, 19%. Whoops, there we go. We can scroll down 20% business. Yeah. Really nice mix of people. So not surprising, the data science field has a lot of, a lot of interest across all boundaries. So without further delay, I am pleased to pass things over to Seth. I will stop my screen share so he can share his notebook with you. And I introduce Seth. Thank you so much for being with us today.

Well, thank you so much. Let me hope that I get the. Let me get up the notebook here. Okay, so hopefully you can see a notebook that starts out with a title and an abstract. And as you can see from the absence of any line numbers here, this is all live. And so we will have all the risks and excitements of a live demo. So this book is intended, really, and this presentation is intended for an intermediate user of Wolfram language, someone who's seen a map function or knows what a list is, but not for someone who's, you know, been using it for 20, 30 years. And what I want to show you is here, I want to take a look at one of them. Focus on one of the chapters in the book, chapter four, which explains how to use the query function to really improve your programming and to make you comfortable with something that's otherwise a bit mysterious. In the second half of the presentation, we'll actually do a real sort of experiment where I use some of the lessons that I've discussed in the first half. So we're going to take start with the Titanic, which is you know, sadly these days been turned into just a data set rather than a horrifying world event. But a lot of the data for the querybook is contained in a resource data object contained in the Wolfram data repository. And we will now bring that in to Wolfram language. And there it is. Good. So we have a resource object and we're now going to take from that resource object, which contains an awful lot of stuff, a list of data about the Titanic. And what you'll see is this is fairly standard Wolfram language code. And the only thing that's unusual here perhaps is this post affix thing, this, this sort of afterthought of, hey, normally Wolfram language wants to display 20 lines of a data set, but that takes up the whole screen. And so I thought for a webinar it might be better if we just had five items. So that's what the resource function format data set is doing. And so there's, there are passengers on the Titanic and you can scroll through them if you'd like and see who they are. And also we're going to be making use of an imaginary data set of a dinghy that went off the Titanic, which is sometimes useful because it just contains a few people. And so if you just want to see what's going on, having a little tiny data set is useful. So there's our dinghy. All right, so supposing we wanted to do an experiment in which we wanted to determine the mean age of the passengers on the Titanic grouped by the cabin class in which they were sitting. Maybe older people tended to be in first class and younger people like Leonardo DiCaprio and Kent Winslet and other imaginary passengers wanted to be in third class in any event. So here's a piece of Mathematica code that you, you might think would work. And the idea is to group the data set the Titanic by the class, get the age for all of those people and then reduce that list by determining the mean. And so let's actually execute that code. And what you're going to discover is that that's not a very pretty answer. And the reason that it's an ugly answer is that we're missing data on the ages of some of the passengers. And so when Wolfram language tries to compute the mean, it does what Wolfram language does, which is to really sort of literally compute the mean and to say, well, I'll just add in 30, whatever the value of missing is, I'll add in 39 of those for first class, etc. So that's not very good. And so what we should do if we're going to continue in this simpler programming paradigm is to first, first filter out people for whom we don't have a numeric value for age. That's what this select function does here. And then the rest of the code is the same. And you can see that if we do that, we do get an answer. And as I hypothesized, younger people tended to be in third class and older people tended to be in first class. And so there's, there's nothing actually wrong with this code. It's perfectly fine. But the question is whether it is as readable as perhaps one might like it to be. So let's come up with a. That's sort of the Chapter 3 method of programming. And as I said, it's perfectly fine. And you can spend your whole life doing programming this way, but I think there's actually a better way, and that is to use the query function, which is basically, it's a recipe for building functions. It's a functional programming construct. And so if you're comfortable with things like map and nest and fold and through and a variety of other Wolfram language functional operators, it's time to get familiar and comfortable with query. So the idea here is that we are going to group the data by class, we're going to compute the age of each person in the group, and then we're going to compute the mean. So all the building blocks are here. And I know for the moment it's confusing as to, well, why is this one first and this one third and this one second? But we'll get to that. But my basic point here is actually just to show you that query just creates a function. This is just, just like you can have a standalone map or a standalone folder, so too you can have a standalone query. You don't need to have any data associated with it. And once you have that functional programming construct, well then of course you can. The main purpose would be to apply it to data. And so if we now take our query named Q and apply it to the titanic data, we get this, this information. And you'll notice, by the way, that I didn't have to do any further fussing about missing data. And this is one of the virtues of using query is it's been designed to intelligently deal with missing data and to give you some control over it with the usual panoply of options that Wolfram language functions tend to come with. The other thing I want to show you though, is that Just as you can parameterize other Wolfram language functions, so too with query. So here's an example where instead of writing query to always take the mean, maybe I don't want to take the mean, maybe I want to know the minimum age of the passengers or the maximum age of the passengers, or some arbitrary function F of the passengers. And so we can now write a function class agequery that takes an argument op that will produce a function. So now what I'm going to do is I'm going to take class H query of mean and apply it to the Titanic. Of course, doing it this way just replicates what I did directly above. But you can see that I've now generalized the process. And so if we do that, we get, thankfully, the same answer that we got before. But supposing, for example, I wanted the median of the ages of the passengers on the Titanic divided up by cabin class. Well, we can do that here. And again, I, I could do, you know, I could take the minimum, I could do all sorts of things. And as I mentioned, I haven't had to deal with missing values at all. Wolfram language has been smart enough to figure out what to do with missing values on its own. Okay, so you'll notice the syntax I've used thus far is to explicitly use, use query, and this is in fact the syntax that I prefer. But there's an alternative syntax which I'm going to call implicit query, where you don't actually write out the word query and instead you sort of substitute in the name of the data on which you're operating. And this is executed the same as if I wrote out query. So this is gonna work the same way. Some people like implicit query formatting, some people prefer it. The way I do it here, that's really up to you. You, it, it's your choice. Okay, so what else can I tell you basically, about query? Well, for one thing, it takes options, and the main options are what to do with missing data, what to do with what happens when it doesn't work. And you can see the default behavior is if anything goes wrong, it just stops. And then finally, what to do about some issues with parts of data sets. So supposing, supposing I was concerned that there might be missing data, and I didn't want Wolfram language to just treat missing data as if it wasn't there. Maybe the fact that the age is missing, maybe it's a flag that we're not really sure what cabin class the person is in, and so we just don't want to include them at all. So one way to do that is to change the value of the missing behavior option to none, and then it's going to behave exactly as the way that Wolfram language behaved when, when I wasn't using query at all, when I was just sort of using an old fashioned group by construct. So that's one option. And you can see that. This seems to be duplicative code. I'm not sure what's going on here. If, if I for whatever reason wanted to spell out the missing behavior option, but I wanted it to ignore missings, I could just go back to missing behavior is automatic, although it'd be simpler in this case not to include it at all. Okay, so that is in some sense a minimum amount that you need to know to make query work. The hard part about query is to sort of have in your head a mapping that goes from what is the order of the arguments here and what the. What code is actually going to be produced? And you One way to get an answer to this question is to spend a lot of time reading the extensive Wolfram language documentation on query. And there's nothing wrong with doing that. In fact, it's probably a good idea at some point in your life to go through that. But what I'm going to suggest is that there are actually some shortcuts that I think will save you a lot of trouble. And that I think are sort of a pedagogically useful way of understanding what query does. But what you sort of want to know is that what query does is to produce a pipeline of operator forms that is sort of shortish functions that are chained together to operate on the data. Now the exact pipeline that gets built depends on what order you put these, these functions in. And the sort of general rule is, and this is a, an oversimplification that it might be considered an unofficial simplification, people might not like it, but that your first argument works on the whole expression, the second argument will work on the first level of the expression, and subsequent arguments work at lower levels. That's not exactly true, but it's sort of a useful first approximation of what's going on with query. So let's actually illustrate this and we're going to do that through a little toy expression G of 8 of H of X, Y and 9 and we'll just call that expert. And you can see I'm not gonna use query here at all. I'm going to map V over the expression and then I'm going to take that output and apply U to it. And this is sort of the typical inside out functional programming style that many people use for Wolfram language. I use it and I certainly started out using it. But what I'm going to suggest is that there are some alternatives that you can use with query. But in any event, here's what you get and you can see that V is being applied at the first level of the expression. It's being applied to 8, it's being applied to the H of X, Y9, and then after that we're just going to take U and apply it to the whole thing. So that is certainly one way we could create this expression, if that was our inclination. But let me show you a simpler way, which is to just say query UV of expression. And what you can see is that this is going to give us exactly the same result as we see above here. And you might say, well, that's just not really that much simpler than this. Why do you need query? And yes, it's true that in this particular example, query is not the greatest lifesaver of all time, but there are certainly other circumstances in which writing your pipeline, pipeline of functions using query or implicit query is going to make your code look a lot better. And once you get used to it, you're going to realize that it actually becomes more readable. So how do you know what the order of operations of that occur as a result of applying query? And there's a simple answer that's buried deep in the documentation, but I'm going to give it to you here, and that's just wrap normal around the query, don't include the data, just use the query, wrap normal around it, and it's going to give you this answer here. So if we apply it to my query uv, it's going to say map v U. For those of you who aren't familiar with slash star, the technical term is right, composition. It's a composition of operators. The plain English way to read it is and then so map V over the expression and then take your result and apply U to it. If you want to see the full form of what I just did, you can certainly do that. And so now you know the order of the operations. Because when you take normal of a query, it's just going to produce this long, potentially long pipeline of what it's going to do. And so instead of reading the code inside out, as you often do in Wolfram language, as I'm illustrating here, you read it left to right and so you're the program looks procedural and yet it's still a functional pro. It's a functional programming style that looks procedural. So let's look at another example now. And here we're going to take a piece of code that's query, first argument, group by F, second argument is G, third argument is H. So what's it going to do? Well, to find this out, we're going to wrap normal around it. Remember, if you take one lesson from this webinar, it's when in doubt, wrap normal around your query and it will tell you what query actually does. And if you do this enough, it's like, you know, you hear language enough and you pick up how it works and you're able to generalize. So what's going to happen? Well, first it's going to group the data according to F, whatever F is, and then it's going to map over the result, another composed operator, which first maps H over the result and then applies G to the rest. Now, I will tell you that this is not perhaps the most legible code because it's got this nested mapping in it. And it would be so nice if the Wolfram language kernel were to output something like this, which was to say, first group it by F, then at level two of the result, apply H and then apply G. This is identical actually to this. It's just that it uses the nice map level resource function, which as I say here, should be promoted to be part of the main language, but hasn't yet, and then it applies G. So, by the way, some of you might be curious about, well, what did Q do? Remember old Q here, which we created way, way back here? Well, we can now see what does Q do. And the way we'll do that is we'll just apply normal to it and we're going to see, well, what Q does. And I'll just put Q there so you can remember it. First it's going to group by class, and then for every resulting class, it's going to map over that. And what's it going to do for each individual? It's going to compute their age, and then it's going to take the results of those ages and compute the mean of it. Okay, so that's all great. The, the issue though, is that all that normal does is it lets you figure out the order of operations of a query that you've constructed. But that might be frustrating because if you can never get the query arguments to produce the order of operations you want, all normal is going to do is to tell you every single way that You've gotten it wrong.

Wrong.

So what I now want to do is to sort of move to the next level of understanding, which is to have an intuition about how to structure your argument so that when you do normal it to check 95% of the time it's you're going to get it right instead of being frustrated and thinking that you got it wrong. So there are a couple of rules that we need to know and a couple of exceptions. The first rule that I think you should know is last in, first out, that is which accountants will know is the LIFO principle. And it's the order of operations is generally in the opposite order of the arguments to query. So in the example above here, QueryUV, first it's going to do the V part and map it, and then it's going to do the U part. Things that come later in the query. Arguments that come later tend to go to the front of the line when it comes to the pipeline. And if you think about it, this actually makes sense because if the function is operating at a higher level, you can sort of destroy things before you even get to the lower level. And it's generally easier to think about. Well, let me work at the lower level first and then take those results and operate at a higher level of the expression. What I have here in this notebook is a visualization of this process. And I will say in working with various people, some people find this very helpful, other people find it more trouble than it's worth, I guess, since I wrote it, I think it's helpful. So I'll just show you that you can actually visualize using this function labeled query tree form, what the query is doing. So here's a query uvw I'm gonna put it in query tree form and then just cause I like to frame things, we'll frame it. So what you can see is, well, first it's mapping W over the data and then it's applying V and then it's mapping that and then it's applying um. So again, if you find this helpful, good. If you don't, that's fine. Here's another example where the first argument to query is itself a composition. And you can see how this gets implemented in which this part, the V and W part is the same as it was before. But now we sort of have three things going on. First we're doing this right composition here and then we're right composing the results first with U1 and then with U2. And here's yet another more complex example which just illustrates the same idea. Okay, so that's the first heuristic is last in, first out. But is that always true? And the answer is no, it's not always true. The motivation for the LIFO idea is that you want to do things in an efficient way, but not wreck anything. So the example I might want to give you is sometimes you, you don't want all the data, you want to just look at little pieces of the data and then combine it. So rather than load every row of data in, do the analysis, and then throw things out, it might be simpler just to get little pieces of the data first and then process that. But that's sort of a general concept. And the general concepts is do things that are efficient and won't wreck things. But that's not really an idea that the kernel can implement. And so instead the developers of Wolfram Language have come up with some formalizations of that that will generally accomplish this goal. So let's look at this piece of data and you can see how order of operations might matter. So here's a list of lists, 3, 4 and 5, 7, and we're going to run a query on that in which the first argument is F, the second argument is G. And what you'll see is that first it's going to apply G to 3 and 4, and then it's going to apply G to 5 and 7, and it's going to combine that and apply F to the whole thing. But we might not always want to do that. And so let me give you an example of when this might matter. So supposing we have this data structure, and what we want to do is we want to add one to the second part of each expression and then only select the results that are for which the total is now even. Okay, this is an example where order of operations might matter. If, for example, the first thing we do is to add 1 to the second part, that will turn into 3 and 5 and 5 and 8. And then for the total of 3 and 5 is 8, the total of 5 and 8 is 13. And so the only thing it's gonna select is the 3 and 5, because that's the even thing. So if I, if, if the order of operations is first to add and then to select, I get one result, which you can see here is if the order of operations were different, I'd get a different result, because if I took three and four, that's odd. So it wouldn't be selected. If I take five and seven, that's even. It would be selected and So I would not get the same result. Okay, so let me talk to you about what operators cut into in line, as it were. One of the operators that cuts in line is select, because oftentimes you want to get rid of data before you you operate on it. And so many of the operators that cut in line early are converted by Wolfram language into something called a slight slice. So here are some examples of operators that are slices. This span function here, which says to take from the first to the third items, that's if I normal that query, it's going to get converted into a slice and it's going to go early a take operator. Again, you wouldn't want to process all the data and then only at the end take the first five rows. Rather, it's much more likely that you want to take the first five rows of some large piece of data, just look at that and then do something. So again, it makes sense that this take five comes first and you can see it gets translated into a slice operator. Or if you just have something wrapped in key, which is a way of getting data out of an association, that too will come first. What else is useful to know? Sometimes you'll see code that looks like this. You'll see a query where there's a string as one of the arguments to query, and that's gonna be converted into a slice. So if I say query A of F of an association that has A going to 3 and B going to 4 4, what it's going to do is it's going to take the A slice of this, which is 3, and then apply F to it. So the slice has cut in front. And you can see that here when I, when I normal this query, you can see that the slice operator A comes ahead of F. So there are lots of variants of slice select that also cut in line. I showed you that. Select cut cuts in line, but select first key select and things like maximal buy and minimal buy also cut in line. They go ahead in the pipeline because they're all just essentially getting rid of most of the data. And that's generally a good thing when you are trying to do complex operations on a big data set. And all this piece of code does here is, is to just illustrate this. Here I'm just taking the normal form of various things. Select even Q, select first even Q key, select even Q, blah blah, blah, blah blah. And you can see that in every one of these circumstances the G comes second, which is contrary to the sort of default rule that I told you about, which was Last in, first out. But things like select and slice are exceptions to that rule. What else comes first? Frequent things that cut in line are deletions, because again, you're getting rid of data Cases often comes first, and frequently sorting operators will be done first. That's a point that we'll take a look at later in the webinar. All right, there's another technicality, which is that the form in which you express the function matters. Remember what I told you is that Wolfram language is not yet, nor am I sure it will ever be smart enough to just straightforwardly implement the idea of do things that are efficient and won't wreck things. So instead, the developers had to come up with some forms that they thought should be applied first. So let me show you an example of this. And the illustration I want to give you is the function sort and the function sort of hashtag ampersand, which you might think this is exactly the same thing as sort, but query is going to treat them differently. So let's take a look here. Here's our matrix. And if I say sort the matrix according to the total, it's going to do this differently than if I say, if I use this form of sorting. What's going on here is that, well, let's, let's actually normal this, okay, if I do normal of this, you are going to see that first it sorts. It understands that sort is something that should come cut in line and then total it. And so it's going to, when, when you say sort here, it's going to sort it by the first element. And so the first element is, the first element is going to be this negative 19 thing. And the sum of negative 1 and 9 is 8. And then the next lowest element is 6 is, is 1. And so when it adds 116 together it gets 17. And that's why you're seeing this result be 8, 17, 13, 9, 7. But here, if I don't give it a straightforward sort, but I give it this slightly more baroque form of sort of, you can see that the order of operations is different. First it's going to calculate the total, and then it's going to sort things by the total. So if I first total everything, it's going to turn out that the lowest total, let's see, 13, 7, 8, 9, 17. The lowest total is 7. And so 7 is going to come first, and then this negative 19 thing is going to come second. And so what you can see is that the outcome of this query is going to depend on how you Write the function. So just be careful when you're using query to recognize that form matters, the book cover of which is here has a much more thorough discussion of this issue of the form of various operators and the way that they affect the order of operations. If you're ever in doubt about whether something by the way, is a descending operator, meaning it cuts in line, or an ascending operator, meaning it doesn't get to cut in line in the pipeline, there is this somewhat obscure function called descending queue that you can run to determine whether a function is going to be treated as an ascending or descending operator. Okay, so I've told you the basic rule, and remember what to keep in mind. Generally it's lifo, but there are some functions that cut in line, particularly those that tend to get rid of data. But in order to use query effectively, it's also useful to understand a few complications. And so I'm going to go through four of those complications right now. The first is what happens when one of your arguments to query is itself a composed function. So I'm going to give you an F dar G or select F slash dart G. Etc. Okay, what do you then do? Because what if select F is something that cuts in line, but G does not cut in line, Does G sort of get the benefits of being right composed with select F, or does it not? And I'm going to tell you the easy case. If only all the operators within the composition are ascending, I.e. they don't cut in line, then composing them doesn't change that they don't cut in line. Or if all of the operators are descending, meaning they do cut in line, then when you compose them, they still all cut in line. The complication comes when some, some of the operators in your pipeline within an argument are ascending. That is they don't cut in line and some of the operators are descending, they do cut in line. So what happens then? So let's take a look at this experiment. Okay, so here we have the first argument to query being composition of functions. We have select, which generally cuts in line. We have sortby, which generally cuts in line. And then we have H and I, which aren't special cases, and so they generally would not cut in line. So, and then we have the second argument is J. So what's going to happen here? So the first arguments, select F and sort by are going to cut in line. Okay, but H and I, because they are not privileged, they're not line cutters, they're not descending, they don't get any sort of special benefit by virtue of being put in composition with select F and sort by G instead they get placed after J or map J here as it's composed and they then come afterwards. So they, they. When you mix ascending and descending operators together, the ascending operators that come at the end don't get the benefit of having been prefaced by descending operators. So let's just take another look here. Here's another example. This one's a little more complicated. First we have select which cuts in line. Then we have G, which is doesn't, it's not a privileged operator. And then we have sort by H, which in the ordinary course would cut in line. So does sort by still get to cut in line when it's preceded by something that doesn't? And the answer is no, it's lost its special privileges. So if we look at what's going to go on here, what we're going to see is that this whole thing has sort of been placed, it's lost its special rights and so J is going to come first and then we're going to see this whole pipeline of operators applied. I know that's complicated and frankly, a lot of the times when I have to use constructs like this, I do what I've told you to do, which is to just do a little experimentation, normal and see if it gets what you want. Okay? And the great thing about normal is I've never seen a normal operation take more than an eye blink to complete itself. This is very fast. So even if your data is super complicated, running normal on a query is not okay. Second complication is group by and group by is sort of a special case because it does kind of destroy things. It restructures the data and it doesn't fit this paradigm of getting rid of data. But nonetheless it comes first because that's generally what you want. So for example, if we were to imagine this query here, and let's actually just put a normal around this, what you're going to see is that it's going to first group by sex and then map the length. And if you think about it, that's kind of what you want. If, if, if you were to take the length of each item of data, it would just tell you, well, the first piece of data has a four items and the second piece of data has four items and the third piece of data has four items. But then when you asked it, well, what's the sex of that? It would say, I don't know what the sex of 4 is. 4 doesn't have a sex. And so instead what it's going to do is it's going to group each of the passengers according to their sex and then it's going to determine how many such pass passengers there were. Okay? And so we can see that illustrated here. We'll take our dinghy and now we'll run our query against the dinghy and we will see that there were in our little dinghy there are four females and one male who managed to get on the boat. So this issue of form recurs, that is how you write the group by matters. So that if I don't write it as a sort of a straightforward group by, as I did here, but write it as a function in which group by is sort of one of the arguments to the function. And I then run this on dingy, it's not going to understand understand that it should first do the grouping and it's going to fail. And you can see by running normal what's going on here. First it's getting the age of all the passengers, it's computing that mean, and then it's trying to group by. But the, the sex data is gone. Whereas if I write the group by this way it works much better as we want. So that's an example of again, the form of your operators matters and again, by running normal you could have figured this out. Okay, so let's actually do some problems. And what I want to do is to just illustrate a few ideas here. First I'm going to do it the chapter three way and, and then we'll do it the chapter four way using query. So supposing I want to get the sex of everyone on the Titanic over the age of 70. So here's how you might have learned to do it. Take the Titanic, write an anonymous function that looks for the a whether their age is over 70, use that as a filtering basis and then once you've done that, get the sex of everybody. And that works. Okay. Or if you like using a functional sort of a pipeline, you could build your own pipeline. Here's the pipeline. A select operator, select everyone over the age of 70 and then determine their sex. So that also works, but here's the big boy way or the big person way and it's to construct a query and in which the first argument is to select and the second argument is to get the sex. Query will then construct a pipeline of operators for you and you can get the information on the Titanic. So let's do a couple other examples. One is to how would we get the median age of the passengers on the Titanic who were under the age of 50. And again, here's the way that you might have learned to do this. And I would. I might even go through this because I think this is pretty hard to read. It will work. But I personally find this kind of code hard to read, as I think do many people. So you could again, construct your own pipeline. And there's nothing wrong with this. First select the people who are under 50, then group them by whether they survived. Then get their age and compute the median. And I don't know why that's there. Well, maybe there was a reason it was there. Hold on a second. Oh, I see why it was there. Okay. It was there because first I construct the pipeline and then I apply that pipeline to the Titanic. Okay, but here's a simpler way to do it, I think, which is to use query. We have a composed function as the first argument. First we're gonna select people under the age of 50. Then we're gonna group them by survival. We'll get their ages, and then we'll compute the median. And so I think that's a simpler way of figuring it out. Okay, here's another example. What happens if you have two data sets and you want to join them? As you know, this is what relational databases are all about, is joining data. So let's have an imaginary data set in some sense about what were the cabins like on the Titanic. We'll get this out of the resource data. And then in the old days, before query, and still today, you could join the two pieces of data together, the Titanic data set and the cabin data set. And the linking factor would be the cabin class, because both the original Titanic data set and the Titanic cabins data set have this class column associated with it. Just one footnote that isn't really part of my book. But join across does not like data sets. You've got to get it sort of out of the data set before you can join. And so that's what the. That's why there's a normal here is to convert a data set into just a bunch of associations. So you could also do this with query. And I show you here how to do that, where you're making. Running a query on the cabins. And that will work. All right, let me skip down, because I can see time is getting away. And another thing that you need to know how to do is how to do nothing. So supposing you don't want to do anything with the data. All you want to do is to take the Titanic data and Just get the age and sex out of it. And you want to do that on every row. If you were using I'm going to get myself in trouble here. But if you're using pandas or something like that, you've basically put nothing as the argument. Wolfram language doesn't like putting having blanks. And so just use all, say basically get me every row and for every row get me the age and sex of the people on the dinghy. Or if you don't like all, you can use identity, which basically does nothing. And that's another way to do nothing, which is sometimes what you need to do. So one virtue of WOLM language is that it and queries that it can work with hierarchical data sets, deeply nested data sets. The one that's I'm going to use is the planets data set, which you can see here. And supposing I wanted to compute the mean. I'm sorry, I wanted to compute the radius of Phobos. So we can just use sort of a nested query where first we get Mars and then we get its moons and then we get the moon Phobos and then we get its radius. And so that would be a way of doing it. And we learned that Phobos is really, really small. Here's a little more complicated thing. Supposing you wanted to compute the mean mass of the moons of Mars, something that. There's a wonderful video, if you can find it, of Steve Wolfram himself struggling to figure out how he could do that using query. But this is the one occasion on which I know how to do something he doesn't. And so it's. This is the order of operations. You could see what the order is by just normaling this, as I've told you many times. So it's going to say get a slice, that's the Mars, then the moons and the all than the mass and then compute the mean of the result. So the mean mass of the moons of Mars is 6 times 10 to the 15th kilograms, et cetera. All right, so let me skip that and let me talk about what happens when things go wrong, because things will go wrong when you use query or use any programming language. You're going to make mistakes and learning how to stay calm and cope with them is a big skill. So let's get our dinghies again. And supposing what we want to do is we want to get the age of the passengers and sort the list of ages within each passenger group.

Group.

So here's what I might do. I'd say group them, sort it. And what you're going to see is that it does get an answer, but it's not right. Like I thought it was supposed to be sorted, but 16 is coming ahead of 2. That can't be. And so again, run normal and you'll see what the problem is. You'll see the problem is that the sort of is coming too early in the pipeline because we don't know the age yet. And so it's just sort of sorting on the value of the row rather than on the age of the passengers within the row. So what you can see is that you can change the order of operations. In the book I talk about how you can have a query within a query to change the order of operations and you will then get the result that you want. All right, so what else can go wrong? Well, here's something that can go wrong. Supposing I want to, for some obscure reason obtain the second digit of the passengers ages. You would think that would work, but it fails. And you might want to know, well, why? Why did that fail? And it doesn't even produce an answer at all. Remember what I told you was that the default failure action for query is, it just, it basically stops. So what you can see is the problem here is that many of the passengers had only one digit in their age, they were eight years old. And so when you ask for the second digit, it's telling you, hey, there is no second digit. So how could you, how could you sort of figure that out without this sort of, this is sort of cheating. Like I sort of knew what the problem was and I could try to show it here. But what if you didn't really know what the problem was? Well, one thing you can do is to change the failure action and say, don't abort, just keep running it, set the failure action to none. And then if you run that, you can look at this output and say, oh, wait a second, One doesn't have a second part, that's why it's failing. Two doesn't have a second part, that's why it's failing. And what I go through in this part and which is gone through in the book are other alternative ways of working with failure action to either debug, as I kind of did here, or just say, look, if you've got a problem, just kind of skip that row and move on. And that's what failure action drop does. Because sometimes you don't want a huge computation aborted because there's, you know, you got a 50,000 row data set. And item 33,007 has a problem with it. You don't want the whole computation to collapse, so just tell it to drop it and that'll work. I mean, it's dangerous because there might actually be a real problem there and you're now sort of ignoring it. But if you just want to get your work done, that's a good idea. I'm going to scroll down, skip this. There's another thing you can do called Encapsulate, which provide gives a capability of sort of getting a lot more error reporting out of failure action. And I show that here. All right, so what I want to get to now is actually let's do a real experiment. And what I want to get to is let's. I work in the legal field, as Jamie Peterson told you at the beginning. So let's actually do some research on the United States suit Supreme Court. And there's a big data set maintained by Washington University in St. Louis on decisions of the Supreme Court since 1946. They have decisions that are earlier, but the data is not as complete. So what we're now going to do is we're going to import to basically suck in a zip file. And if we do that, we can then it'll tell you, okay, here's how many files are actually contained within the zip file. As it happens, there's only one zip file here. And now knowing that, I can say, could you please get me that first zip file? And so we will then get a CSV file whose dimensions are 9161 by 53. By the way, this is also a really useful resource function, but we want it to be in the form of a data set. So this is not in the form of a data set. Maybe if I augment terse this at 10. Yeah, you can see a little more clearly that this is not a data set. It's just a list of lists. What if I want to turn it into a data set? Well, here's some code whereby I could do it manually. That'll actually work. By the way, you'll notice Here I have 20 rows of data being produced. I might only want fewer rows. So there's. Here's a way of doing that. You can just set the data set to have a default of a maximum of eight items. Here's another way of importing in which you say, look, take that first file and import it as a data set and treat the first line as headers. This, I think, is a little more sophisticated way of reading in data. And there we go. And we can see, by the way, if we take but look at the first row of that data and take the keys of it. But if I didn't normal it, by the way, it's still wrapped up in a data set, which is sometimes annoying. So to get it out escape the data set, we just use normal. And so here are all the columns in this data set. And these have, you know, I have to look in the code book that's published online. But I, as a law professor or as a lawyer, I would have some idea what these things mean. All right, so one thing I might want to know is what sort of actions does the Supreme Court actually take with respect to cases? And so here, what I'm going to do is I'm going to say take the case disposition field or column of this data set and then how many are there of that case disposition? And then I want to know from biggest to smallest. So I'll. And then reverse sort it. So if we do that, it turns out that disposition 2 occurs the most frequently. Disposition 4 occurs next most frequently. I will say that's kind of a useless answer because nobody knows what's a disposition 2. I don't know what disposition 2 means. However, Washington University in St. Louis has a code book. We can sort of implement that code book in Wolfram language. And then we can then use the key replace resource function with query to get a more coherent answer. And you'll learn that the most common thing the Supreme Court does in some sense is to affirm, that is to say, the lower court got it right. But there are lots of other things that go on which are various modifications of either affirmance or reversal. All right, let's come up with another thing. Supposing you'll notice these are absolute numbers that might not be what we want. Maybe we want to know what fraction of the time, what percentage of the time the Supreme Court takes various actions. And we can use our friend query again to kind of normalize these results. Here's our little pipeline. Take each item divided by the total of all the items, and then round the result to the nearest 0.001. And so if we do that, we can get a result. All right, here's a more complicated experiment. This is actually something that I actually thought this was kind of interesting, which is people complain that the current Supreme Court reverses precedent too frequently. That is, most of the time courts are supposed to follow the rules that were set down by prior courts. That's how a common law Judicial system tends to operate, such as we have in the United States and other sort of UK derivative countries. And that principle is called stare decisis. And people have been complaining that this Supreme Court is reversing precedent too frequently. So is that in fact true, or has the Supreme Court always done this? Maybe it did it more in the past. And so we can use the tools of query to explore this. So what are we going to do? We're going to. Here's one thing we could do. We're going to look at cases in which precedent was altered. We're going to determine whether that alteration was more liberal or conservative. According to the people at Washington University in St. Louis, we'll sort that so it's consistent. And then we will group those determinations according to who was the Chief justice at the time. That's generally sort of a metric of when does one court change? It's when does the Chief justice change? Change. So there's our query. You'll notice that the query doesn't yet have any data on which to operate. If we're confused about what order it's gonna. This, this complicated query is gonna operate in, we can use normal on it and we'll learn that, okay, here's what it's gonna do. First, it's gonna figure out whether the precedent was altered, and then it's gonna group it by the decision direction, etc. All right, and if we then run this on the Supreme Court directory, what we can see is actually it looks like this current court, where the Chief justice is, John Roberts, may not be as active in reversing cases as were prior courts. We can do a little fancy thing here, and you can see how I'm going to use a switch statement within a group by here. And what we're going to see is that actually the Warren Court was more active, the Burger Court was more active. It's just that within the Warren Court, it tended to move more in a liberal direction. The Burger Court was more split, and the Roberts Court has moved in a conservative direction. So one of the things we could do to explore, well, maybe this is just because the Warren Court lasted longer and the Burger Court lasted longer. So what we might want to do is to figure out is to sort of normalize what the rates are at. How many cases per year was the Roberts Court reversing in a conservative direction and the Warren Court reversing in a liberal direction? And this code here is going to do that. It's going to make use of the structured date way of interpreting Data. Because if you look at the original data here, you can see that it has its dates in kind of a crate a strange way. It represents dates in a strange way. And fortunately, the structured data of Wolfram language is smart enough to figure this out. What I think it's done. Thank goodness. And so what we'll now do is you can see how many years each court lasted. And what we can now do is use merge as one of the functions within query to bring together this. How long did the court last information with what sort of precedent alterations did they engage in? And if you do that, here's the code within query to do it. As it turns out, the Roberts court is not reversing precedent really any faster than any other court. What's alarming to at least some people is that they're reversing it in a conservative direction at a higher rate, with a higher ratio. Although other people might be alarmed that the Warren court had been reverse precedent a lot in the liberal direction. All right, that's my little legal experiment. The final thing I want to show you is that I use Google Sheets a lot. I don't know about you, I find it a very way of sharing documents. But Google Sheets doesn't do everything, not yet. And so it would be nice to get Google Sheets into Wolfram language. And this is something that many people don't know how to do. So let me show you how to do it. You need to get the URL for your spreadsheet, and hopefully you know how to use Google to do that. And then all you need to do is to change this last little bit of the URL from USP equals sharing to export as an XLSX as an Excel file. And once you've done that, it's going to be just like an Excel file. So if I now say, well, what are the sheets of this Excel file? And by the way, this is some Excel file, or I'm sorry, some Google Sheets file that somebody produced, I think the center for Disease Control produced, about vaccinations in the United States. Anyway, once you have that, you can start to bring in all sorts of data and you can import it as a data set. And so you get all this data on when were people vaccinated, and hopefully these Google Sheets People, People. There we go. And then once you have this data as a data set, you can use all the techniques that I've taught you to get information out. Let me show you an error. Okay, so it's often useful to see things go wrong. So supposing I want to know, well, how many vaccinations did occurred in each location. So here would be the code that I would use to do it. But when we run it, we see what's all this business and what this business is of the 14 empty strings is that the people who made this data in Google sheets coded empty coded missing values as just empty strings. And that's not what we want. So what we can do is there's an option to import which says import. If you see an empty field, treat it as a missing. And once you've done that, then at the end of the day what you will see is that Wolfram language, see how it's these dashes are with the way that data set represents missings. And so now it'll work just splendidly because query knows how to deal with missing values. And you can do all sorts of things like produce timelines of when did vaccinations occur. You'll notice by the way that when I do it this way, the graphs don't have a lot of stuff in them. That's because data set tends to suppress details. One of the things I go through in the book is how to use something called an item display function to really fine tune the visualization of what's in a data set. And so I'm going to show you that here where now we're learning, you know, when did things happen. And you can even do very fancy things like here I'm going to combine data from Colorado, New York State and Texas. Okay, so the book has lots of other stuff in it which if you download this notebook, you will learn about or better even yet, just buy the book. There's a very elaborate index for the book created with great care, which will help you navigate. There was one other thing I wanted to show you and that is the following sort of nightmare, which is let me reshare screen and I'm going to show you a different notebook. So right, you know, there are these things, large language models and you know, one question you might have is do I really need to learn all this this Professor Chandler, can't I just use the new chat notebooks features of Wolfram language and have it do the work for me? And the answer as of December 2023 is you still have to buy the book. Although it's not bad. And maybe in three, four years you won't have to buy the book or I'll write a different book. But here's the question I asked. Suppose I have a data set such as the Titanic in which the columns are class, age, sex and survive. The age values are mostly present, but sometimes missing. How would I obtain the mean age of the passengers based on class? And actually, the chat feature gives us an answer. It's not exactly the same answer that I. I would have provided. For one thing, it's in this implicit query format, and for another, it uses left composition rather than right composition, but it actually works. So that's another option is if you're having trouble, just run a chat notebook and it might actually give you a very good answer. By, by the way, one thing I was going to say is when I first gave a presentation like this about three, four months ago, and I did this, chat failed. It didn't do this, right? And so it looks to me like chat is getting better. And so then I wondered, well, does that mean that it can do everything? So here's another question. Find the median age of Titanic passengers restricted to those under the age of 50 based on whether the passengers survived. And it attempts again to come up with an implicit query. But because evidently chat had not read my book with adequate care, it botches the order of operations and this is not going to work. And so the moral of the story is that you still need to buy the book to learn how to use query properly or read the documentation at great length. But I'm being slightly facetious here, but using chat enabled notebooks may well help you. And so I would use the two of them together. And I think by using both of them together, you will end up sort of dealing with all the boring stuff in data analysis, getting your data in the right format. I don't think Wolfram Language would have let me do so, and I didn't think of it at the time, but maybe the title of this book should have been Query how to Deal with all the Boring Stuff. Because it's the boring stuff, as you know, that ends up taking 80, 90% of your time. Once you've got the data in the right form, then Wolfram Language has absolutely wonderful routines to do machine learning and statistics and clustering and whatever else, plotting, whatever else you want to do. And in the real world, the struggle is often getting your data in the right form. So I hope that this has helped you understand how to use the query function, as I said, to write elegant and efficient programs. And as I understood it, there was going to be some opportunity for me to answer questions at this point.

Thank you so much, Seth. We have been, Our staff has been doing a really nice job staying on top of the Questions, and we've answered many of them. I see one in that Q and A pane. If you click that next new column, you'll see one new question from Douglas.

Yes, I see that.

Oh, I see. It's more of a suggestion, a tip, right?

Yes. You can use Join at different levels to either add rows or columns to data. And I believe that is covered in the book. There are several ways of doing it, but Join if we. I don't know if it will show up, but if you're using Wolfram language and you go to the documentation for Join, what you'll see is that there's an optional last argument to join, which is join the objects at level N in each of the list I. And although it doesn't really specify it there, this will also work with data sets, so you can add both rows or columns using Join at the appropriate level.

John is asking if in your handout you cover that Google Docs sharing. Is that.

Yes, that. That is in. It's in the notebook. If you scroll down to the bottom of the notebook, it should be there just like the. The rest of. Of the stuff.

Thank you so much. And I think we've got. I think that covers all the unanswered questions, actually. And if you want to click over to the answered column, Seth, you'll see what, what we did. Go ahead and answer already. I. I did post a link to the survey in our chat pane. So if you're looking for a attendance certificate, you can do that by answering the question on the survey or if you have further comments you want to share the survey is another way to do that, to share your feedback with us. We also have. Paige had posted about the book release on our Wolfram community, and I posted there about this webinar. So we'll remind everyone about that community link when we send the recording email out probably later today. So that's another place where we can all communicate with each other following the session.

Great. Could I just say two more things in response?

Absolutely.

In the answer questions. The first is yes, I plead guilty. I used a beta of version 14 to draft this notebook, but absolutely nothing in this notebook depends on version 14 because the book was drafted in version 13. And so if you're running version 13 and you don't have access to the beta, don't worry about it at all. You may get a little warning message saying this was created in a later version. Just ignore the warning message and everything should work splendidly. The second thing I noticed was I do make extensive use of resource functions in this notebook, like format, data set and key replace and the like. And the book does go through some, I think, very useful resource functions that will save you a lot of. You won't have to reinvent wheels. And I would also just urge you, if you're a Wolfram language enthusiast, as part of your sort of daily news reading scan, to see what's new in the function repository. It's a place where people. It's sort of the equivalent of packages. Not quite. That's the packlet repository, but it's sort of little mini stuff and often there's. I mean, it has nothing to do with my book really, but there's really fascinating stuff in there. People also sorts of diverse ideas about how to use Wolfram language. So I make use of the function repository in my book and I would urge you to make good use of a function repository when you use Wolfram language.

And Rob, thank you. Is reminding me to share the poll that we had set up. And that poll is asking about if you could add a new chapter to the query book, what would be your highest priority? So.

And what.

Sorry, who.

What chapter? One.

Oh, it's. It's just now out, so people.

I see. Okay.

If you go to the open poll area, we'll start to see those responses. Excuse me.

Okay.

And of course, people might want the opportunity to read the full book, if they haven't done it already before commenting on that. But that would be an interesting conversation on community as well.

Okay, good.

Yeah. Thanks, everyone for responding. Well, thank you so much for sharing time with us today, everyone, and it's been really a pleasure to have you here talking about your book. Thank you so much. So many good tips. You covered a lot.

Thank you. My pleasure. Bye now.

All right, everyone, we will close things up. I'll leave that poll up for just a few more seconds, but remember to go to the survey to share feedback and request a certificate, if you like. Thanks, everybody. Bye for now.