EntityValue and EntityStore: Accessing Your Own Data via the Wolfram Data Framework

Hello, everyone. Thank you for joining us today. Sorry for the delay getting started. Hello.

Hello, Paul. Welcome to our live stream today, leading in talking about entity stores and a variety of other things. If you have questions, please put them in chat. I'll let Noah also introduce Aaron, our guest here.

Aaron Enright is our guest today. Senior information Scientist here at Wolfram definitely knows his stuff and is here to share it with all of us. So without further ado.

Hi. Hello. Thanks for having me today. Yes, I have stuff to share on Entity Value and Entity Store and how to create your own entities. Let me get my screen up here. There we go. Entity Value and Entity Store. These are ways of accessing data, some curated data through Entity Value and through Entity Store could be your own data or other data sets not provided by Wolfram, but out there in the wild and using them for computation and computational exploration. So what is Entity Value? It's a way of getting curated data into your Wolfram language set session. There are many ways of accomplishing this. We have entities which are, you know, things that exist out in the world, and we have values, properties of those entities and their corresponding values.

Oh, okay.

So hence entity value squished together. Yeah. So the easiest way to access an entity or an entity property pair is through control equal. If you press control equal, you get a natural language interface to our knowledge base. So if we want, say, the city of Brisbane, enter Control equals and press Enter. And there you go. The orange blob there is just a pretty representation of the entity parts of this rectangle that are important. Here are this button here with the three dots for alternate interpretations. Clicking on that, we see that Brisbane is interpreted as a city and it's using the one in Australia. There's one in the United States that could be used too. You could also click and use it as an administrative division. We're happy with it being the city in Australia. So we'll check the box here. Click, click the checkbox there to accept the interpretation. Now, if we wanted more than that, we wanted the city of Brisbane's population, we could say population of Brisbane. And let's go even further and say in the year 2000, we want it there. So, yes, we're happy with all the assumptions. There we go. And then a shift equal to evaluate or shift enter. I'm sorry, to evaluate. And there we have it. 1.581 million people now.

Now, where is this information actually coming from? Like, is. Is this information do you have on your computer right now?

Because it's cached? Yes, but it, ultimately it comes from a database at Wolfram that is accessed over the Internet. It's the same Data set that WolframAlpha accesses. So you're, you're accessing the exact same data that is used in a WolframAlpha session visiting that website.

And so if anyone were to put in the commands that you put in here, would it give the same results or would they need to have cached them?

Okay, no, no, it, it caches automatically. So, so you don't have to really worry about.

Very neat.

You know, repeat evaluations will go very quickly.

Speaking of doing the code that you're seeing today, by the way, as always, the notebook, the completed notebook that you're looking at now in the form that it's in at the end of this presentation, is going to be uploaded after this video is done at some point. So if there's a time where you're not sure where the comma went in the code or something like that, you will get a chance to just take a look at it in its entirety later. So we, we may not go slowly enough to be able to, to go through every line of code today, but don't worry, you're going to get access to it.

Yes, absolutely. So, yeah, so like I said, we can do computations instead of just looking stuff up. You can do something like the population of Brisbane in the year 2000 divided by Brisbane's area. So we have 258.87 people per mile squared.

Neat. So you can do population density.

And you do unit conversions of that. So you can end up with it per km squared, which is more understandable by somebody in Brisbane.

Oh yeah, yeah, because it's what us, Liberia and Burma. Who, who does.

I believe. I believe so. I believe so. So. So yeah, so that's the easiest way to access an entity. If we take the input form of the orange blob for the entity Brisbane. You see this format here with the head entity, the type and the entities canonical name there. And that is leads into using entity values as a, you know, typing it in yourself instead of using control equal. The structure of an entity value call is basically the head entity value. The first argument is your entity such as Brisbane. The second argument is their property such as the city population. Fleshing it out more. An entity has a type and a canonical name. An entity property has a type and a canonical name. Now the type for your property will be the same as the type for your entity. You can't mix mountains and rivers, but your canonical names will of Course, be different. Optionally, you will have qualifiers for entity property calls. That's where we specify that the year is the year 2000. And we'll get to that in just a little bit. The more complicated calls. So there is, you know, an entity value call for Albert Einstein's date of birth. If we were entering that, realize they have the orange blobs there. Instead of, we can say entity, value, person. Oops, let's try any value entity, person, Albert einstein. And we could say birthday. And there we go.

I like how it's recommending what you might be.

It's very handy.

So what you just typed and the line above it are the exact same thing, just in different displays?

Yes, yes, that's basically it. You may note I'm using only the canonical name instead of the entity property wrapper. That's just a convenience that's left in there for you. So you can say entity property, person, And there's your more fleshed out form.

So what exactly is and isn't an entity?

Like what, what kinds of entities do we have? What areas do we cover?

Oh, sure. What, what, what qualifies something as being an entity?

Okay, well, here are all of our entity types. If you evaluate entity value with no arguments, you will get a list of all the available entity types.

There's quite a few.

There's quite a few. We've been hard at work. There are 353 different entity types that are exposed in the Wolfram language. Some of those are helpers for a broader type, such as for food, we have the food entity type. And then all of these different food facets, let's say, that are covered by other entity types that feed into the food entity type. The food entity type is very complex as you have to take into account. Is it raw, is it cooked, how much, how, how is it, how is it prepared, and so on. So yeah, there's quite a bit to go on there.

Yeah.

And it's growing. We, we are continually adding to our list of entity types. We're continually exploring new things that we should be adding and getting them in there. I'm sorry.

Oh, sorry. My connection must have.

Just a second.

We had a.

Okay, so. So, yeah, so look for that number to get bigger. Another little detour here. I mentioned property qualifiers. If we look at the city, population, property and it's with the date qualifier, we can see here that the raw version of it is entity property and the type city, the core property, population, and then a list of rules. And those rules are the various qualities Qualifiers for the property. In here we have the date going to the year 2000. You can list properties with entity properties. You can list the qualifiers for an entity property like so if you take the property itself and ask for qualifiers and spell it correctly, you get that. For example, the net worth qualifier gives us qualifiers, gives us the date option there.

Oh, okay. Yeah. Because the question of what someone's net worth is changes depending on when you're asking about.

Precisely. Yeah. So if we wanted, you know, Bill Gates Net Worth in 2020 looks like we have it for 2019 as the closest. December 6, 2019, pretty close. Probably a net worth published somewhere and we picked it up and that's yeah, 1 billion so on. Population has a lot of qualifiers. Yes.

We had a question coming in from the chat that looks interesting to me. Someone said a few minutes ago that they started using the entity framework in their structured query language SQL database. They're liking it is the WolframAlpha data in an SQL database and this entity framework is generating the SQL to query it. Which by the way, some details of that question are beyond my computational knowledge, but you might understand it better than I did.

Yes. And any value under the hood forms SQL queries and they go out to our database and, and, and query it and get the data back and then presents it through here. So yeah, it is a. An SQL backend that entity value is interacting with.

We have a follow up question. Is there plans to add vehicle data for various types of cars and trucks? I'm not familiar if something like that would already be in there or not, but we have a audience question who's curious about vehicle data?

Oh, well, we have, I don't see automobile data listed in there, but we do have automobile data available through the, through wolframalpha.com itself.

Cool.

So it might be, it could be for any number of reasons that it's not exposed in the language. It might just be that someone hasn't gotten to it yet. It might be something that simple. It could be data licensing issues or anything like that. So. So there we go.

Even though there's over 300 things listed here as the types of things we have data about, there is in fact data about a lot more that you can access.

Yes, very cool.

Very cool.

Yes. So where were we? City qualifiers. City population qualifiers. There are many of them because we incorporate data from the American Community Survey and that slices and dices population counts in many different ways. So if we ask for the city population qualifiers, we get all of these here. Age, citizen status, etcetera and if you wanted to know, well, fine. What, what can I put down for what races do we cover here? You can ask entity, property, city, population, qualifier values that will give you a list of rules where you have the qualifier on the left hand side and the right hand side is, are a list of values that you can use as qualifiers there. Okay, I will, I will confess that I do often have to go back and look at like do a control equal and say, you know, population in the year 2000 and see, remind myself, oh yeah, that's how, that's the syntax for qualifiers, you know. So even, even, even I, who have been working on ND value for as long as it's been around sometimes run into that.

That makes sense. So doing control equal is not just a good way to get an answer to your immediate question. It also shows you the kind of code you could use to ask other questions.

Yes, yes, exactly. Yeah, it's, it's a good code discovery tool as well as just a entity and discovery tool in that way. So something else fun to do with entity value is doing entity lookups with an implicit entity class. This takes the head entity class and a type. Here we're going for countries and a list of rules where it's a property goes to a value or a function that yields true or. Excuse me, or a range or such and that will give you a list of countries. Now it doesn't here, it just looks like it came back unevaluated. But if we ask for the entity list of that entity class, then we get all of the countries that we were talking about. Now you can take this and not do entity list and use this in a NC value call, which you'll see right here we have entity value and then that entity class definition and then we ask for the GDP and it gives us the GDP for each of those countries. This is way more efficient than mapping over that list of countries as this gets optimized into a single SQL call, a single database call. Whereas if I had mapped over this list of African countries, it would give, it would make a call for each individual country.

For people like myself who might not be familiar with some of the terminology, what does map over mean in this context?

So you can take a list, say these countries, right?

Understanding that this is an inefficient way to do what you just did.

So yeah, let's do it the wrong way. So you can say map and we've got a list there as the second argument to map. And for the first arguments, map, we will Say entity value. So what this will do is apply this function to every item in this list and return the resulting list. So there you have it. That was very fast because it's cached on my machine. But you would notice a bit of a speed difference between doing it this way and doing it the way I did up here with the entity class as the first argument.

And this has to do with the number of times that it's having to connect to database in order to do this. Right. In other words, the implicit entity classes and defining your own entity classes make it so that it will be more efficient when you go to actually gather the data.

Yes, very cool.

It doesn't have to be right now, but are you going to show us how to cache information on the machine?

There is, there is a way that you can pre cache information on the machine and yes, that's coming up.

Okay, very cool.

Yeah, So a little more just demonstrations here with an implicit entity classes you can have multiple criteria for it. So here we're asking for African countries and we want it to take the 10 smallest by population and there we have them there. Here similar thing, but we're taking the 10 largest by population and getting their GDP and returning it as an association. Reverse sorted. So we can see among the most populous countries in Africa, Nigeria has the largest gdp.

What does the term association here cause to happen.

That returns here? Let's, let's go down here.

And because I understood population and GPP and I see all that happening but.

I'm not quite sure, the plain call without association just returns the values.

Oh, so that's how you can see the name of the country.

Yes, here it associates each value with the country.

Very cool.

I believe you can also use data set in this too if you want a data set object.

Right, yeah, there you go. Now we're getting fancy. To answer your earlier question of getting.

Caching and things like this.

Yeah if you're going to say, you know, I'm going to look up thousands of city populations, you can do an entity prefetch on either the whole domain, say city or country or what have you, or just on a particular property within a domain. And what this will do is go and download a pre made cache of the data, unpack it and set it up and you will have all the data cached on your machine to perform computations on.

So I'm reading 18, almost 19 million values that you now have much faster access to.

Yeah, I am. Oh, you know what, when you take into account time series and stuff.

Okay. Because there's a lot of data about different years of everything. You could be asking about different.

And for American cities, you have all the slicing and dicing of the population where we have, you know, the number of Hispanics of Cuban origin over 10, you know, and stuff like that. So. So yeah, that probably accounts for why that's such a huge number. But yes, that's a lot of. That saves you a lot of database lookups. So there. Any questions before I move on?

So if you do have audience questions, remember to put them into the chat so that we can see them. I do have one question, actually, before we get to any stores, which is we were talking about sources earlier, so I believe there's a way to check the source of any given property, if I remember correctly. How would one do that? If you're wanting to see how the information is sourced.

In this day and age, that seems pretty important. You know, you can't have. You can't have the data source be. Just trust me and have a lot of people enjoy that.

Let's see here. Which we go with Movie.

Let me be clear. Wolfram's data is very, very trustworthy. And I know that that's why we call it curated data. But knowing where it comes from is pretty important.

And see, what am I looking at here? Geez. Entity properties of movie. I'm trying to remember how to do this here, actually, because.

Yeah, maybe autocomplete will help with a particular property or if you just pick one.

Yeah, let's see here. I will have to get back to you about that.

Makes sense.

I believe it's.

If we try source, we may add.

A note in the uploaded copy of this notebook.

Oh, there we go. It's just source, I think, and we just don't have the various MOVIE databases in there as a source. So I will talk to the Movie data developer and ask him about that. Yeah, so see.

Why is having different.

So sorry, John. I think we just caught apologies for.

The continued connection issue, but.

We'll find out about that.

Yep, we'll. We'll include some examples with code that calls it in the. In the version of this that gets uploaded.

Mm. So. Okay. Any others? No. All right.

Far so good.

Great. So let's talk about Entity Store. You kind of teased at the beginning about adding your own data and using it within entity value within the Wolfram data framework. Here is how we can do it. There are many entity stores out in the world that are available to play with. Little plug here for the Wolfram data repository, where you can find a bunch.

Small question. Does store in this case mean hold in reserve or mean place where you buy things?

It is. It is a store, like a. Like a warehouse for your data.

Cool. This is not, not an invitation for people to buy access to data.

No, no. These, the entity stores from the Wolfram data repository are free to use. They are free of cost. You know, like we have a store for census tracts, we have stores for FDIC institutions and so on.

Oh, looks like.

Yeah, Bob.

Messengers. There we go. Yeah.

Just quit that. Yeah, there we go. So, yes, one of the entity stores available on the data repository is one that's a favorite of mine for it's a little project of. A little project of love that I've had over my time at Wolfram, and that's the Shakespeare & Co. NC stores. Now, very quickly, Shakespeare & Co. Was a subscription lending library and bookstore in Paris between World War I and World War II. It was frequented by many famous writers that we know of. You know, Ernest Hemingway, Gertrude Stein, James Joyce and so on. Lots of the lost generation of American writers were there. So if we.

Get the. Were there as. As people who were.

Using it. Members of the library. Yes, yes. So what happened was the woman who ran the stores, Sylvia beach, left her papers to Princeton and Princeton University has digitized, is still working on digitizing. In fact, this is an ongoing project, digitizing all the records, hunting down who these people.

Were.

And. Yeah. Providing data for us to play with. And it's a very interesting insight into. You get to see what people were reading and so on. So let's make sure I don't have it registered. Now, to use an entity store, you have to register it first. That makes your Wolfram kernel aware that it's an available entity type. So we'll entity register those three entity stores. There's one for members, one for items, the books and magazines, and one for events. The events would be when somebody checked out and returned a book or bought a book. Those kinds of things. Not, you know, tea parties and such. Here we have our properties for members, properties for items, and properties for events. These.

Three. Yes. So, first of all, this is already very cool to me. I just am interested in the utility of all of this data from just the history of literature sort of perspective. But I'm a little unsure about in what sense is this using your own data here, because it looks an awful lot like the previous examples you were calling that are.

Wolfram's data. That's. That's very true. This is my own data in that Any user could make this dataset. And we'll get to how to make that in a. In a little bit here. Any user can make this kind of data and then access it the same way as you access the curated data.

From wolframaph. Oh, okay. So this is accessible to you not because it's on WolframAlpha, but because you yourself put this data into the.

Entity store. Right. And you need not be an employee of Wolfram to do that. You just have to have a Mathematica or Wolfram language interpreter.

At hand. And then if you want to upload something to the data repository, you can also do that next step so that it's available to.

Everyone. Correct? Yes, yes. You can make a very quickly, you go to the file menu and a new. Whoa. A new repository item and you pick data repository item and it pops up a notebook that you fill in. And at the end you can click this button to submit to the data repository. You can also click the deploy button to deploy it on your own.

Cloud account. And now that you have submitted this data, you know, before today, if other people ran the code that they see in this notebook, would they get the same results you are? Or is this something that's specific to you?

Because it's. No, they. They can. This is for.

Public use.

Oh, fantastic. Yeah, thanks. So. So yeah, let's get into things that we can do with it. Here we're asking for any event that featured James Joyce and that's using an implicit entity class. And here we see many events. Now the numbers are just IDs from the underlying data we have. The underlying data is a big JSON file that looks like this. And this is the members file. And you can see all the name and sort name and so on and so forth. It has great things like their addresses and coordinates and so on. That's the data that I got and put into the Wolfram data framework. So yeah.

We have. And so this is Joyce as an author or as a person who's buying or checking.

Out books. As a person buying and checking out books as a user of this library. So yeah, here's one of the events, the data association of it. You can see that here is James Joyce borrowing the Master Builder. The Master Builder we get asked for a non missing property association is something that you can do which is the same as basically as asking for an association and then delete missing. There you go. So let's see here. These events are of different types of. I want to get just purchases. I want to know what books James Joyce bought. And so we use multiple criteria for the implicit entity class. You see this, we can ask for any of those and let's store those to a variable called joistsales. And we can use entity value on the sales to ask for the start date and of. Of the sale, the date of the sale and the item from the sale. And we have to do that here. Now note there's a list here and a list here. This hearkens back to earlier on when I was talking about identity value, where it's more efficient to give it a list rather than map over a list. You can do the same with properties too. So you can list properties and entities in your entity value call. So there we go. Now a little wiggling of that makes it suitable for a timeline plot that we have here. And we can see all the different books that James Joyce bought on down to the, you know, between 1933 or so up.

Until.

3839. Yeah. Because the coordinates were present in there. We could do a geo histogram of where in Paris people lived while they were using the library. And you know, things like a date histogram of the membership years where you can see a big decline with the stock market crash of 29 and then. And the onset of World War I. Pretty or two, rather pretty effectively kills the library off with the Nazi occupation of parents.

So yeah. Just the thought of how many pages of written information someone would have to leaf through to begin to have all of this specific info at their.

Fingertips. It's Yay.

Computers, huh? Yeah, a.

Little bit. A.

Little bit. This is definitely, you know, the, the upshot of having the ability to deal with things at a high level rather than having to go and do things by hand.

Right.

So. Right. As you.

Said. Right. Computers. Yeah. So let's show people how to make their own so that we can. Well, they can, they can do so then. So an entity store. You give it the head entity store and you have your type. This can be whatever you like. Book in my library. And then the right hand side of that rule there in the entity store is an association within the association you will have an association of entities. You can also have an association of properties that. Here's the entity canonical name E1, the property canonical name P1 and its value and.

So on. So these are just examples. Other people might be writing.

Out full words, skeleton of, of what it's going to look.

Like. Yeah. In practice, one would want to generate this programmatically, not, you know, go and type through things, but as A as an example, just to show you how that looks like.

Yeah, right. That's right. So if we evaluated that, it would come back with this entity store blob with the entities and properties listed there, that there are no entity classes or property classes and so on. So one of my co workers, who unfortunately can't be with us today as he's ill, had. Michael Feltus, who's a fellow developer on WolframAlpha, has started a project to make an entity store of books in his library. And so he got himself a barcode scanner and he's scanning ISBN barcodes and feeding them into a text, putting them into a text file that he's reading in, in the Wolfram language and doing what we're about to see here. So we're going to use a service connect to Open Library. Now, Open Library is a kind of a. It's a crowdsourced database of books. So. And they have an API as well as bulk downloads of their data and Service Connect is the Wolfram language way of hooking up to an online API. So we open our connection there to the Open Library. And we can see here that if we give Open Library in ISBN, it will look up the book and give us all kinds of information on the book so we can. We don't have to go and look up. Oh, goodness. Where, where was that published? How many pages does it have? What was the published date, and so on. So here, this is just me decameling a string, figuring out how to do that real quick so that I could take things like bib key and publish country and.

Decamel them. I actually don't. I'm not familiar with.

The term. Oh. Oh. Camel case is where you squash together capitalized words as like how dynamic map is all one word here, but the capital letters are like the humps of a camel's back. So they call it.

Camel case. I noticed that the returned thing that you got when you made this, you know, second to last thing of asking for the normal version of that service object with, you know, some additional information in there, that's the same format that then you would go put to an entity store. So when we're talking about doing things programmatically, we are really going.

To just slot this right.

In.

There. Right? Yeah. Yeah. So, yeah, I cut you off. Was that what you were.

Getting at? Nope. Yep. And also, apologies, that appears to be some sort of like recurrent delay that we keep getting. So if I talk over anyone.

After.

That.

But.

Yep.

No. Yep. So far only the two or Three times that I've mentioned to John. Could we not understand you? You've been. You've been coming through.

Mostly clear.

Spectacular. Good. Okay, so here's all the properties that we were able to get from.

Open.

Library. Okay. Okay, so.

This is. This is not.

Wolfram's data. This is not Wolfram's data. This is data from Open Library. That's correct. This is stuff that we're going to be putting into the wire Wolfram Language session. And so we'll call this the M felt, this library in honor of, well, the owner. And here we have our Entities rule. I'm doing something crazy here. It seems what I'm doing is taking each entities list of properties and values and adding to it a label which is the title of that entity. This will make the labels on the orange blobs nicer instead of just like it being a camel case smushed together string or. Or, you know, an ISBN or what.

Have you. Oh, okay. Is that why there weren't many orange blobs in the output above? Just a couple. Because we hadn't.

Done this? No, this is actually because we process the data once it. So we. You make the call to Open Library. Open Library returns data. The Wolfram language takes that data and looks at it and says, oh, languages. Well, I'll turn those into language entities. And you can see there a published country. You can see that published place was New York. And it put the New York entity for the administrative division of New York in the United States in there. And same with the date. Turn the date into a.

Date.

Object. Cool. So that was kind of just automatically done for you by the service connect author. Yeah, we have a label for the entity store as a whole Michael's Library book. And then we have our properties where we're again doing the same thing. We're mapping, but as we did with entities where we're putting labels, we're mapping over that list of properties we made up here and decame them. Taking the decameled URL and putting it back into the form that we want and so on. Oops.

Oh, no. Stray space.

Syntax.

Stuff. Yeah. What did I do? Oh.

Right here. I like that. The red is letting us know where something is out of place as far as it's concerned. I'd probably be control Zing quite a bit at.

This point. Yeah, There we go. So, okay, a good example of it going.

Wrong sometimes. So here, command Z if you're in on.

A Mac. Yeah. Undo, undo, undo, undo. To the rescue. So here we have the NT Store object. I'm making sure I don't have it registered in there already. So I unregister it and then I'm going to NC register the symbol that holds that.

Entity store. So I, I take it if you're unregistering it because you want to show.

This working. But if.

Right, right. If, if, if anyone else was starting from scratch, they wouldn't have.

To start with, you wouldn't have to unregister it. And if you do register it twice, you just get a.

Shadowing error. Right. So to avoid.

That.

You. Right. Pretended we'd never done.

This before. Exactly. And you can see even with the shadowing error, it still works out just fine. So, so yeah, so here's the entities in there. An example of the properties.

There includes. Hang on a second. This is, this is Michael's.

Home.

Library. Correct. Okay. Very.

Very neat. Yeah. You know, we have the nice labels for the entities and so on. Now all kinds of post processing could be done here. We could say we maybe we want these to be URLs and so we know that. Okay, we want to post process that and so it ends up looking like so and so on. So yeah, there we go. Entity, Let's ask for the properties of the filters library there. Now we have a data set of a random one from there. Okay. Any, are there any questions going on.

At all? Well, I.

Have one.

Yes sir. Now I'm, I am a fan of the functionality you mentioned where you are uploading this data and then anyone can.

Explore.

It. Yes. To the extent that they want to. And I'm a big fan of the idea of that being something that is persistent. That data is important. So we should be continuing to store it, continuing to share it, continuing to use it. But I am curious if you were the one who added some data to the entity store. If you're the person who put it there, can you subsequently remove it or is it just there for the.

Long haul? Oh, sure you can. You can do all kinds of operations on entity stores. There is, let's see here, let's go to. The entity store documentation page and then from there and again, this is, I refer to this page. I find every other time I'm working with an entity store I have to refer to the documentation to, to figure out what.

I'm doing. I love our.

Reference materials. It's.

Just fantastic. Exactly. The language is a lot more intuitive than a lot of other languages I've learned. But it much more than that, the fact that we've written down how it works. Examples of extensions you can do with it, how it plays with other sorts of functionality I really like. I have never been excited by a manual before I started looking at the documentation.

For Wolfram. So we do have an audience question. So the audience question is, when you use entity register, is this for the kernel session only? Or in other words, does it automatically unregister when the kernel.

Session stops? Right. It's just for that kernel, that session. If you win it and come back to it, you'll have to re.

Register it. Okay, then what would your recommendation be for someone who wants to, you know, they go to all the trouble of getting the data together and then they are going to register this and then it's going to be unregistered in the kernel session. So is the data repository your preferred method? What would be some ways that people can, you know, have it.

Permanent deploy things to my own cloud account and then that way I have quick and easy access to it. There's a handful of things that I've put in the data repository and a bunch of things that are just deployed publicly to the cloud, where I skirt the data repository and instead just have it up on my own area of the.

Wolfram cloud. Is that what you'd recommend if you were, say, a researcher who was looking to make the data publicly accessible to other people, or would.

That be. That is indeed one of the ways that you could do that. Yeah, yeah, that, that you. The URL for that website is totally shareable and the data is all right there and gettable as a, as a cloud object. So when you call resource data my data set, resource data will go download the data, do its thing, and then you'll. By do its thing, I mean interpret it and bring it into the Wolfram language session and then, and then you can entity register it and be on.

Your way.

Very cool. So just to make sure that I've got the steps down as well as the audience has the steps down. If you're wanting to create a new entity store, you would create the store and then you would put it on the cloud. And then anytime you start a new kernel session, you would just register the thing that you can then pull down from the cloud. Is that your, your.

Basic recommendation? Yes. Or you can use put. Good old put and store it to disk. If, if you don't want it out publicly on the cloud, you want it just on your disk, you can deploy. That data repository submission notebook that I showed earlier has an option for deploying to your own.

Computer. Cool. I take it, we'll. We'll need to show the, the cloud step that you're talking about some.

Other day. Yeah, it's set up to.

Do.

That. Gotcha. Although it is generally a one liner if you don't want to do anything fancy. If you want to like do fancy things with permissions and stuff. Perhaps it's slightly more than a line, but I think that that's not.

Too bad. Right. I'm talking about making a resource object so you have a web page that displays the data and shows it and that you can then share.

With.

People. Gotcha. Yeah. So. Yes, you can change the entity store here like they have as an example in our workflow for setting up an entity store. They do, you know, they add entity classes, but it looks like, yeah, you need to change. If you need to change an entity store, you unregister the old store and register the new one. In its place you can add computed properties. Oh, here we go. Here's the section I was looking for. Changing values. So let's see here. Oh, it's just as simple as an assignment. So there. If you wanted to change a value on the fly, you can. Let's get down here. Let's say we wanted this.

Complete Faulty Towers, which is a great show if anyone. Oh, and how, my goodness, John Cleese at. I won't say his best because, you know, infinity times two equals infinity. Now most of it is his best, that most of his work is his best. But yeah, it's a heck of a. Heck of.

A show. Let's say we wanted to change this to, To include John Cleese a little bit at least. And then when we look.

Here and Connie Booth, who also does a great job in.

The show. There we go. Yes. So there I changed a value on the fly there we had. The authors used to be just Connie Booth. Now we've added John Cleese. And then when I asked for the association of the complete false towers, we can see John Cleese is.

In there. Now that. Is that a change that you've just made for yourself or when Michael is not next looking at this, will Michael also see this.

New association? No, Michael will not.

See this. Great. So you, so you're using Michael's data, but you are categorizing it just.

For.

Yourself. Right. And then if you wanted to create a different object from that, you know, post processed or whatever you wanted to do with it, you could create a separate one if you.

Wanted.

To. Right. Well, that it's, it's good to hear that if someone else is making their data available, the things other people are doing with that data will not then be able to just delete the data from the publicly.

Shareable version. Right, right. Yeah, it's. You're. You're allowed some data integrity there. That is true. Here we go. Any other questions.

At all? Thank you for a very useful session. Thank you for thinking it's a useful session. I very much agree. This is one part of the functionality that a lot of people don't necessarily realize is so easy to work with. This is really great to have you here. If there's any other questions, please do put them in the chat so that we can get them answered. Also, if you haven't checked out the blog recently, if you head over to the Wolfram blog, we have some cool stuff.

Up.

About the ChatGPT getting its Wolfram superpowers. So if you have any other questions about this, this is also something that Aaron and Michael both did have involvement in. And so any audience questions about this, we might be able to give you some interesting tips, tricks, you know, whatever you're curious about, feel free to put them in.

The chat. I will say that I did explore a little bit using ChatGPT in an entity store where I tried to entity register the Shakespeare and company entity stores. And I got, I took a screenshot here of my response. Use the Wolfram plugin, which is limitedly available but pretty fun and well, you know, jump on it as soon as you're.

Able to. If anyone hasn't heard about the Wolfram plugin because it was just released at noon yesterday, GPT plus, I think at this point is including access to other IT companies, apps and information here and there. And Wolfram is one of those companies to extend itself beyond its learned data. So it won't just be going from the training that it has, it'll also be occasionally, if you allow it to querying some other sort of system that is very good at something, to be able to fill in any sort of synergistic stuff in there. I, in the last five words of that I felt like I was getting very buzzwordy, but you get what.

I'm saying? That's a very.

Buzzwordy thing. Yeah, that there are things that ChatGPT and GPT4 and GPT plus are better at than anything I have ever seen and that speaking is someone who works for a company that's very proud of our natural language processing. Seeing its ability to, to create natural language and to understand natural language queries is amazing and Combining that with Wolfram's ability to do quantitative calculations, to have quantitative data. It's. It's.

Very exciting. It is one of the things that again, check out the blog posts, if you haven't seen them yet. One of the things that this post talks about is that there's been a long history of having sort of two branches, at least two branches in the road that you can take to trying to generate, you know, AI like systems. And one of them is symbolic, one of them is probabilistic. And so having the ability to combine the best of both worlds and things like these plugins is.

Pretty cool. There's a lot of promise. There's a lot of promise here in this, bringing those two approaches to.

AI.

Together. Mm. So it's.

Pretty exciting. Not to mention just being able to reach outside of the 2021 cutoff that the training data has right, for, for the GPT system that we're able to query Wolfram, which has up to date data on a lot.

Of information. So, Aaron, you were also, you were mentioning that you asked for the, The Shakespeare & Co. Entity store and it looks like it recognized that this was a thing that existed that you could go.

Yeah, it did. It didn't actually register it and use it, but it gave me the code for.

How to.

And. Very cool. And you know, it tells me here that I have to re place the member ID and property with them. Member ID and a property in the code there. And, you know, knew enough about that and properties that gives a couple of sample properties that we can go from that we can use.

And so.

On. That's great. Oh, sorry.

John, go ahead. I was just gonna say one, one exciting, you know, possibility that, you know, being able to have text generation combined with symbolic capabilities brings, is that you can sort of have an additional way of learning how to do things. Right. So you can ask for how would I do this? In Wolfram language, for example. And right then with, you know, having a skin like this available, the results that I've seen have been pretty good. It's like, you know, the, the things that you can, you know, documentation is a really good way to learn how to do things. But occasionally there's something where it's like, I don't want to take the step of looking through documentation to find what the right part is. And so being able to just ask, hey, you know, what's a way I could do.

This? Neat functionality. Yep, yep, yep.

This it is. I had a question about the entity store and using your own data sort of Thing you showed us the example from the Shakespeare and Company, what the original data looked like, the digitized version of that. But I was wondering, is there some specific kind of data that this has to be before you start importing it or putting it in the. Like. Does it have to be a CSV? Does it have.

To be a. No, it can be any format. You know, you have spreadsheets or a relational database or, or, or, or CSV files like you said, or, or JSON files or any sort of thing, as long as you can import it into the Wolfram language. Which.

Let's see. Are. Yeah. Is there a. Is. Is there a defined set of things that are or.

Aren'T.

Importable there? You. That's quite a.

Few. So, yeah. So yes. So you can import any of these file types here and, and that can do.

It for you. I recognize this is a weird question to be asking because we all work for Wolfram, but is there any notable gaps that we're working on there? Like, is there anything that you. That, you know, we're. We're working to add formats that we don't currently have, or is it generally that most of the formats that people use for data are.

Ones that we're. No, the import export team exists and is still working. They are, you know, new file types come out. They, they have to adapt to that as well as, you know, types being revised or, you know, maybe you can remember some years ago when Word radically changed the DOC format and became Docx and so on. You know, we have to adjust to those.

Kinds of things. And so we're, we're up to date. But what is up to date changes.

Every day, so. Exactly, exactly. And there's always, you know, something out there that we just haven't considered yet. And when we find it, we, you know, consider it.

And add it. Yeah. Yes. Speaking actually of that, that last thing. So at the American Physical Society conference, I think it's two weeks ago now, we had someone who works in molecular dynamics. And so there's this particular file type. I see that I could convert to an existing file type, but I'm curious if there's plans to add this other one. And so the answer to this as well, please. If you're someone who has a particular file type that's really important to your work and you want to see it in Wolfram language, feel free to reach out and let us know this just so that we can better prioritize which things.

People find important.

Right, Absolutely, definitely. If, yeah, if you're a user and you're like, oh golly, this particular file type I want isn't in this long list. Do please reach out and let us know so that we can get that included.

In the list. Yep. Yeah. User feedback is very important. You learn. So. Yeah. All right. Anything.

Else there? No. It looks like the people in the chat have gotten their questions answered. I see something that I'm not sure if it is a question or a statement saying. Seeing the SQL's queries for WolframAlpha databases would be so useful to understand some of the data structure and confirm the understanding of the.

Output it generated. I don't. There's no like debugging, you know, internally. Yeah, we have tracers and, and stuff that for, for our debugging purposes. But I don't think there's any way to see the underlying.

SQL query from. It's not.

Outward facing. Gotcha. As far as I'm aware, it's.

Not. Yeah, yeah. But hopefully all of this has given a bit more of an insight into the way that data is stored, accessed.

And.

Interacted with. Yeah. In this system related to the import format. So when you're importing things you can also specify how you would like it to be imported in many cases. So like if you want to import something as a data set or import something as like a particular thing that you have a lot of experience with doing the processing steps to get your data, you know, into the shape that you want it in before your entity store, you know, remember that you can.

You can specify. The last thing we heard John was remember that you can specify. And then we. We've.

Lost.

That cutoff. Yeah. I'm hoping that he can still.

Hear us, But.

Oh well, no, the dreaded communication cut off seems to have occurred. But I, I just want to say this is something that I am glad to have learned as much about in this presentation as I did. So thank you Aaron, very much for all the work you did. I know that Michael Felt is who you mentioned, also did a lot of work on this presentation. I want to thank him right now. I know he's too. He's homesick disposed today. But. But yes.

He definitely was. Yeah. Michael, if you're watching.

Hello. Thank you. And we have a comment from the chat, from a viewer just saying thank you gentlemen, this was a very interesting talk and thank you all for thanks attending. Aaron, is there anything else that you wanted to let.

People know today? No, just gather up data and things you're interested in. Make entity stores or data sets or what have you with it and submit it to the data repository because that's a growing project and that's a fun place to showcase your data and show off.

What you've done. John, it looks like you're back with us. Do you want to complete the thing you were saying or did.

It get addressed? No, I think we got the most of that part. But the. The point about uploading things to the data repository, I mean, this is a really great way to, you know, build this notion of having open science and open data, which is growing increasingly important and.

Relevant, I think. Yes, yeah.

Very much so. You.

Know, this is. It was one of the things we had in mind when we, when we started the data repository was in addition to just being a place for casual, fun things like the Shakespeare and Company Entity Store, it also could be placed for serious scholarship to be done and presented and archived and.

Made available. So. Yeah, I mean, I, I see scholarship and what you're showing us.

Today, I mean. Oh, yeah, absolutely. Me too. My undergrad's in English. That's why this is a near. A project that's.

Dear to me. Right. Just. Just knowing what a famous author, who is. Was a genius, just a trailblazer in their field was reading right before they were writing some.

Of.

Their works. Right. What they were studying, what, what they encountered feels. It's.

Amazing to me. And Aaron, you also mentioned when we were speaking earlier, an interesting project idea to sort of try to trace the influence of a particular author. So you're talking about Gertrude Stein. Like, how would you computationally think about. Okay, you know, I haven't thought. Make a network graph of who was checking.

Out these books. And that's exactly it. We made a graph of people and the books that they had checked out. And I need to incorporate dates in there and I need to figure out who checked out. Let's say Gertrude Stein checked out. A portrait of the artist as a young man who checked it out after her is.

What.

I'm wondering. Huh. And is there any consistency, any pattern that we can find in, like, Gertrude Stein checks out a book and these four people seem to.

Check out things after her, suggesting that.

These people were recommending in her salon and she was saying, yeah, so try Joyce's Portrait of the Artist. It's fantastic. And so on.

Yeah, yeah, so. And using things like entity functions, one could try to, like, compute, you know, probabilities of, you know, given that one person checked out a book, what's the probability that, you know, another person will check out a book later? I mean, all kinds of Interesting possibilities for what you could do with.

Data like this. Yes, yes, I see. Oh, he checked out the Joyce or bought the.

Joyce book. Interesting. Very cool. Just want to say.

Thank you again. Thank you for the opportunity. It was fun.

To share this. We have another audience question here at the 11th hour. Let's say that you're creating an entity store. Is there some way to version your entity stores, anticipating subsequent updates? What would you recommend for someone who's wanting to, you know, have. Have a data set that's going to get updated versions as.

They go along? That is a very good question. We have versioning on the data repository. I've done two iterations so far of the Shakespeare and company and it's needing to be updated again. So there's another iteration in the works. So we do do versioning there. As for versioning for yourself, I am not sure. I would have to ask. I'll ask Bob, who develops these things as the head of that team, if that's in there or something that he should consider to add. But a very good.

And important question. One would think that the just the name of the entity store element might be one way to do that. Just what you were.

Calling your data could be, you know, you could always say copy one, copy.

Two, copy three. Right. But then it starts getting into the whole like version, final version.

Final part two. Right. Yeah, I was there.

Too in college. A version actually. Final. Not.

Kidding this time. Yeah, yep, yep, yep. I remember that. So, yeah, unfortunately, I don't believe there is a good way of versioning your own data within an entity store. You could of course dump it to disk and use git or the versioning system of your choice and do it that way. But as far as built into entity store, I don't believe there.

Is versioning info. Yeah, this reminds me of another question which I think is related, which is that let's say I want to add qualifiers to the properties that you're adding to your entity store. So like, if you have. Oh yeah, and things like this. I don't remember if we showed the syntax for how to do that. If so, if we didn't, how would.

You do that? Well, you would treat it just like any other property. And you know, it's. You would have the. The property in a list with the qualifiers to it in there. And I believe it would just work that way. So you would have, Let's say you had population and then a list of qualifiers, you know, date goes to and then and so on. So when you add the data that is one of your properties and a value there, and you can keep on adding things incrementally that way. That's my vague and wavy answer to that question, unfortunately. But yeah, I have not played with attributed properties in an.

Entity store yet. I, I have a question for you because I've looked at the data repository and often been very impressed with what I ran into, but also gotten the feeling as I was exploring data that there, there were elements of the data that I was missing. Not that was missing from the repository, but that was in the data and I didn't know about. So as someone who works with this sort of data a lot, I wanted to ask what's a first step you might do to see what categories of data are in the data that you're looking? Yeah, when data someone.

Else uploaded, I would, I would look at the properties first using entity properties and then the type and see, that gives, that tells us. Oh, okay, we have all of these things. Ooh, a VIF ID, huh? Wiki data IDs for people and Wikipedia, you know, looking at what we've got there. And then similarly with the entities. Now, oftentimes entities, they maybe hundreds or thousands of them. So maybe you just want to take a random sample of a.

Hundred.

Of them. Right. If you wanted types.

Of entities there to know all the information that we had about, say, New York, there was.

Quite a bit. Right, right. So, yeah, exploring the entities and properties that way, what do we do? And then you can, you know, you can do plots and stuff and see what the range of your data is and see that it follows a distribution at.

All or so. And if these, if these properties, like say. Item in the, you know, the, the particular book, if those have subcategories of data themselves, is that something that shows up in this entity property call or, or.

Am I misunderstanding? Try that and get back. No, I, I think I understand what you're saying. You're saying that when I asked for entity properties, if end date here had qualifiers, would they show up in this list? And the answer is for typically an entity value, the answer is no. You would have to do, bring up the property, let's say end date here and ask for qualifiers and you'll see.

That there are. In this case, there isn't. It's just right. Every end date is just an end date. There aren't special kinds.

Of.

End dates. Right. And if you map that request for qualifiers over the whole list, that would be one way of seeing, you know, what are the qualifiers that exist for all of them. And in this particular entity. Sorry, I don't think any of them have qualifiers if you're doing it for either your own or a different.

Type of entity. But this is helpful information to me, I'm sure. I'm going to refer to the end of this video because, yeah, I often have taken a look at data and just seen there's so much here. I really need a sense quickly, of what sort of category is we've got in this case, it looks like there weren't just, like John said, nothing.

No deeper subdivisions. Nothing.

Real.

Exotic here. Right. But there another thing you will see sometimes, and I believe there's no reason why you can't do this in your own entity stores as well. You'll definitely see it in the ones that are built into the language. But, you know, if you ask for an entity property that can sometimes point to a different entity, that then will have its own things. And so, you know, in this way you can sort of chain.

These things together. Right, right. So first, the members. I have Wolfram language entities that I have gone and found wherever possible. So if we brought up, you know, our friend James Joyce there. Not to pick on Joyce, but it's just convenient. Convenient. We were. Oh, yeah. Wolfram language entity. Is that what it's called? Yes, Right there we. And here we can.

Say data set. So this is a different data set than the data set that is in the.

Shakespeare and Company. Right. Here we are. This is data set from the curated data at Wolfram. So.

Yeah, we have. And as you can see, there's an entity, you know, down toward the bottom of that. That list that came up in the data set. So you can definitely explore.

That.

Almost endlessly. Mm. But our time for this presentation is not entirely endless. So I'm checking the chat again and seeing that people seem pretty happy with the information we're giving them. John, was there anything else you wanted to ask today? I feel like I could talk about this for quite some time, but we should probably be letting Aaron go.

At some point. Yep, at some point. I think. I think so. For. I think for entity properties, there's sometimes a description property of the property. Am I remembering that correctly? Not necessarily in an entity store, because that.

That.

Takes some. Right. Some doing to get.

It in there, but, yes.

I believe so. Is there one that comes to mind that briefly we could just show how that works so that people can see how.

That'S in there? Oh, let's go to the docs. Let's see. We can ask for. There we go. Yes, for the definition of.

Of a.

Property. Yeah, definition. And that.

Will give you. As you can see. Yeah, like if you're someone watching at home, as you can see, you know, just checking real quick, what was the syntax for that again? Can be super helpful. So.

Yeah, definition. Okay. So yeah, entity property movie, That's how we would do it. But obviously we have no definition for name.

It's the name. Yeah, that, that one's pretty self explanatory. But yeah, if there's a, a particular property that you see and the name doesn't make it obvious to you what it is, try that definition thing and.

That.

Should help. Right, and just as we're ending here, I think you did a great job of explaining it, but I often need just a summary at the end of learning about a new thing. So the distinction between Entity Value and Entity Store, how would you summarize what the two sorts of those two functions are doing?

What are they? Entity Value is a way of looking up data. Entity Store is how you can prepare data for use.

By Entity Value. And, and does Entity Store always allow anyone in the world to access the data that you've stored or is that something you have to do? Some special steps to.

You, you would have to put it up, upload it to your.

Cloud account or. Which again we'll show some other day how to do. Yeah, sorry to talk.

Over you there.

That's all.

Right.

Very cool. But so great summary if we don't have any more audience questions, which it doesn't look like we do. Thank you very much, Jaron again for joining us. And we're sorry that Michael was sick today because that would have been also fun to talk to Michael. But if you're.

At home. Hello and everyone who joined today, thank you again for joining us. The notebook will be uploaded to the permanent version of this video subsequently and you'll be able to explore it at your leisure or just start exploring with your own data and see where.

It takes you. Well, thank you very much for the.

Opportunity here, guys. Thank you to talk with you, Aaron. Take care and thank you everyone for joining us.