Live CEOing Ep 805: Language Design in the Wolfram Language [Engine Connectivity Engineering]

Okay. Hi everyone. Welcome to another Wolfram language design review. And we're looking at a variety of topics related to connectivity with the outside world. So first, IDE support.

Yes. So we had some leftover business there because we are almost ready to have the new version of the VS code extension. There was a discussion about the support for notebooks now notebooks in the way Visual Studio Code and Jupyter understand them. So these are entirely product specific and there was a question of whether we really want to have some file format that tries to mimic some of our notebook support but is really specific. Now we had a meeting with Ian from Front end and there was a really nice possibility to make this more seamless. So these notebooks will be specific to Visual Studio Code which is the market leader in all of this. IDE and others do that as well. But we can easily develop an importer into our notebook expressions because I mean this is just JSON and we can write a bunch of. Then the interesting thing is that once you have that you can add.

We can just import, we can just.

Open, we can open these notebooks in our front end. It's a one way street but that's fine.

Can't we do a save as?

No, because we would lose all the information. And I think the point, what we are trying to do is somebody developing code, doing some debugging and examples and then import it.

So hold on the statement they lose a bunch of information all. Is it the case that, I mean save as in plenty of situations save as will lose information. It'll just give you a warning saying you're going to lose information. Do you want to do this?

I mean we haven't looked at. But I mean it, it's, it's entirely doable. But I mean the books are much.

I understand they're much weaker.

So the question is does it make sense? Do we, do we want to support it? If we can. We.

I don't see why we shouldn't. I mean we're just. With a big warning message that says you're going to lose all the materials. You probably should use our notebooks instead. You know, don't, don't use this, this much weaker thing.

Yes, but I mean if that's. If. Yeah, then why not? I mean.

Yes, but I mean with, with an appropriate warning message dialogue. I think that's the right thing to do.

Yes. I mean it seems entirely doable. I mean it's both structured documents and.

What does Abandoned Full Notebook Render? What does that mean?

Well, there was a question of whether we want to have a full notebook support inside Visual Studio code. And it would be doable by duplicating the cloud front end. Inside Visual Studio code.

Why the hell would we do that?

Yes, yes. I mean, that is the conclusion from all these discussions. It would be an enormous amount and it's no point. I mean, we have a front end, right?

Yeah. Right. So who cares? Right?

Yes.

So if that's fine, then it's only the question of the notebook of the file extension. I think it should be. Should make it clear that this is not our notebooks, but it's Visual Studio specific. Visual Studio Wolfram notebook.

I understand why we need the W in there. We have the note with the NB extension. If somebody else has tried to put their version of it in there, so be it. But I think VSNB seems like a.

So Suman may have. I mean, he did research on all the existing no performance. So is there a conflict?

No.

If we call it ws.

Okay.

Via San B, is that fine?

Okay. Yeah, there is no conflict. Conflict.

Okay, very good. So.

Okay, let's move on to the next item.

Okay. Sorry, that was a double click instead of a single click. So there is an old project called Wolfram System Extension Points. Now, the project's scope is to document all ways in which our system can be extended. And much to our surprise, this was actually done. And there is a very nice. If I'm trying to share that Wolfmus document that collects all of these at the time of writing about a year ago, all of these extension points, like you can add new import export formats, you can add your own random number generators, wavelengths. This was Bradley, right? You did.

No, I believe it was Carlos.

It was Carlos Antoni did this.

Okay. Yes. So this is a very nice document. So it has links to the existing documentation which describes how to do all of this.

Can you scroll down please and see what. What's there?

So these are things which are internal, which are not that well documented and we could think are in parallel. Why not? Information is something that we might make available to everybody. And Jesse may have a way to provide information about adding your own remote batch computation provider stuff, stuff like that. And then there's obviously some internal stuff, like internal context System services is something we're still working on.

So.

This is a good list. But let me go back to my notes. It will get outdated if it's not maintained well.

Okay. The point of making this list was to understand what things packlets should be able to have in them. And that was the number one Thing.

Yeah, I mean it's not just packlets, right. I mean it's for example, style sheets in front end and stuff like that.

What Stephen means there, Roman, and that was one of the points of this list was that these are all things that could potentially be sort of supported by the packlets.

So you load the packlet.

Could you go back to Solis for a second, please? Okay, so import export. So which of these are now supported in packlets?

There's been nothing new has been added in this list for a while. Like there's no import export support in packlist. You can't declare, you know, an import converter in a packlet yet.

Do we want that?

I think that would be useful.

But alternatively it's just like there is a function that does it and as part of packlet initialization you could run that function.

Exactly.

Style sheets can be added packlets. Right.

So can we notate this please by what can be added and what cannot.

So I mean, there is actually some meta information there, how to extend it, so called import expert. So here you have to perform an initialization. There's an argumentation supported by Packer Manager. I think that's one. No, but yes, I think that's the line that we want to work on to make this more.

Well, I mean, we have to prioritize these. Is it. Does it make sense? Okay, so I mean, here I would.

Assume that if you have a packlet that performs, that provides certain import export functionality, then just calling that register import as part of the initialization, which seems perfect.

Sure. But the question is, you know, many packlets have their own, you know, foobar import type thing, you know, special functions for importing. Whereas what this would allow one to do is to knit it into existing import export. But it doesn't completely succeed because it's not going to be on the documentation pages. It's not, it's not completely obvious. It's what people want. It could be that for a specialized packlet, one is better off having a custom import function.

Okay, yeah, that's true.

Could you go back to the one.

Thing this does let us do though is you type import, you know, block comma and then some unknown format and the packlet manager then says, okay, does any packlet claim to supply.

Oh, sure. No, I agree, I agree. Now another question. Let's see. So yeah, that's the thing like, like go search packlets, you know, if you say import of some weird file extension, does it go search imports for that? Okay, could you close up that thing and Let me see the rest of the list. Okay. Random number generators. I just don't see a call for wavelets. I don't see a call for. I mean again, what are stream methods?

See what this does?

Pretty low level. You can define a custom stream and you know, supply the read write methods and such.

I don't see this being that important. Close that up again, please. Ok, here's something I don't think is on here is auto completion extensions. See what I'm saying?

Yes. So LSP server type things.

Yes. I mean, you know, I've added a new I, I've added some function, I've added some string value for something kind of, you know, set up autocomplete for that.

Is the. Currently the only way to do this is through the documentation. Right.

I don't know where you fill in.

The table of possible.

I think this list looks nice, but I don't think this list is complete.

No, I mean as I said, I mean it. If it's not maintained then it will, it will last.

Go back to Atlas, please. Go back. Okay, go back there.

It doesn't have any compiler stuff on it. It's definitely incomplete.

New Arduino type boards can be added in device framework.

Okay, I mean look, there's a bunch of things missing here. Okay, so let me talk about another thing the. Remember I had that thing. Well, so for example, right now I have some GPU code, external GPU code. I want to be able to package it up and have it run on any platform. So put it in a WFR like place. I mean that's the type of thing a packlet should be able to include. Something like that. Am I making sense? Yeah, and that seems to me more important than many of the things that we have on this list. I just don't think.

Can I suggest that we. I mean the external evaluate discussion of Photon will take some time and it's very important we do it today.

So. Right, let's, let's, let's.

I don't know why this thing is probably bought on weekend.

Okay, Ricardo, go ahead, share please.

Oh yeah, thank you.

Yeah.

Okay, so where we are basically we are trying to design external evaluate objects and how they work with sub values and you know, functionality, everything that you can do with them. So let's start by starting a session and work with a pyron module that is about manipulating IP address and network. So what we are doing and experimenting with is that you call a subvalue and sub value just mean attribute extraction, attribute lookup and so for example, in Python, if you do address version, it will be equivalent of doing dot version. So you will see in all this code there are comments, and into the comment there is like the equivalent Python code that you're running. So we have been importing IP address and when we are doing version we are calling.

Okay, and that's the default. But so now what is it? Okay, show me this.

Okay, now we can go with casting. So you can put another operation. If the operation is the symbol, external object or string we are doing, or expression, we are doing the gusting to, to that type. So for example, we, we can add for version, this is just a number. But you can ask to. To have it as an external object or you can ask to have it as a string.

Let me, let me, hold on. Type address aversion comma real or address aversion comma integer.

No, no, no, it doesn't work that way. The type needs to be defined into the external evaluation system and the system.

Show me what it does. In that case, just show me what it does. Show me what it does. I know you don't know what it.

Does, but it's gonna break or agree.

I understand, because unfortunately. Okay, all right, so fine, so what should it do here? Well, first thing it should do. What's that?

No, no, go ahead.

I mean, where's the discovery interface for this? How do I find out what types of.

Okay, well, there was an idea of putting things under information. And so under information we can ask for different things, including operation. And one could be return types could be done with information. So basically you will do information of the object and then I guess button types.

Yeah, well, presumably it will offer that as part of the association that is formatted for information.

Yes. So one possibility is this one.

I mean, there aren't that many external systems that support the external object notation. And I think expression and external object are by far the most important ones that are universally supported.

Okay.

Expression will give you an integer VLF depending on what it is.

I mean, the default is expression. Yeah, the default is expression and it falls back to external object when we don't know how to translate it. But yeah, go ahead.

One thing to come out here is that there is an automatic system here, Stephen, that by default decides which type is appropriate. So usually these things are only when for some reason you want to divert that.

I understand. Okay. So on the face of it, it seems reasonable. So last time we had talked about the sub value thing and I think we had concluded. So. So scrolling up a little bit Please. Right. Actually, why don't you. Why don't you go on, scroll down. I mean, so far I'm. The casting thing looks okay. Although I have some questions about it. Yes, come back to it.

Yes, I know. Okay, let's do it real quick. And then, then you.

The heck is that?

What's that?

What is that? What?

What?

You just did that.

Oh, okay. So basically string is a. Is a. Is a magic method in Python or all stuff with double underscore magic pattern in pattern that they are like method that all object have. So every object defines a string method. Okay. And so what I'm showing here is that if you do version string, basically you are asking to get, you know, dot version dot string. So this returns a function, because this is a function and you're getting it as an external function. So then you can simply call it even if this is not the only way to do this.

I'm sorry, I'm confused. You can't have both the thing above and this thing here. In other words, you can't have both the second argument to the sub value be sub casing. I mean, in other words, to what extent? I mean, in general, there could be many nested attributes, correct? Yes, nested attributes are a thing. So let's talk about if.

Why don't you see the field picture? Because. Because it will become clear. Why is this thing. This thing is a shortcut that I will explain. Okay. Multiple object extraction. So you slap a list in the first argument and basically this is what it means. It means that you got the object up to here. Now you're asking for different things.

Yep.

I guess now you can put external object as well.

Yeah. And you get two external object, both object.

Perfect. Now if you put another list.

I don't like it. You can understand it, but I don't like it. I. I don't like it. Okay, so. Yeah.

What's that name? Name. The. This is the named version. So you boot an association.

I understand.

Same stuff.

I understand, I understand. Okay, keep going.

Okay, now you can. Okay, how you do your methods. So you create an object, comes back as an external object, and you can use external.

This is now a different branch. We're looking at a different thing. Right, so. So I should comment on the property extraction stuff first. Is that correct? Or is the thing that you're showing me now dependent on?

No, then you, you can.

All right, so look, here's my comment. Scroll up, please. Okay, I. Okay, I'm assuming that in this multiple attribute. No, not multiple. The one above. Nested attribute extraction. Right. You could go A, B, C, D. Correct. Can be any depth. Right. Okay, so I've got two questions here. So first of all, my. Okay, is it the case that you can think of those expressions, as opposed to strings that appear in the argument list, in the subvalue argument list, as being the equivalent of sort of a pseudo, just like the underscore. Underscore. STR is a pseudo property that you could think of expression string being there as a pseudo property. Does that make sense?

Not sure what you mean exactly.

Okay, so what I mean is this. What you can't do is have the second argument of the sub value sometimes mean nested attribute extraction and sometimes mean typecasting. The only way you can get away with doing that is.

I'm getting away with it and it's later.

Yeah, I think the idea is that this.

We need to go after by explaining external operation. And this is what I wanted to do. Introduce external operation first and then go with this stuff. But basically you need to understand what is an external operation and why it works this way.

Yeah, okay. Okay. Yes, okay.

But. Okay, before we go into external operation, there is a simple way that you can call a method with. By using external functions. So you put the object as a first argument to external function. Second argument is the attribute and then you can just call it. So, for example, if the object itself is. Or on the other end, if the object itself is a function, you can directly use external function. But basically this is calling.

I'm totally confused. I'm totally confused. Let's start from the beginning.

Let me quickly explain from the other side.

Okay.

Just a single sentence. We want to make sure that method calls for languages which support methods will be done within the same interface of external function. If people are prefer to think of everything as a function, not as a method. And then we unified by allowing people to call external function with two arguments. First, the object. Second, the name of the method. We say it's basically still a function. It's just the function that is named.

This way and there aren't.

And in this way you can call a method.

The method is just dot blah or whatever it is just.

Yes, if you.

Yes, but what do you mean? What's the. In this case, is there an alternative? Is there a thing.

Yes, if you want. If you want a Nasta chained method call. We have a more general syntax for that. But that does not involve external function as the main calling.

Not sure that's the right thing. I mean, if they. If you can do the chain you should be able to do the whole chain, not just the single, you know, the single method case.

Right. It's more, it's more the.

Maybe. Yes. I mean, maybe. I was thinking that, that here you're going to specify multiple string, but this is really more syntax sugar on the generic case that we see.

The generic case.

Okay, so let's start introducing external operation. External operation is what is basically, you know, we are exposing basically the actual operation that you can do on the evaluator. So one operation, for example, is evaluating a string of code that. That operation is called eval. So when you, you write it like this, two plus two. But in theory you could also write this, this is more explicit. Basically you are saying, you know, that I want to.

In our evaluator is a node. Correct, Correct. Okay, but it is.

This is just a tree, like a Wolfram language expression, more or less the way you need to think about it. And so, for example, let's say that you want to write rep of two, which is, you know, the function rep, which is transforming a string in Python. And so you do eval of rep, that will give you the function and then you call this guy with the number two. Yeah. This is how you will write this. Okay, now if you want to write something more complex like this one, it's going to be slightly more complicated, but still it's going to be something like, you know, you need to write. First you need to have the map operation, then you need to have the rep, then you need to call range of them, et cetera, et cetera. And so this is how you would write this piece of code explicitly, by nesting operation. It's something that you probably don't want to do very often. There is an application, there is a useful operation like import, for example, that we want to expose that basically allow you to import a module, for example.

So what are the possible first arguments of external operation that will vary with the external evaluator?

Exactly, yes.

So, for example, Python. What's the list? Is it a long list?

Not so much. It's going to be called eval, get attribute, set, attribute, things like that.

This looks like a reasonable design. Yep. Go ahead, Stephen.

There's an application at the bottom, I think. I'm not sure. Did you add that, Ricardo, for like a neural network?

Yeah, but I'm not sure.

Scroll back up, scroll back up, please, please. No, no, down, down, down, down to what you were showing before. Down to what you were just showing. Right, right. Okay, okay. So what I want to know is that next thing okay, so I understand the external operation import, blah blah, blah, what's the three argument? External operation.

Okay, No, I mean every operation is bararg in general. In this case this is just, you know, import IP address and then get the object IP address.

So this is that. What is the translation between the verags thing, arg1, arg2, arg3. What's the translation of that to thing dot, thing dot, whatever it is, is it uniform or is it. I don't understand it.

So the dot dot dot is going to be the operation get attributed. The operation get attribute takes, takes an object and an attribute name. So it's going to be.

No, I don't understand that. Yeah, you've got the three arguments there. Okay, what is the. Oh, I see you've got the translation underneath.

So for example, get out.

Wait a second. Okay, I don't understand the grouping. So in n31 the comment refers to the thing below it, correct? Yes.

Okay, those two things are equivalent. Those two things are effectively equivalent because this is something where basically in already existing syntax you can do for example rep of 2 and this means function call. So this is going to be equivalent of calling rep with the number two. And so basically what I'm doing is that I'm calling IP address with this string and this is a more explicit way of writing it where basically I'm saying import.

Yeah, I get it, but I'm still asking about the three argument version. I'm still asking about the three argument version. What does that mean?

Okay, so it means import the module IP address, get the attribute.

Okay, now what if there are nested attributes? So what if there are two arguments? What if the import there has.

Time you, you, you really need to. To only get one level.

So what happens if that function has two arguments? What happens if the import function had two arguments?

It doesn't. So no, because this is an operation that mimics the import is not directly like calling import or whatever. This is like one operation that we are designing.

In the language food. Okay, how do we guarantee that operations are monadic only have one argument?

We don't.

Okay, but then, then how does this work? Because this has a heterogeneous thing. We've got. First we've got the operation, then the next argument is an argument of that operation and then the next argument here is a dotting an attribute property.

No, no, no. I mean in general the operation is like perform language. It is, it's var. You know, you can have an operation that is image identified. It takes one argument or you can have another operation, takes two or three or more argument. It depends on the operation.

I understand. Okay, could you just write out external. Okay, just write out. Just type on a separate line. Type on a separate line external operation of O or op or something.

I think I can explain it.

Okay, and then, then, okay, op and then arg1. Just type arg1 comma og2 comma prop1 comma prop2. Okay, is that valid? And what does it mean?

I have no idea what it means. You wrote it.

Okay, so wait a second. So my question is it's not valid.

That's okay.

And explain why it's not valid.

No, it is valid. I mean, it won't fail not to be thing, but when, when, when you execute it, it will fail because OP is not divine in Python and.

No, no, no, no, no, no. I mean replace with op an operation supported in some language. Okay, okay, okay. So now if that operation takes two arguments, right? And then it takes a nested property.prop1. Prop 2, where do we go? Do you understand what I'm saying?

Stephen? Again, at the bottom, there's a more complicated example that I think has these multi for these multiple argument cases.

But it can be, it can be nested. I don't know, if you want to do stuff like Prop 1, Prop 2, you can just keep nesting it.

There is a simple explanation here. Basically this is tied to the dot operation. So it requires a second argument in the fully expanded form to always be external object. Everything else just follows the standard sequence of arguments for the method.

You're not answering my question. No, Stephen, this operation I'm trying to understand. In language X, when we are defining an external operation, the syntax you are proposing has two kinds of things, which is different from what we have in Wolfram language, where we don't have two different. Different kinds of what amount to function arguments, right? In this case, you have the things inside the parens and the things after the dot, and they're just. They're sort of like two branches of function arguments, correct?

No, Stephen, Col is always parens and get attribute is always dot.

I'm.

No, I'm asking.

Weird.

If the call has two arguments, okay?

The center doesn't have to argue and call is varag as well.

I didn't mean that. I mean, okay, does anybody. Todd, do you understand?

I think, I think there's some confusion here, because even that OP that you're seeing, that's not the name of a Python method. That's one of these magic.

I know, I know. So, so you're saying magic things only have one argument, right? That's what you're telling me. In Python, the magic things have one argument and therefore this is not ambiguous, correct?

No, it depends. The. The magic thing can. Can have more. For example, there is equality method that takes.

How does this work?

Hold on a second, hold on a second. That's not what he means when he says magic, Ricardo. He's not talking about. He's talking about Paul Eval import. He's talking about those as the magic. Whitney Stevenson.

What I'm saying is, in the Wolfram language, syntax here for external operation, we are not distinguishing between arguments and properties. They are being just fed in there as varargs. Things. Is that going to work? If you tell me you will never want more than one arg there, then the syntax is unambiguous. If you tell me I don't know how many args there are going to be there, I have a random number of args and then a random number of props, then the syntax is horribly ambiguous.

But I don't understand why you mixing those up. The args are things that you do with call, and props are things that you do with get attributes.

Because we have a syntax here that is doing that. It is saying use, put in something with call, put in something with. With get attribute.

But which one, which one are you looking at in on the screen that does that?

No, is the import. I mean the import operation. I mean, this is just. I mean, we.

Okay, I don't understand why you guys are not getting.

Forget about this, forget about this operation. But the thing is that this operation is doing is importing a module and getting something out of the module.

I know, I know. And if the importing of the module needed two arguments, which I know, that particular magic thing doesn't require that. If it needed two arguments, you have a messy ambiguous syntax. Does that? What I'm saying, you're relying on the fact that that magic thing only has one argument. And I'm asking the question. It is not at all obvious to me that in every language we can imagine, the magic thing that appears in an operation will only have one argument.

So isn't the case that the only thing that can have multiple arguments is call which calls a function?

No, you can invent anything. It could be a, you know, in some language this could be, you know, open a door to hyperspace or something, and it has 17 arguments.

But the point is, you see in Python, for example, call is vararg, get attribute is not vararg. And you split them in two different things so that you don't have this ambiguity.

I know, I know. We are putting them together in one thing. That's what I'm complaining about. You know what I'm saying?

No, my point is that for external language X, there will be a documentation of call, of import the documents, whether they're varag. Whether they're one arg.

So let me be clear on what you're saying.

Node JS is going to be called require, for example. I mean, with this design, you can really.

Yeah.

Do anything. Meaning that you cannot make a suggestion.

Let me make a suggestion. Okay. I think your syntax there is ambiguous. Now, it could be that in any particular language. Go back up to the thing where we had op, arg1, arg2. No, no, no, no, no. Just the thing I had you type in. Okay, yeah, okay. That is messy and ambiguous. All right. So you can make the decision for any given language. Oh, all the ops only have exactly one argument. Right. We can document that. My suggestion is there's a fallback here, which is external operation of an association, where the association has a thing that says, you know, operation goes to blah arguments, goes to blah properties, goes to blah. Whatever it is.

I'm not sure about that. I want to give the external operation plain and simple, like Wolfram language, and it's powerful enough.

Okay. Do you understand the point that I'm making?

I think. I think this example that's highlighted here that you had is not really. It's not meaningful. It doesn't mean anything in Ricardo's style Python.

It doesn't mean anything in language X. We have no way to guarantee.

I don't just take off out. I think you're thinking that this is some sort of chained call, like a function call with two arguments and then a property.

Yes, that's what I'm thinking. That's what.

That's not at all. You would not write that this way. Just change my call.

Yes, but. But also, I don't think, like, why we should design something like this. It's up to us to design operation and why we should design something messy like this. Okay, fine.

Okay. All right. Yeah. Can I be a bit.

That's not how you do chaining. That's not how you do chaining, Stephen.

I think that the question here is whether it's possible to design. These are S expressions, really.

This is lisp.

And so the question is whether you can design any abstract syntax tree of any language just with S expressions. And I think you can. Well, not they might be ugly, but they're generally not.

No, but this is a single list. That's the whole point. It's a single level list, that Verax thing inside external operation.

No, but that's the point.

It's S expressions. You define them any way you want and in case you nest more than one, and so you can have an S expression.

So you're saying you nest external operations.

Of course.

Which is what?

Ricardo? Fine, fine, fine, fine.

All right.

I guess it.

So basically this, this is how. Yeah, I mean internally, for example, when we do this, this thing.

Right, No, I get it, you'd convert it into a bunch.

I want just to understand that like the new imp, that it, when you take this association, is basically transforming this guy into eval, this guy into a code, et cetera, et cetera. So it's going to be a nested operation. It's just that.

I understand, I understand. So you're saying we are responsible for defining what external operation of does, and we'll make sure it doesn't do anything crazy. Correct. Okay, all right, fine.

Now the other thing is that the only evaluation that external evaluate the external operation is doing is that when you call it with sub values, whatever is the sub value that you're calling with, they're gonna be pushed and you are going to understand why. Just wait, please.

Oh, gosh. Yeah.

Okay, so now let's say we are calling the network with. Sorry, we are calling this object with sub bodies. Aust, as I was saying, is going to be an attribute.ost. this is going to be a function. You can call it like this and it's going to be. I know, I understand, Google, etc.

Etc.

So now you can call, you can do OST, external operation, call. What this does is that you need to imagine that AUST is going to be transformed into external operation. Yeah. Of get attribute almost. This is what it means. Really?

No, what are you saying? Fill that in. Get attribute there. Yeah.

And this is going to be a Google.

Wait a second, wait, wait a second, wait a second. Get attribute of us, the call attribute. No, I don't understand that. I don't understand.

Okay, so what is happening here? I'll show you. Basically, when you run this guy, okay, what is really happening is that we take this guy and we call it with this guy, okay? Then we take the result of this guy and we go with this guy and we keep going on. And in the end this guy evaluates by pushing this thing at the beginning. So you get, get attribute, network cost and Then you're moving this guy at the beginning of call, which is what is happening here. And so when you run this operation, first of all you're getting the attribute host and then you are calling the result of this operation with this object, with this call. And so now basically, yeah, this is, as I was saying, this is equivalent of writing aust. This is just a shortcut, which is why the other operation where we are doing OST of external object, external object, what really is, is get attribute of OST and cast to external object. So basically everything that is not an external object. Yeah, okay, get a meaning that.

Yeah.

I mean, possibly we are not solid is not a big fan, but yeah. So basically you're going to get asked, for example, you can call it ost, then we're calling the method string on it, right? This is, this is what it's doing. So in this operation, for example, let's say that we call the OS method and we return it as an external object. Now that we have this external object, we can call the next method. We can call it multiple times. And what the next method does is that it's getting the next item out of the Python generator. And so you basically you can get a generator out of Python that is that a generator in pyro can be infinite without consuming it directly in or from language that will blow up your kernel and memory, etc. So you, you, you get the generator as an external object and then you can manually call the method call. Sorry, next. In order to fetch the next object here I'm doing three times. You basically need to do it I don't know how many times, and then at some point it will start failing because the iteration is exhausted. Yeah, okay, yeah. The other thing that you.

Wait, let me read that for just a second, hold on. Stop, stop, stop, stop, stop.

Sure.

Out 88 is an iterator object, is that what you're saying?

Yep. So basically here I'm getting out the host and I'm calling it a string, just to show you that what the call method does really is to return a generator. Now, in warframe language, when we encounter a generator, okay, to expand it.

I get it, I get it, I get it. In the external object, you know, visual thing, why are you not calling the string operation rather than giving the rather useless information that is in that external object summary box? Yes.

We could.

I think it would be useful to show what it is, because what I'm seeing from that is nothing terribly useful is module is class, you know, who cares?

It can be Done there is like a quirk, for example, some Python object. They could be like, you know, for example, a database connection, a database query calling a string might, might trigger an operation that is expensive. It is true that when you do it in Python and you, and, and you are doing it interactively, this is what is happening. You are triggering that operation. So this is a front end and it's.

Has a callback here. When it's kind of trying to format, it has some callback. It could have a timeout, you see what I'm saying? So that you can actually see.

Yeah, but there is another problem. Yes, there is another problem with that, Steven, which is that it may very easily go out of sync because in general we are dealing with mutable state and it will get the form that the object had.

At the moment.

I think that we run the string operation when we return the external object. This way it won't get out of sync. I think, I don't know, it could.

Get out of sync. Listen, listen, listen. It's the thing that is on the screen, okay? It's just like if I say x equals 7, I type x, I get 7 on the screen. Now I type x equals 8, I type x, I get 8 on the screen. But I've still got that previous 7 sitting on the screen.

Right?

Do you see what I'm saying? So at the time when the printing happened, the value was this. And I think that's a perfectly reasonable thing to do with external object.

But Stephen, I think that sort of like in cases where we had the boxes do some kind of external operation, like for example, the entity label, it's stuff that we've had to put weird things in the, in the front end to support it. And many people who are trying to do something with large amounts of data.

Just turn it off because it's, that's because we didn't do our job properly. That doesn't, that's not an excuse to not do our job here. We should get something that is actually useful to see rather than a big long list of things that say that are not classes, which is not very interesting.

What if a type name is the. Rather than some trying to represent the value of the object as a string?

Well, the type name will be a good start. But you might not know the type name. I, I'm just telling you, I think we can make this more useful. Right?

Yeah, but I think I might agree.

I'm not sure this is an external language. You have no control on that. Maybe, you know, maybe string is triggering Something that has, who knows, you know, increases some counter. And then you can do it programmatically.

Okay, I'm just telling you, whatever an interactive session in language X does, that's what we should do here.

It is true that when you do in Python, right, You evaluate an expression in Python and you do the number two, or you import a date, time and then you do date, time, date, this thing formats. And so if this is an external language. Well, yeah, okay, whatever. Yeah, right. This informats. So we should. We can probably do that because Python itself is doing it. And this is, you know, this is no different from. Right, so, okay, so that interactive session, more or less, I think it's. It's okay. It is true that when you're calling string over a query or for example.

We'Re going to run out of time here. We're going to run out of time. So let's.

Okay, yeah. Not much. The. Another useful thing that came out is that, you know, you can call an object with a function. So let's say that you have a function. Yep, I'm creating three functions here. And then I'm calling the object network with. With the function. Basically what I'm doing is to call the function over the object, really. But this is useful because you can chain it after other operations. So for example, you can do ost, you can do call, then you can run topple. So you are converting into tuple, and then you are asking for the length.

I don't understand. This. I don't understand. This is on the sub value. What the heck are you doing in these sub values?

Well, I could fail, right? I mean, I could. I could simply fail. I can say that the external function is not allowed to run on sub values. Right. But it's not terrible.

What, what, what are you doing? What does that thing convert into? Is it right associative?

It's.

It's doing that list of sub values. Is there some kind of right associative thing going on there?

Yes, yes. So basically I'm creating the function, then this is the network. And in calling with the object, again, really what you're doing is the opposite thing. You're calling rep with object. And this is useful because you can chain it. And so saying, you know, I'm calling ghost, you know, then I'm calling double, and then I'm calling length. And this will roughly.

It's really right composition, Stephen. It's really right composition. And it's literally right composition. If you write it in this way, it's just expressed in More convenient sub value syntax.

Right. And we.

Yeah, so just write it out. Just write it out. If you were writing F of F of list, you know, just type F F open bracket A comma B comma C. Okay, now write out what that really is. Is it C of B of A of F or what?

No, I mean obviously for a genetic F. Oh, okay. Yeah, yeah, this is.

Write it out. What is it?

Of F of B of C. Yeah.

That'S what you said.

Yeah, that's what I said. Is that what it is?

It is what?

I don't think that's quite what it is. Are you sure?

Because if you, if you, if you go here, you will see that basically as I was showing before. Okay, let me do it manually.

Can you show example with the right composition? We have it just literally.

I don't know if I.

Somewhere on the screen that would explain things.

That was here.

Correct.

But not with the object. I would like to do it the same with the object. Which is what we did before. Right? I mean we. I can quickly write it again. Basically, when you're doing this, what you're really writing is a get attribute operation here. So we need to expand it.

Sorry. Okay.

Get attribute.

Sorry.

You have an extra P as well.

Yeah.

Okay. And what you really do now is that you call this guy with network, right? And then you call this guy with the code. Then you're calling it with double. Then we're calling with length. Conceptually this is what you're doing. Obviously it isn't quite this, because if you take double, this is an external function evaluates immediately. So obviously there is some transformation involved. And this is, for example with this call, this is the external operation that you get out where basically, you know, every one of these object got a reference. So this is the operation, something that the user never needs to see. But you know, when you're getting an object on the other side, there is an operation for that that is get reference of a number that correspond to a pointer for that object.

Okay, let me ask a meta question. Yeah, yeah, Meta question. Have you been using this syntax? Yes. To write non trivial code? Yes. And has our node been using any of the syntax?

Not yet, but I think he can answer.

Okay, so let's look at this one.

Yeah, it's not working yet. I mean I'm kind of, you know, as soon as I can use it, I think I can give some real world feedback on this. But this is what we talked about this morning. Like as in the real world example that I am working on. That's a KERAS is a machine learning framework that has all sort of like, you know, dot syntax and arguments, all sorts of things. And then Ricardo translated that for me into a, you know, external operation syntax.

Please scroll down.

Well, it doesn't work because I didn't spend time making it work, but in theory should work more or less basically every one of these guys, an import operation that you pass some data to by calling it, by calling the operation with data. And then again, this is again another import operation that you do it by applying, but by calling it with. With all these guys.

But we didn't really discuss arrows in the rule syntax. Right.

The rule syntax is uncontroversial because it's already into the system.

But you can use that sub value thing as well.

Sub value. We are not using sub values here. But yeah, I. Yeah. What's worrying you, Stephen?

I'm a little concerned. Okay, so take R as an example. Leonid.

Right.

Do you feel like something directly analogous can be done there?

In principle, yes. The only difference is that R much less emphasizes method calls with the dot, although they have that syntax as well. Because for R you R has slightly different concepts of methods. They are all generic functions for R. So essentially, because R is much more functional language than Python for R, methods are kind of the poor man's analogs of functions in that sense. In R the situation is even simpler because essentially you can get away by just having just functions.

And so this syntax that Ricardo is proposing, you can have a version of that for R and it will be okay.

True or not? Yeah. The only thing is I have to say that for R you will have fewer cases when you will chain things like that.

Oh yeah, okay, I got it. Okay. Other interactive languages that we already support.

So for example, no, they'd be. I already have an implementation for node. The only difference is that I think I will call instead of import, I will call it require. Will work quite the same, I guess, because it's going to be a part and an attribute that you are extracting.

Have one meta comment pretty quick. Basically, this syntax works best when you have object oriented languages with the calls where you use dots to have chain calls. That's where it decomposes. And what this design tries to reconcile is the concise analog of the chain function calls with the proper Mathematica style. Sorry, Wolfram language style. Nesting, Functional nesting. And that's what this engine does?

Essentially, yes. But also like the fact is that this design is kind of generic in the fact that when you're calling it with, with object with sub value, you know, a single string gets translated into a default operation for the system. So we can imagine for example that for SQL an external object is like a query or something and when you call it with sub value, I don't know, it can be a column, could be something different than getting an attribute, you know. And so it leave us room to basically implement what's best for every language really.

Right. But I mean also one point I was trying to make is that one real difficulty we had with the design was object oriented languages which used dot notation because we wanted to translate dot notation and chaining into the functional nesting. And this is what this thing does. For languages which do not emphasize dot notation, things are even simpler and R includes ARRIS in that category of languages. So I think that with this design we now cover pretty much all there is.

Okay, the just throwing two things in there and then we'll wrap up SQL and the shell. Just talk to me about whether this is relevant in either of those cases.

Okay, so shell I'm not sure. I, I, I, I, there's not really.

An external object type thing.

PowerShell has, has structured data.

It might be, I mean we, we can invent an external object that basically contains because right now we are returning a success and a failure object by default. Maybe that thing can, can change and we can return an external object. I'm, I, I'm not sure for shell. I haven't been thinking it through for SQL. I think it's relevant in, in a sense that basically, you know, when, when you're doing an SQL operation, you can ask for a, a million of rows and you, and, and, and, and having a value that is external object allows you to, to have an iterator on the other side of the, of the cursor. So it makes sense to do it there even if we are not really, we haven't discussed SQL at this point.

So yeah, I would rather, but I think put SQL into a separate category to be discussed yet because it's okay.

Yeah, but I think it's, it's something that, that will hit eventually.

It might.

I don't see.

All right, okay, but it's not, but okay, my last question. I mean is there anything here where, I mean, I think I understand what you're doing here. The fact that you are able to use this and it seems to work nicely for you is good. You know, we are clearly trying to sort of build a giant marble building in the middle of a jungle type thing. In other words, we're Trying to make some kind of, you know.

But I must admit that, you know, last time when, when you raise all those concerns and you force us to redo this design was a good thing because this is now better, I think. And also like we learned and everyone agrees that this is now better because previously the ddd, this is strictly more powerful. And also like, it allows us to expose external operation which allow us to do lazy evaluation, which is something that people one need to do immediately. Probably when you're prototyping and writing and doing stuff interactively. But when you.

No, I like that's a good thing.

I mean, the only thing that I can compile, right. It's. It can be seen as a compile target.

Right. The thing that I'm concerned about here, okay. Is that these languages have function call, parenthesis, notation, they have dot notation. And we're kind of somewhat munging these together. Although as you say, there's a full form in which they're pulled apart with external operation, which is okay, I think. And then the question of what the el cheapo version of that is for each language is something that you can presumably just document.

Right. And also I think that this, one of the strongest point of this design is that it makes writing these chained codes that people write all the time in this object oriented languages really snap and very close conceptually and by how it looks to how they would write in their target language. So that would make it the transition, if people want to write something like that in Wolfram language, much easier.

Yeah. Okay. Anything else for this?

No, I don't think so. Yeah.

Great.

We have a green light on. I mean, I remember you were having some concern on multiple attributes stuff.

Well, I think I'm. I mean, I think you've convinced me that it's okay.

Yeah.

There is an explanation of this thing is expanded to a list of.

Wait a minute. Wait, wait, wait, wait, wait. What's this thing here? Hold on one second here. Wait a minute. What the heck is this? Lines.

This is multiple attribute extraction.

Basically put this syntax in. I mean, this is. Oh gosh. This is using. So the concept that it is essentially listably mapping over some multiple things. I get that it's going to turn into a list. This second case here, the nested list thing on line 1673 is.

Okay. Basically what I'm doing is that, you know, I can see what you're doing. Every ARC that is here got pushed in this list here. Because doing a combinatorial product of these will be crazy. Like if you have another List here and another list and another list. Probably you don't want Cartesian product.

No.

Yeah. And so, and so what we are doing is that we're pushing arcs year and year, more or less.

So in other words, what's happening is that first level of the list we can consider as a set of properties or.

I know, I get it, I get it. But the fact that you're treating. Okay, yeah. What was Verogs you have stuffed into a list now? So what you're telling me is address of list, list ABC is equivalent to. Could you write it out? Just address. Just write out.

Yeah.

Address of list, list, yes. Abc. No, just a comma, B comma, C. A comma, B comma, C. Okay.

A comma, B comma is equivalent to.

Maybe forget the strings. I don't care about the strings.

I got it.

Is equivalent to address of the thing without any, any braces.

Correct. Except that this guy will return an extra list.

So the, the list goes outside. Right.

So, yeah, so it's a list outside.

So this guy will return a list with one. With one object inside and this guy directly returns one object. That's the only difference. But yeah, they are equivalent.

Are you happy with that? I mean, it's a little weird in my book and I mean, I think there are many cases where.

Well, I think this is useful and is especially useful for interactive use, I think, because you want to explore an object and get multiple stuff out of it. The problem is that it would have been better if this guy will be returning an association. But it can't.

No, no, I forget that I'm. I'm just concerned about whether the multiple listed lists are sent.

They're going to be flattened, but. Yeah.

What do you mean they're going to be flattened? They're not flat.

Well, yes, they are flattened at some point. Meaning that if you have. Because what I was saying is that if here you have another list and then you know what, what I'm simply doing.

And what if they don't agree in length?

No, no, they don't. They.

They.

They don't need to, because this guy.

No, I understand, I understand. Those are. Those are going inside.

Yeah, they go inside. But they go inside even with the list. And then I'm just going to flatten on the second level and then I'm going to normalize.

What the heck is that? No, what is that? I don't understand that.

Well, I'm not going to document that. You can write that. But, but, but, you know, I'm. I'm just I'm just saying that we are flattening on the second level and then we're normalizing this list.

Okay, I have to go to something else here. I have to go to something else. I'm afraid that I'm a little concerned about this. I, you know, I would implement, you know, I would document the syntax that you were really confident about and I would leave some of this other stuff that seems. I mean, this multiply nested lists where, you know, where it's kind of a.

Toggle of whether you get it makes sense. I mean, it's not really necessary for the first iteration.

Fine, yeah. Then we've got some freedom to deal with what these lists mean. But I think the things. The fully qualified syntax and I guess I'm reluctantly agreeing on your right. Composition thing. Okay, that all seems to make sense to me.

Okay, great.

And I think this single level list as a way to get multiple, you know, that that is what we've done in entity value and what we've done a million other places.

Okay.

So what I'm saying this is the double level thing is seems to me questionable.

Okay.

I don't think we have precedence for that either. And I have to run, but thank you all and this is, this is nice. Good, good stuff. All right. Okay.

Thank you everyone.

See you all soon.

Bye.