Live CEOing Ep 769: Language Design in the Wolfram Language [Engine Connectivity Engineering]

All right. Hello everyone. Welcome to a Wolfram language design review. This is actually one we don't usually livestream, but today we're talking about language design. So we are, we are going to be talking about integration of external languages, particularly Python. And whoever is sharing. Is that Ricardo or Roman? Why don't you take it away and describe what we're talking about here? Nope. Who is Ricardo?

Hello. Hi everyone. Okay, so, okay, so basically we are doing an incremental improvement of external evaluate. Specifically we are talking about external object. This is actually something that is generic to every evaluator. Not only Byron, but we are demoing Python today mostly and some SQL. So basically if. We don't have a direct translation to the Wolfram language object. So for example, in this case what I'm doing is to import and work with a library that allows you to work with IP address. When you execute this command, you're basically referring to a Python module and we don't have an equivalent to represent this in Wolfram language. So you're getting back an external object. Now we are adding more capabilities to the external object. So primarily is the ability to send it back to the evaluator that unfortunately you were not able to do it before. And what we are also adding are those method call that we are going to present today. So let's say that you execute an external evaluate call that returns to you a module in Python and then you want to write the equivalent of this Python code. So basically calling a method and passing a string, you can do it by doing lib. So the object that you already refer to and then you do method call and then you pass the name of the method and all the arguments in a positional way. And this gives you back another external object because the IP address class is not something that we understand as well. And so you're getting back now another external object. And now let's say that we want to get an attribute of this object. So for example, the version of the IP in this case is an IP4.

I mean, question. I mean, first of all, this is similar to the way that we're doing data structures and I mean, next question, is there a pretty print that a typical Python object has for itself? Is one of the methods typically a pretty print type thing or not?

We, we can add it. Yes, there is, there is a thing that is called rep, and basically it's what you see in, in the terminal.

Okay, so then we could display that in the external object.

We, we can decide to display it. Yes.

Seems useful to me.

Yes, I mean the, the problem is that this is a reference to an object. So when you do something like this, obviously you, you, you are doing an operation that might be expensive. It depends on the implementation on the object. Most of the time when you're asking for the rep, which is something that, that, that is coming in the terminal is not something that is terribly expensive, but sometimes it can be for know, SQL Alchemy or Django, they provide connection wrapper to a database. When you do a wrapper of that object that is actually triggering a database call.

But can we have a timeout? Can we do a timeout? Can we try that call? And if it takes more than, you know, half a second or something, we give up.

Yeah, I mean we could and we might decide to do it all in an interactive session for sure.

Yeah, we don't need to do it otherwise.

Yeah, or hide it behind the plus sign. Right. For more more.

But then basically interacting with the boxes will trigger an evaluator call, which is something that we don't usually do in boxes.

I think we shouldn't do that. But I think, I think the idea of having a timeout seems useful to me. If we can do that, if that's easy engineering. Because what could go wrong? I mean if we have a timeout, if we can really time it out after half a second or something, it's not going to be that bad.

Right.

Question. In that initial thing, the Python session prologue and so on. Yes. What other things do we have in that when we have start external session, what other decoration do we have to the registered name in this case Python.

I mean so the available things are. So there is session prologue, epilogue, there is a common prologue comma d log then there is the actual description of the evaluator. So the path or you know, the, the new thing that we have done in version 14, where basically there is an association, etc, why aren't we putting.

The session prologue in association? I don't understand why we have to put. I mean that Python thing is going to the registered evaluator. Registered external evaluator that is called Python.

On a session basis. Maybe you, you, you only want this on a session base.

No, no, I understand that, but we've got two different things. We've association that represents the evaluator and then we've got the decoration to that which is the prologue that might be local to the session.

Right, right. But the association that describes Python or something that is for now is Python specific. On the other hand, session parallelog is generic is for Every evaluator, which is why it is not inside the, the, the, you know, the environment or evaluator, sorry, etc. Etc.

Yeah, I think it makes sense to separate the specification of the evaluator with things like Session Pro.

No, no, I agree, I agree, I'm. I'm just asking about the syntax of this list with rules and so on, whether that's an actual, you know, properly promoted option to start external session or what it is. I don't remember what else we put in.

Well, in theory you did the full form is an association, you should write system goes to Python and then all those things. And so this is just syntax sugar for the full form. There is, there is an actual association, etc.

I don't have an objection to it. Okay, keep going.

Okay, so we are saying you had an address, you want to access something else you do. Get attribute. Get attribute is a way to. Yeah, get an attribute and return the value. Again you can return reverse pointer, etc. Now for example, here we are instantiating IP network range and then we are calling this method. So basically what we're doing here is to call host. So now this is returning a list of all addresses that again we don't know is related, but we are returning to an external object. Now this is the function that I was talking about, rep. Basically rep is a function that gives you whatever is throwing out in the terminal Python and what you can do now is to do something like map of this function with all the host and this is returning the string representation of this object. It's important to notice that this one, it's one call.

Basically I don't understand that syntax at all. I mean external evaluate, what does that arrow mean?

Okay, so in external evaluate, this is a function call. So this is syntax sugar for doing roughly. Let me, let me make sure that's.

Shorthand for, you know, function arrow arguments.

Cool.

And then so I need to. Okay, so no, I need to go map. So I will need to do external evaluate of map. And then I need to call map with rep and then all the hosts and I think this should work if there wasn't a bug.

The health has been mapped of the EF rapper.

Yes, but this is probably something I'm not evaluating all the world host hosts.

Is probably not evaluate.

Okay, well, not sure what it is, but, but basically this would be the equivalent.

Understand that syntax. So you're saying that in external this is a functional function, those are its arguments. I just don't understand syntax.

Okay, but this is a syntax that already exists. Basically this is function call. So basically this is map. And this will be the equivalent of writing map with rep and then all the oss here.

But can't you do that with the string template as well?

You would.

Yeah. I guess my question is more like do these external objects and external functions work by using the backtick syntax?

Because I would reach for that not.

Knowing about this arrow syntax.

Right? And I don't understand what the relationship between this and the angle bracket, star syntax where we get callback to Wolfram language is. Because this is basically having to take a Wolfram language object, external object, for example, and provide it as an argument to map. So why isn't there something like map and then angle brackets, star, external object, blah, blah, blah.

You could do that as well. You can do that as well. But you couldn't do it if those were all objects. Right? So if you. Because this is working as well. So if you now have an external function here, right. You, you can also do this one and obviously you can call the function as well. But you know, in for more complex operation, basically by. If you're using this, the syntax, you don't deal with strings.

What happens if I have a nested function going to blow it up? What's that?

No, no, no, no. Because the syntax for rule doesn't work inside, it only works for the.

I know, that's what I'm saying. If you have a nested function, you can't use that. But let me just understand. Could you just write out map, you know, instead of the map, angle, bracket. You know that map is a, is a top level thing. So you've had a function called map. So instead of the map, do angle, bracket, star of the map, the word map. Replace the word map. Replace the word map by angle, bracket, star, map.

Yeah.

Okay, since you've defined map up above function, this will work as well.

Yes, this will work well. But I don't know if this is more readable.

No, it's not. I'm just trying to understand. Okay, so just, just so we are understanding this, you can do it as well. And I mean in terms of syntactically typing it in. Okay, so what we're saying is each one of those, when it comes in, it will be converted to the appropriate native Python thing so that the evaluation can take place.

It will be refreshed from the, from the original session replace. And yes, the evaluation will, will be done and then it returns the results.

Okay, so that arrow thing is merely a shorthand for function application. Yes, yes, basically.

Yeah.

Then this rep, you can call it with number two, for example. And this is now a string of the number two, you can call it with map. And this should give. I don't know why this is.

So you can't call map of repper comma host.

I have no idea. But basically this will give you the representation of the function.

Yeah. So why is it, I mean, why can't you in the top level code just say map of, you know, rep hosts as a piece of orphan language code? Map, map.

You can do that as well. Yes, you can do that as well.

Let's see it work. Yes, come on.

Unfortunately, I have an idea like. Yes, I have redone the old implementation design and.

And of course, because we're doing a review. So you. Do you redo it today because you broke it.

Yeah.

Yes, that was working this morning.

Okay, but what, I mean, that to me is the most elegant version of this. Yeah.

Why is that call in 50? Why is that call and not method call? Does that, does it need to be call? Is that different than. Is this like not a regular method call or is it just a method call?

No. Okay, so this, since this is a function, you directly call the function with arguments. And so. Yeah, I'm really not sure why it's not working.

I mean, the fact that you can take a Python function but the result.

Is equivalent to this one, the result will be equivalent to this one.

I understand that, but that's wonderful. I mean, that's much better than having to write external evaluate of the evaluator, comma, whatever the fact that you've got that, and I mean, presumably we've got good messaging, if the evaluator in the map and the wrapper and the hosts and so on doesn't match. See what I'm saying? If you said map and I got that from a Python 3 session and I've got a wrap up on a Python 4 session and they don't match, you're not going to.

Yes. So basically we. There will be a message that says, you know, that the object with ID whatever doesn't exist in the current session and.

Yes, okay, but just try and clean those messages because to me, this is a very nice way to do things, to just say, I've got a function and oh, by the way, it's an external function.

Right.

It happens to be an external function, but I can treat it just like. In other words, the fact that you can sort of, I mean, if I. Right. I mean, it's very nice. So in other words, if you say wrapper, could try it map. Say that, the lowercase map and then say instead of hosts there say map, wrapper, comma, range of 10. Range of 10. That should work, right? It should convert each.

That should work. It's not working. Let me try it here. Let's see if that works. I'm not sure. Maybe it's a bug in external function. Yeah, here it's working. But yes, I will fix this thing.

Because that's very beautiful. I mean, the fact you can just treat. In other words, an external function is just like a function function and you can use it in place of a function function and it does automatic conversion of types on the way in, et cetera.

Right, but that's, that's the way things work right now too.

Yes, yes. This is something that already exists. What, what I'm adding here is the fact that you can put an external function here and it will just work even with the old syntax. And so now you can put objects, functions, etc. And the old syntax with rule, et cetera, et cetera, everything works.

Now I don't really believe in that rule syntax. What I do believe in is if you, instead of range of 10, you had some list of objects like hosts, for example. The fact that that works is non trivial and lovely.

And it will be converted in one backend operation. So it will be done just in one shot when you're doing it like this.

Okay.

Another thing that you can do so with the system is the ability and we want a way to chain multiple calls. So basically let's start again from IP address and let's say that this is something that we've wrote before, right? You do IP address, IP address and then you get version. Now let's say that there is an error and I'm simulating an error by doing a spelling mistake here. You do it on the first one and what happens is that you're getting a failure and now you're calling the failure with sub values. And in theory you should be handling the. The error that are coming out at every step. That can be. That is both a problem in efficiency because this thing is doing too much.

I don't understand. Why aren't you just making a thing called external failure that does the right thing with the sub values?

Oh, well, we could, but that would be a big, big.

Yeah, okay.

But you have a different incompatible change. Yeah, no, we cannot really do that.

Well, you sure? It's not a good idea.

I mean, yeah, but that will be a big income.

Keep going, keep Going what were you.

And, and so what we have done is that, you know, you can chain calls by using a list instead. And so now what you're doing is that this guy will basically throw one error. The first one.

I'm not really that keen on this kind of weird sort of syntax within syntax type thing. I mean, you know what with, between the arrow and this, it's like you're learning a whole new language to be able to make these calls. And that's just not what we want.

I know, but you know, if.

It'S.

Not those things that are already in place and the best we can do now is to.

I don't, I don't agree with this. I don't agree.

I mean we have something similar in, in data sets, right? Where this, whether you can just write about just about anything. There's a lot of new, new, new ideas in that one argument of, of a data set where you can put in queries and selects and everything.

I know it's not the best design.

Yeah, but, but this is really like a big requirement of this system because the, the we have been debating this thing endlessly and, and, and, and the reason why you want to do that is because otherwise you will need to wrap failure.

I understand what would happen if we had a concept called external failure that.

Basically everything that is working on that is doing underscore failure wouldn't work anymore.

Well, unless, but most things like confirm, for example, you know, most things like.

Failure queue and stuff like that will work obviously.

But.

Yeah, but I'm not sure like an external failure. In any case, why do you think it will solve the problem in the end? The external failure will have properties that you need to extract and it will have a system that is similar to this one. So it doesn't really solve the problem.

How are you doing? I mean, I don't understand what you're doing to do that second thing, where does it pick up? So it's running both of those things together. The first one failures out. Yes. And then what? And then, but then the whole point.

Then this guy throws an exception and.

Right, and it eats it and the second one eats it. So in other words, external failure would just say, oh, I've got sub values. Ignore the sub values. I already had an exception. So I'm saying the problem that you have is online.

Yes, I understand. But then, then this external failure, you, you, you want to access it programmatically and if you don't do it with sub values, how would you do it? I mean the failure Will have information that you need to access.

Okay, so we got a designer suit here. Okay, so just. You know, I'm not buying the fact that we have to invent this weird. You know, every time we invent one of these weird things that requires people to memorize yet another kind of weird structure that isn't our standard structure, we lose a large fraction of people.

I mean it is a form of function composition. I wonder whether we could there was a better way to represent this. I mean the basic idea that you have only one round trip to Python, you don't have to worry about intermediate failures, I think is very worthwhile.

Yeah, I agree, but so, all right, look, you're basically saying that method call allows. Sorry, your mouse is right on the place where I need to see what's going on. Okay, you're basically saying that an external object. Can take. It's really weird. I mean it's kind of like a.

The only alternative that we got here is that basically that the original design was entirely lazy and so all those code will always returning external object and then you need to explicitly send it to evaluator. And that was done in order to avoid this specific problem. Error handling and error propagation. Yeah, it seems silly, but then you will need to wrap it every time in external evaluate and this is silly.

How far down do our sub values go?

One level.

Only one level. Yeah.

That's the thing we wanted to.

Fix for 35 years and up values, right?

Yeah, yeah. Okay, so I'm trying to understand here. This. What is the syntax? If you had only one of these. Is it in a list? You're doing method call and that's arguments in a list, Is that correct?

Yeah, I mean the, the alternative is to only do this syntax which is. Which was my original preference. But then, then you know, this is basically a syntax sugar to this one.

We've got. No, wait a minute, wait a minute, wait a minute, wait a minute. We've got that vero form.

Oh my gosh, Ricardo, this method call and get attribute thing, do we them do.

Do you.

Sorry, do we need method call and get attribute like. I understand.

Yes, you need.

There are Java but like if you know how to use objects in J link, this is like again entirely new. Just even having method call is entirely new.

Okay, so, okay, let me. We can go back to the. Steven, we will go back to this issue. We'll continue for a little sound here. The idea is that Those methods, those APIs. The first argument is going to be something that is specific to the evaluator we can, you know, get attribute and set attribute is something that probably is common to a lot of them like node js or whatever. But for example, it doesn't really make sense in SQL, right? Or, or in other system. And so the overall idea is that, you know, the evaluator defines and document those methods. They can be Python specific. We can decide whether we want to go and put a lot of Python specific things or adding just a small set of standardized things. But what we are doing is, for example, set attribute and get attribute, get item and set item. There is also a way to, to explicitly set the return type. This is something that for example, is Python specific. Every object got a length and we could have a length method that is returning the length of the object.

And so somewhere there, there's a discovery interface that says this evaluator for its external objects defines these operations. Can I see that? Is there?

Excuse me, What?

No, no, I'm saying given that S. Yes. Whatever it is the external session or the, the evaluator being used on the external session, there's got to be a way that I can get a list set attribute, get attribute method call, blah, blah, blah, blah blah. Where is that?

Yes, it could be either properties on external object I haven't implemented well that.

That, that, that on the individual external object. But you're telling me it is global to this particular evaluator? Yes.

Of the external session object. Yes.

So, okay, so then for each one of those there is a list just like for. How do we do that for data structure? How does data structure. How do you do a query for data structure for a given data structure like Linked List? How do you know what, what operations you can do on Linked List?

Not sure about that.

Let's look it up.

I still don't understand why we can't have something simpler like J link had.

Well, I will, I'll jump in here as being the J link guy and liking J link and all that. But I think that I like the method call kind of syntax here. I think we need that. I don't think the J link model doesn't work really perfectly well for Python. And I think that we need a sort of reflection style where you're directly calling these things sort of by reflection sense. You know, here's the object, here's the operation I want to perform, it's a method call, and here's the name of the method.

Yeah, I mean, for example, like obviously this whole thing is broken now because I've already dealt notebook. But you know, for SQL, you can imagine having something like this where you say return type of roles. This is not implemented, but it will be implemented. Basically you do a huge select star from a huge table. You get back an external object that is a reference to the cursor. And then there are some methods that you can do over the good sort.

But you're changing fundamentally the construct of what external evaluate does, because then you're not evaluating. So if you say external evaluate, or I'm assuming that the moment it is.

Evaluating, it's creating a cursor on the.

Other side, it's just creating a cursor, it's not evaluating. So every time I do 10, then I move the cursor.

Yes.

So when does the evaluation happen? At line 84 or 85? Kind of both. It's changing.

Here you are creating. Yes. Here you are creating only a cursor, and here you are fetching and. Yes. And mutating the object.

Yeah, but then that thing, if you evaluate it twice, if it's the cursor, you evaluate it twice, you get twice the different 10 things. So that is like, okay, maybe it's a useful thing to have, but it's fundamentally different to what we had before in external SQL.

Yes, but right now in external evaluate, you cannot iterate a huge table. And this, this is a way to do it.

Yeah, I know, but you. What I'm saying this is useful, but it's a different syntax because the return.

Type is an external object and that is mutable. Right?

Yeah. The overall idea is that external object is something on the other side that is mutable.

And it doesn't conflict with the existing usage. Right. Carlo, this is a new thing for working with objects and it doesn't. It doesn't conflict with the old use of external value for SQL, does it?

Because. Because of that arrow external object. Well, that's return type thing. I mean, just.

Yes.

Yeah.

First of all, I see.

Sorry.

I think it's. It's like. It's confusing that the modifier for what gets returned is in the first argument.

Second, it can be also in this other thing.

And the second thing is that a cursor is an iterator, and I think it's very important that we introduce iterators consistently throughout the language.

Not exactly.

I wanted not have something that's an external object that sometimes can behave like an iterator. Sometimes does it.

Exactly.

But external object is something that is generic to the framework. I mean, the SQL can define in the future when we have iterators, when they're ready, whatever. You can have another return type here that says iterator.

I think it's over generalization and it's going to be. Languages are, I think languages are substantially different. We are now trying to shoehorn some very specific SQL like concepts into this model of objects which, you know, I don't think that particular example is really good for the SQL. This requires a separate foot. How to do this? It's a special case.

Well, I don't particularly agree with this statement, but this is just an example to say that in external evaluate there are a lot of different system and different system can have different methods and this is what I was going for so that, that basically those APIs are going to be different for every system. Now we can decide that, you know, we, we are not exposing external object at the beginning in SQL. I'm not sure why, because I think it's, it's, it's a great thing to have because waiting for iterators is going to take a lot of time on the other end. This thing is ready and you can work with it tomorrow and allows to get things done. So you know, the moment we have Iterator we get, we can just have Iterator here. So I don't really see a problem here.

Well, I think we can discuss it in a different. Maybe.

Sure.

Because I think it's a deeper discussion that for this overview. But my point is that different languages are substantially different and external object to my mind is a more or less a reference to something that exists on the other side as a mutable, which is a cursor. Yeah, well, okay, maybe, but I would say it's much less typical example than for languages like Python. It's a much more special case. That's my point and it has to be done with care.

Right.

Because the stateful objects in SQL are very different from stateful objects in other languages. Very, very different.

Right, but, but, but in any case, the overall discussion here was to say that we need this first argument because this first argument is going to be substantially different for, and allows you to do very different things in very different evaluators.

Okay, but so the concept, look, it sounds to me like external object, like external evaluate, you know, it is introducing you to a different jungle depending on its first argument, so to speak. Yes, and so some things are going.

To be common to every evaluator. Like call for example is going to be function call or you know, get attribute is probably going to be present in a lot of them, not SQL. Obviously eval is the Method that is evaluating call code, but you probably don't have to deal with this. But yeah, there are some, there are a lot of APIs, some are internal, some are external and yeah, some are common to everyone.

And by the way, I, I will have to go in about five minutes. But. But what. So what are the design decisions we have to make? So there are these slightly weird syntaxes for, you know, the combined. That, that, that syntax online 50, 58. The combined multiple function call center. Yes, yes, that's.

That has to go back to the drawing board.

I mean, well, you know, I, I'm not a huge fan. It looks weird to me. But it, I only saw this the other day. I think Sean Sharon gets the credit for suggesting this. But it solved a lot of problems. Yes, the minute we saw this it was like, oh, that solves both the eager lazy problem and the issue about whether methods recall always, always return objects or sometimes expressions. That did solve a lot of problems.

Well, so wait a minute, on line 55 the not.

Ah yeah. There is also another problem that, you know, this thing, depending on what is the return value, could return a string or an object. And now it's not only about failure. This thing could return the number.

I get it, I get it. I'm asking.

And in this case it will not return the number two because.

No, I get it, I get it. Because it knows what it's doing next. Exactly. So my question is on line 55, the fact that it is the syntax doesn't have a list there that it's a varargs method call is weird given that you're going to go listing, you know, you're going to use var args to mean the sequence of operations there. And in the other case, so, so I have a question. What, what would happen if. Could you have something. Maybe this doesn't work live of method call. Just liber. Method call. Close bracket. Okay. And then a sub value of its. Of its. Of what it. Okay, that's one thing is putting that in a single list, which obviously has to work. But that then this method call. Open bracket, open bracket, open bracket. After that. After that. Right. And then put the IP address and so on in there. Okay, now let's look at data structure. Because data structure. I believe that data structure always does.

But then you will need to. To. To add a lot.

Just let's look at what data structure does. Okay, this data structure always have a. No look at how a data structure is used. Listing of data structures. Go to listing of data structures at the top There, up at the top. Okay, now just pick a data structure. I know, linked list.

Okay.

Let's say whatever, whatever you want. It doesn't matter the list.

Okay?

Yeah. Okay, now let's look at the examples. Okay, there. So, right, so this is always operation and then arguments. Right, Right, right. And it has no way. Because it's always returning a data structure and there's no. This consideration of erroring is not a thing. Yes.

I mean, my original design was exactly like this. You, you, you, you do this operational. You're always getting back an external object, but that, that requires you to. To basically use another call in order to see the value.

Okay. The question is, how do we package in this case where we want to go to sort of a full form where we are putting in multiple calls there. Right. The method of packaging is this weird thing where you are replacing the varogs of the original call with, you know, list verogs. So my question is, is there an alternative that we can come up with for how to represent. See what I'm saying? That, that down there. I have to say. Go ahead.

About your. Your lib of method call of. You know, maybe. Maybe there's something we can.

I'm not sure about that. But listen, I think it would help a lot if that last one down there had a list around it. That one there had a list around that. No, no, the whole thing. The whole thing. All the arguments.

All right.

List. No, no, a list around those arguments. Right, like that. Because then at least you're not dealing with this incredible weirdness of the varogs. Means one thing in one case, another thing in the other case.

But then you would only allow this, right?

No, you could allow the varargs version as a courtesy. That will be equivalent to putting it in a single list.

I see.

Well, but on the other hand, with the varargs. Lists. They are lists. You know, you don't use lists in the single call form. And so in the.

There is no ambiguity.

Yeah, there's no.

I understand. Because these are strings here. But they wouldn't necessarily have to be strings. Right? Why can't those things instead of strings. I have got to go, unfortunately. Why. Why can't those things instead of being strings, be expressions, like external objects, for example?

Well, the first thing is a string.

The first thing is.

I know, first thing is a string. But what about the others?

The others? No, but how many of them?

Are there any number?

Yeah, well, there's a function.

The syntax is only thinly not ambiguous.

Yes, I agree. Yes, thinly not ambiguous.

Yes.

Not ambiguous, but only thinly so.

Yes, that's.

That's true.

Look, if you. We could continue workshopping this. This new idea of using a Varog set of lists was. You know, we. We liked it for some reason. To solve a lot of very hard problems. Okay, Maybe I have to rethink it.

See you guys later.

Okay. All right, thanks, everyone.

Bye.