Live CEOing Ep 726: Language Design in the Wolfram Language [WordFrequencyData, DigitSum, and More]

Hi, everyone. Welcome to another Wolfram language design review, part of our incremental language development series. And looks like we have several topics today. Okay. Word frequency data.

So we have Alan and Francisco and their team here. We'd started talking about a few things related to word frequency data last time. And so we wanted to have a checkpoint on a couple of things. The first being that 2 grams are at least partially ready to roll out. I don't know, Francisco or Carlos, if you have any additional information. I know you said server side. It's there, but we still need to kind of hook it up. Is that correct?

Right.

So because we didn't have any design in the work on language side, we haven't created yet the API endpoints for this data, but we already design exactly.

The same as it was for 1 grams.

My assumption was that you'd basically have a string that has two words in it, and that would be a two gram. But I didn't know if there's some other need for markup or. The second point, talking about text element, indicates that we do have some markup language for handling part of speech and other.

Okay, but hold on, hold on, hold on. There's a normalization question, right? I mean, if you've got. I mean, to me, it's like you give the 2 gram and the most relevant thing is the time series. But the absolute normalization would be of all two word sequences in English. How common is three turtles? Right. We just type in turtles of all one word sequences in English. How common are turtles? That make sense?

Yep, that's pretty much what I had assumed.

Okay. And, you know, you should be able to compute, you know, for example, a good example would be word frequency of three turtles versus word frequency of three times word frequency of turtles. Right. And this will be clearly smaller than the. Than the product frequency. But I think the rest of that design should just follow through. Now, if you give a three gram, for example, we should generate a message just saying we don't have three grams. Am I making sense? Right.

I would assume any N gram greater than what we can actually process in the.

Yeah, we just say not. It's not available at this time. Not currently available.

Are there any technical issues with that, Francisco or.

Right. The question was more about maybe the second part of the workforce.

Let's get the first part straightened out. Are we agreed on what to do with two grams?

No.

Right.

The question is how to handle the cases where we want to, for example, set the part of speech.

Hold on a second. I just want to know have we agreed what we're doing with 2 grams? That was the first step here before we get onto this section here.

Right?

Same thing we're already doing, right?

Yes, yes, the result is already clear. Yes.

Okay, fine. So we're straight. We're clear on that. Okay, so now, so we have the markup to be able to do this, is that correct? We have the data to be able to generate this, is that correct?

Yes.

Yeah. And we do currently support getting part of speech variants out of word frequency data. You can.

Okay, but we don't have our word senses, obviously here.

No, this is what we get right now.

In the future. The very notion of a word sense, right. Is, is, you know, really we're talking embedding vectors and things like this. I mean, what is the real thing that we get here? What we really get is we get a distribution of embedding vectors. I believe for every one of these, you know, a word is there. Does that make sense to people? I mean, maybe there's no machine learning folk here. Nobody.

I understand what you're talking about, but I mean, the, the data that we have access to doesn't have support for anything. Okay.

Does not currently. I mean, we could look at getting a much better set of this data from a source that we can discuss. Okay, the question here is we've got entities for. We have entities for word senses, correct? We don't have entities for part of speech variants, is that correct?

No, no. For part of. We have this entity grammatical unit, something we have had.

No, no, no, no, no, no, no.

For the word.

You mean. You mean like an entity that represents on its own green as an adjective?

Yep.

No.

We have senses that we. We do have the sense information and we have. We have word and word sense entities that are not yet exposed really, that are still.

Okay, okay. But we have the word and. Okay, so we could have a word with part of speech as an entity if we wanted to. That's an alternative to this structure here, which is. What is this? Is this a grammatical. What is this? What is this object here?

It's. It's.

It's right above. It's a text element, right? With a string and then hep. Setting a grammatical unit that takes an entity.

I see.

Which is really clunky. I think that we, we borrowed this from. What's the function? Yeah, that's the sentence diagramming stuff.

Right?

Okay, so our choices are this or an entity. I tend to prefer this because I think with the entity we may have words that are non words. Like if I type in nonsense here. What do I get? Does it try and give apart a speech variant or not? Or does it punt on that? Does it only give the raw word in that case? Do you understand my question?

I think so. I mean, I think it depends on what's in the data set. Right. And that the N grams data set has some.

No, I understand, but assume it's astf, you know, whatever. Something that has no does not appear as a word, then I assume it might give frequency zero and then none of these things. Right.

Well, for example, if you try Apple with three Ps, you still get an.

Answer, obviously, because there are misspellings. If I write a word with a bunch of random letters that's long enough, there will be no occurrence in the data set. So what I'm asking is, well, even with Apple with three Ps, do you get these variants or not?

Yes, because you have misspelling with these variants. But in other cases you don't get any data. So if you put up a word that you don't have in the data, you get an MP association.

Clearly we don't have an element here for preposition. Apple never appears, you know, seeming like it's a preposition.

Right? That's right.

Yeah.

I'm not sure Apple ever really qualifies as an adjective, but.

Well, it's down by a factor of 100 from the other uses.

Okay. Apple cheeks.

Apple what?

I don't.

Cheeks.

Yeah. Oh, okay.

You know.

Is there an alternative to this? I don't think. This is obviously silly. I mean, it's at least computably handleable and I can imagine one day being able to have other annotation here, like for example, inflections and so on.

What would the input be for specifying? I mean, I haven't seen. I don't know, the output for 2 grams with part of speech variances. What do we get out of.

I see. Well, how do we make a text element in. In text structure, we would say. Something like this. And we get that. And now what's the input form of that? It's a nested text element.

Yeah.

The current idea for. For handling this was also having a text element with a list of text elements.

I think that's sensible. I don't think you can have much else.

But we don't have any annotation for the outer.

That's correct. For the outer thing. We will not have any annotation.

But like, if you wanted to get 2 grams data for three turtles, then you just take the inner Thing from there, we'd feed that into word frequency data and we'd expect that to work.

Well, that's a separate issue. Making sure that word frequency data can take text elements. That's a good thing to make sure of.

Well, right now, right, that's what the second example under text element in word frequency data shows.

Okay, but the issue is how do you return? In this case, you're returning an association. But what happens when you have. I think what you have to do is just break out all the different cases, assuming the data gives you that. Do you see what I'm saying? So in other words, each thing, it's giving you an outer association of text element constructs with every word tagged with part of speech. So what I'm saying, that seems like the only thing one can reasonably do. Right. And it's also the most useful thing to do. Francisco, you understand what I'm saying?

I'm trying to figure out. Well, I was also looking at text element. The problem is that it doesn't support not having the grammatical units. So maybe that's something we need to improve in text element.

Okay, fine. So, yeah, so that has to be fixed. But do you understand what I was saying? Otherwise, I mean, the way it would look is it would be an association of which I can't imitate it, but I mean, You know, this would be some number and then it would be text structure.

Of.

Of the list of text structures.

Yes, yes, right. Of. Of. Right. Of each. Obviously this isn't real, but the. The annotation here with no structure, sub annotation. And each one is in a text element and that. So we. The outer level, it would look like that.

Yes, I think that will be fine. Yes. We will need to modify text element a bit to support this new notation, but that's fine, I guess.

And the name part of speech variants seems okay to me. Any alternatives to that? No, I don't think so. Great. All right, I think we have a plan here. Let's go on. Ready to go on?

Yep.

Yes, it's clear.

Yeah.

So this is something that's come up a couple of times. The idea of using our word frequency data to create a weighting system for random words.

Yes.

So we think, you know, Francisco and Alan and I discussed this a bit and we think, you know, for the basic case of I want all, or at least a reasonable subset of all words for a given language with their weights, we can basically pre compute that and have that be something that's exactly.

Exactly. All we need to do is to take the words in word list and compute their frequencies according to word frequency data. We don't need to go spelunking for words that are in word frequency data that aren't in word.

Yeah, exactly. And so we'd basically have that pre computed. You know, we could sort them if we want, and that's relatively straightforward. There are a couple of design questions or issues that come up from that. One is if we want to do that for historical cases as well. If we do, it's going to be a lot more complicated.

No, I don't think so. I think we just want frequencies. I, I, I think the idea of, you know, what would this look like in 1830? No, what, when, when old Mr. Morse started doing telegrams, what was the word distribution then? I don't think it's that important. I think if you want that, you can compute it yourself.

That was more or less my stance on it too, but yep, yeah. Basically, if we have one list, we need to generate and maintain and update relatively infrequently. It's pretty easy if we have to do it for every subset of dates. It's complicated.

So look, I think there's an amusing possibility, which just occurs to me, of being able to have word frequency data with a number as its first argument, basically giving the frequencies of the most common and words.

Yep. So that's the second point is basically, is that something that should be syntax for word frequency data, or should we have some sort of word list keyword for it? You know, say, get the n most common words.

Yes, I, I think it would be better to have that as a different function. So, so we can implement that in the server side, of course, the reverse lookup, let's say, of frequencies, but it would be better to have that as a different entry point. So to get, for example, the anything that has a frequency higher than x percent, for example. Right.

Well, I think there's two cases. There is. I want something that has a frequency over 10 to the minus 3 and I want the 200 most frequent words. Those are both two different cases. And I think there's an argument to be made that something like word list or a standalone function would be slightly better suited to marking that up versus overloading word frequency data.

Okay, let's discuss what we need. We need word list, where it's basically, you know, look, if we say word list, right now we just say word list and we have common words as a concept in word list, right?

Yes. Which is a.

So I think what we should do is we should say common words and then maybe something like this, common words comma. Well, look, you know what we should say? We should just have a thing called sorted common words or something like that. Or.

I think the problem with word list is that it doesn't include, for example, inflected words, but word frequency does.

Well, we have to put those together.

We do. We have. You can include inflections with word list. Right.

Okay. So look, the thing we need, the output we need is if we even had this list of words in order of frequency, it would be very useful. And I would even argue possibly, possibly, possibly that we should resort common words, or we could say common words sorted or something like that as a. And we could even have, you know, all words sorted. I don't think we want to overload the frequencies. I mean, if we say word frequency data of cat and dog. Okay, all right, we're getting that. So. I would have thought what we should do is just have a number here. Word frequency data of 20 gives the frequencies of the 20 most common words. See what I'm saying?

I suppose word frequency data of all would be a potential syntax if you wanted to get.

Well, all this is going too far because it's going to be too big, right?

Yeah, we should not support that at all.

Yeah, we don't want to just dump the database.

But it is also the case that you need to cache the information on the current frequency of every word and word list. Right. So all of those are fast lookups.

Yep.

Am I making sense? Yep. So then it's no big deal that you get, you know, the nth most. N most common words, and then you call word frequency data on that list of common words, and then it gives you the list. Right. So it's fine to do it in two steps like that. And if you say, I don't see the use case for, you know, words more common than frequency X, and I think the number of words is small enough, you just generate them all and you just do a cutoff. Do you see what I'm saying?

Yeah. So I think the, the sister design question for this is what the syntax would be in random word to say. I want a random word weighted by frequency.

Okay, great. Let's look at that, because we do have random. No, no, I understand. Let's just finish on the word list. Okay, so obviously, common words sorted we need for all languages, et cetera, et cetera, et cetera, that for which we have word lists. What's happened to the swadesh lists for uncommon languages? Are those in now. Do I need to ask the question again?

No, There are a bunch that are in language data, but I don't think that they've all been added as additional languages in word list.

They really do.

But there's. Yeah, but there's. They. They have been incrementally cleaned up, but nobody has been able to work on cleaning them up for a while.

There's.

There's more. There's a lot of garbage in the. In the text.

Okay. Well, in any case. All right, so. So there's common words sorted, known words sorted. Mm.

I would assume. Stop. Word sorted or adjective sorted is less useful.

But. It's less useful. But I mean, I don't know whether we should support it. The fact that we have that list syntax means that what we could do is we could have a general sorted thing. So we could say adjective sorted, but we could have the shorthand of common words sorted as a single string.

Yeah, I guess that's. Yeah. Then we don't end up doubling the number of keywords. Yeah, I could get behind that.

Okay, let's look at random word. Now. I think we should just say weighted words or weighted random word, weighted comma.

N.

Okay. I don't see any. An objection to that, do you?

No, I just. I wanted to make sure if we were using, you know, if we're talking about weighted words in two different functions, that we're using common terms. But they are. The weighting matters. In random word, it doesn't really matter. In word list, the order matters.

Yeah, it's really sorted. And I think what we want to do is.

And. And wait, so we only have a unique. So weighted words means everything.

No, you could do this as well. I think. I think you should be able to do that as well. I'm saying that's like type and pause.

Right. But what about common words and known words?

It doesn't matter because you're not going to generate the difference is. Is so far in the tail. I mean, yes, if you want to do that, you say common words, comma, weighted known words come awaited. See what I'm saying?

And just weighted words is equivalent to one of those or neither of those.

I suspect it's known words weighted.

Okay.

But I don't think it's ever going to matter because I think you're so far.

That should be like not giving any argument. Right. Like random words without any argument.

Maybe.

So the keyboard for that.

Well, right now we've got a pseudo random commonly used word. That's what we say.

Yeah. Common words is the default.

And I think that's a good thing to do. I do not think we want to weight them there. The use case and we've got many, many examples. It's a, it's a thing, you know, this is the. How well do you know words in English? Right? You can estimate how well you know English by doing that as opposed to the weighted version in which, you know, chronograph will not appear. Do you see what I'm saying?

Unless you're generating really large sets. Yeah, yeah.

It's very unlikely to occur. Right, yeah. This, this thing, if you generate 10 weighted words, they're going to be the and a and things like that most of the time.

They're unlikely to break the top 100.

Right.

Now an interesting question. Yeah, that's okay. I mean, so if you want to generate a plausible sentence word by word. Actually we've got two, two points to make here. If you want to generate a plausible sentence word by word, you do it by saying random word. You know, weighted comma, weighted words comma one or just weighted. We could just have weighted there possibly like that.

I kind of like including words just because we have common words and known words instead of common and known.

Right, okay, fine. Okay. So, so then, you know, you say random word weighted words comma 1 and you just say table of that. Am I making sense? Right.

But you're unlikely to get a sensible sentence from it. You're just.

Yeah, right, right, right. Now one thing that happened that we did for that chat GBT explainer piece that I wrote is we actually did work out what the. Up to five gram word. No, up to three gram, four gram, I forget word, multi word, you know, probabilities were, if you understand what I'm saying.

Right. And that's why I don't like much this notation of world frequency data a number because that wouldn't work fine. For example, for 2 grams. So how will be the notation for.

I have to figure that out.

Right. That's something that will be.

The single word case is much more common. Right. So for 2 grams, I mean we can have some notation like, like that number arrow, the gramminess. It depends on how far we're going to be able to get with different grams. Do you see what I'm saying? If we only get to 2 grams, that's one thing. If we pre compute it for three grams and so on, that's another thing. Right.

Right.

Three grams is kind of very large, so.

No, no, no, no, no, no, no. Not this case of it. Not Word frequency data. When, when it's the most common three grams, it obviously by definition isn't large. Do you see what I'm saying?

Oh, I see it's already filtered.

Yeah, it's obviously not large because it's, it's. Okay, so I had a thing here. Okay, same probability. Okay, so I think this is as far as we managed to get. I think this was a non trivial computation. Am I making sense? Hi guys, are you still there? Yep, yep. Please, please, please make sure you actually respond. If you, if nobody responds, I have no idea whether anybody could even hear what was being said.

Yep, yeah, we got it. I was just reading.

Okay, so the answer is we're not going to be able to get easily beyond. You know, I don't think we should worry about 2 gram most common 2 gram frequencies. I think we should just have word frequency data of a single number meaning the most common N words. Am I making sense?

Yeah.

Okay, I'm going to assume the answer is yes, because if you say nothing, it's just going to be yes.

People said yes. You need to listen.

Okay, all right, let's move on. Anything more on this topic?

Marching orders.

Right.

These are two function or two sets of functions that were suggested by Chat GPT, which we don't have. We have alphabets, we have writing scripts, we have data about them. We don't actually have the data broken into vowels and consonants, which is actually.

I want to say that we have that, that property internally in the Alphabet. I know that never expose.

I know that.

Because it was not very populated, but maybe we can work on that to start exposing.

So do we want a function vowels of English or do we want Alphabet comma vowels?

I mean the notions of vowels versus.

Consonants is kind of complicated.

I mean Alphabet itself is also complicated because we include things like syllabaries inside of it that aren't technically alphabets.

But what's property?

Any properties from property is any property from the entity value.

You know what really pisses me off? I'm sorry. You know, how many times have we gone through, you can't have something listed here without giving it, giving the information on what those properties can be here. Where do you think? Where is that being documented?

Nope, you're right.

It's just infuriating. Just infuriating. We go through this, you know, how difficult is it to follow that rule?

Well, it's easy when you're writing new documentation, but given that our style has changed and we have 6,000 pages, you.

Know, who do you think that style changed okay, whatever. We should fix this, please.

Yep, I will make sure that gets done.

And then it seems to me that the reasonable place to put this vowels thing is that's the second argument. Does that make sense? I. I don't know. You know, this notion of vowels is a little bit complicated in some languages, is it not?

Right. And I think that will be as easy as exposing the property to. To entity value, and alpha will inherit that.

Okay, this has nothing to do with entity value. This is a function Alphabet, right?

No, no, but Alphabet uses entity value.

I don't care. What matters is what the users will see, which is this function. Right. And. And what you're telling me is that if I say something like, you know, Spanish vowels, is that going to work or is that not going to work?

Not yet, but it's easy to make it work.

Yeah. How come it doesn't work? Didn't we go through this, like, 10 years ago? What happened?

I don't think so. I mean, we have a handful of properties that are in Dev. In.

In.

Wait, this is coming from language data, right?

Or.

Oh, yeah, that's what.

That's what.

That's what Francisco is saying. There are properties that can be left, and then we can.

How long have they been there?

Would have to go back and look. I mean, this is one of those domains we.

It's like a decade or something.

I don't think it's a decade.

Okay, but let's get this exposed, please. Reasonable. Any objections?

Well, I mean, my main concern is that there are plenty of languages which don't have distinct vowels and consonants. Right. They just have basic syllables. So, I mean.

Yeah, Japanese doesn't have. It doesn't have vowels. All syllables have an attachment.

So what are we going to do with those languages? What are we going to do when you say Alphabet block, comma, vowel, probably.

Say missing, not applicable.

Right, Fine. That seems fine to me.

I mean, if you say Alphabet of.

Okay, so, for example, in Japanese, isn't.

Hiragana or katakana would work. H I.

Okay, so this would work. Katakana would work. Yep. Okay.

And each of those is a syllable, essentially. Some of them are standalone vowels, but many of them are consonant vowel pairs.

Okay, so when you ask for vowels in a language in which those things are syllables, should you get the vowel only syllables?

You could, but then we'd probably. For symmetry, would we want to give the consonant vowel pairs for consonants?

You mean, is it necessary that the union of constants and vowels is the whole Alphabet? I don't know. I'M sure.

I don't think that's true for every language.

No.

What do you mean?

Are they necessarily disjoint? I don't know.

No, I don't.

There are many letters that are used as vowels and as consonants and.

Actually, yeah, yeah, I was, I was going to say. Right, so is Y a consonant or a vowel or both?

Yeah, both.

We have them in both properties. Yes.

Okay, fine. Then you've already answered the question. So we pick up. So there are languages where it's well defined and there are things which are in the consonant list, things which are in the vowel list. Those may overlap. There may be things that are neither vowels nor consonants. Like some of these languages that have, I don't know, like softening letters and things.

Right. That's a different category of letters.

Right, right. Which probably have a name. So my question, in, like the Unicode standard, is there a designation of vowels and consonants? No, I doubt it. Fine. But there are probably things which make a sound versus not make a sound. I forget what that's called in phonology.

So there are things that are critics as symbols in Unicode.

That's right.

But.

Unicode is about orthography, not phonetics, right?

Yeah, yes, that's correct.

So I don't think it'll have this sort of information in it.

Right, but something like this, that's a perfectly valid vowel in languages that have that. Correct?

Well, it's some. It's somewhat. It's somewhat language dependent. And in some languages, Right, it's a compound. I mean, in some languages, right, it's a compound character thing. And in others, it's considered a unique letter.

That's right, yes.

So what are we going to do with this? Is it going to be considered a vowel in, let's say, German or something? Or not? I don't know if that's even a form in German. I know a is. Do you understand what I'm saying? Okay. In some language that somebody here speaks. Can we discuss that case? Pick a case.

So. So you don't, you don't get that as a vowel itself because it's not in the. For example. Yeah. You know in Spanish you have the accents, right? The accents are not part of the basic Alphabet.

But n with a, whatever that is.

Well, that's a different letter. That's an nea.

Enya.

Right.

So that's a letter by itself. So it's not a letter with a diacritic.

Okay, fine. But if I say Alphabet or German, for example.

You don't get the um loads.

Okay. All Right. Well, that's an answer then. That's not a vowel. Then.

What do you get for he?

Yeah.

Okay, so you get. Yeah, so you get the. That's a yud. That's the 10th letter, which, you know, is. Roughly corresponds to a Y or an.

E. So it's not just, it's not just a softening. I mean, like in. I think, isn't it the case that, like in Russian maybe somebody isn't. Isn't that creature there, for example, just a. Some kind of softening thing?

Yes, the. The fourth from the last. Yes, that's it.

Okay.

Right. So, yeah, so yud can be. It can serve orthographic purposes where it's merely completing, you know, other. It's denoting the end of consonants, but it can also form a Y sound, which is why I say it's sort of like the Y of Hebrew.

Okay, so here you are considering these. I don't know if anybody here knows this language, but I mean, those are considered separate letters, like the enya thing in Spanish. Is that correct?

Yeah, I guess we can ask Gosha.

And this is something we created for either Gosha.

There are at least down to it.

Yes, I guess we could ask both. But I was actually just talking about. About the A with the little thing next to it with. With her this week because Ike Swantek is, you know, has that letter in her name.

Going to say, for example, Alphabet. Korean will give you all of the sub characters that make up consonants, but they're never written independently or the, the sub characters that make.

But look at this case.

And, and, and, you know, I don't know if. I mean, some of the letters in Hebrew are sort of vowel like. But the actual vowels are all diacritics, which are not listed here.

Right. Okay. So look, the basic principle has got to be if it's in. If it's not an Alphabet, it's not a. We're not going to consider it a vowel or a consonant sub case of Alphabet because it wasn't in Alphabet to begin with. If it is an Alphabet, which, for example, the A umlaut seems to be in Alphabet for Swedish but not for German.

Yes, that's incorrect.

Yeah, the. The German umlaut is equivalent to just putting in the letter E after the vowel. So a umlaut is equivalent to ae.

Right, but. So, but in Swedish, presumably that's not the case. But any case, we would. Then in Swedish we would say the vowels in Swedish include aumlaut, for example, but in German, we would say the vowels don't include, just include things which are sub, sub cases of the Alphabet list. Is that correct?

Yeah, I think that's consistent, right?

Yes.

Okay, and so then go ahead.

I. I'm just curious. In Hebrew, did, did we list any. Anything for vowels since some of the letters are a little bit vowel like? Or is that empty because all the actual vowels are, are diacritics.

Can somebody answer that? I mean, I don't have the data to answer that.

Yeah, yeah, that was a question directed at Francisco if he happened to know.

I can look it up right now, but yeah, check if we later for.

Yeah, please look it up for that. German and Polish. Sorry, German and Swedish. Just to check that we are somewhat consistent here. Okay, but so that's, that's what we'll put it in as an Alphabet property and it's also a property in language data. Make sense? Yep. Okay. Syllable count.

Right. So this is another chat GPT suggested function. We don't have syllables in word data. We do have properties of hyphenation, but it doesn't 100% correspond and the word data doesn't include inflected forms.

So.

We could basically just say, well, hyphenation is close enough. Or we could have something that, you know, tries to, to build off intelligent rules. And you know, I know there are, you know, if you.

So I mean, the function syllables, which takes a word and returns the list of syllables seems like a possibility. Right, so actually have a. Just like we have. I mean, you know, actually the name would probably be text syllables. Just like we have text word. Like that. For Hebrew and German. We don't have syllable data. We don't have the validator.

Well, we need to create that from different sources. So yes, that's okay.

But for three dish, it does include all those diacritic marked.

Yes. Which are part of the. Their basic asset.

That's right. Yeah.

Okay. I think that's the convention we should follow. And we were following already.

Yeah. Okay, great. All right. Text syllables.

I still have a couple more examples I just emailed you in a separate notebook because they didn't end up in this copy of the Agenda for some reason.

But.

Yeah, I think being able to get the syllables is helpful. You know, we can have that based on the hyphenation system that is in word data. There's various bits of text analysis that use things like syllable frequency.

Wait a minute, you're sending me this?

Yeah, this is just a subsection here that shows an application for like grade level reading analysis. This is the toy function I put together for syllable count and the difference between hyphenation. But the. The Flesh Kincaid counting system is basically a way of looking at both word and syllable frequency inside a text to effectively associate a reading level with text arbitrarily.

Well, we have word count. Yep. But I think it's. Isn't it more useful potentially to have a text syllables that breaks? You know, that makes a list of lists where every word is put in a sub list with. With syllables.

Yeah.

I'm trying to think if there's any other markup structure that makes sense, but I think a list is probably fine. For each individual word containing the syllable subsets.

Then we need to have another property that's the word broken into syllables. But because what you were saying, Nick, what we have is the pronunciation.

But don't we have that in ipa there's pronunciation. I think there's also hyphenation. I think there's a property for word, and it's a typhonation that is broken up basically into two parts or N parts for n syllables.

What would it do for words that aren't in the list?

What would.

Which do for the.

The text syllables.

For like nonsense strings or.

Yeah, yeah.

Yeah.

Hyphenation gives exactly what we're talking about here.

That's an entity property or something you're saying, right?

Yeah. Right.

So for words we don't know about.

Right?

No, I'm sorry, I was just, just. There was a property for known words.

There's a function repository, syllables thing that Mark Greenberg made that's heuristic based. That seems to do a good job for English. If you want something to look at. If I remember correctly, it's just a bunch of replacement rules or pattern rules.

More or less a more decorated version of the syllable count version.

Yeah, but it actually splits instead of.

Just count, instead of just counting.

Yeah, yeah. Okay, guys, so just, I mean, do we basically agree that we want to function like text syllables?

Yeah. I think Bob's point was what do we do with something that doesn't have syllables? Or. Or we can't parse, you know, I.

Just, you know, proper nouns.

Right.

Yeah.

You got to be able to handle things. New words we've never seen before.

We have to have rules. And if we can't do it, we just put the thing as a single lump.

Right. It's not an easy problem.

Yeah, yeah. I mean, I have to believe in the modern world of machine learning that this is the kind of thing that should be doable. Be a good summer school project, actually.

Okay, yeah, we can. We can always run it by the ML group too, and see if there's anything to do with.

Add it to my list here.

I don't imagine the new big LLMs will be helpful since tokens are.

Yeah, there's a messy story with tokens there.

Right.

But there might be other models.

Right.

I'm sure you can just train it from knowing, you know, from knowing how a lot of words are broken up into syllables. Right.

You could use a basic classifier, probably.

And it may not know for sure. I mean, you know, there are people, there are names where, you know, you just don't know how it's pronounced until you hear the person say it.

Right.

Okay, let's continue.

So, following our discussion last time, Danny went ahead and implemented digitsum based on the spec we have. Ian, Danny and I looked at it a little bit, along with some other people. There's a futurized ref page we can look at, and then there are a couple of design questions that come about from that.

I think this is fine. The third argument is kind of amusing. What is digital root?

So there's a link there, but it is basically the leading value on a digit sum.

What do you mean by that?

So it. It's the highest order digit from a digit sum, regardless of the base, because.

It was the point of digit sum.

Sorry, what was that, Ian? I thought that it was the fixed point of digisum.

That probably makes more sense.

It's like casting out nines.

Yeah, it's a general case of. Yeah, casting sounds like a lovely wfoll.

Okay.

Yeah, I'd seen it come up associated with digitsum fairly frequently, so I figured it was worth it.

I think former content guy Chris Stover already put it in the function repository, if I remember correctly.

Should digit sum support mixed radix? I. I don't know. That seems awfully obscure. What about, you know, Fibonacci base and stuff like that? I mean, it just seems awfully obscure. Does that.

Well, we support. I mean, the argument in favor is that we support it in things like.

You know, digits, and so why not support it? If it's easier to support, let's support it. Is it easy? I guess we. We do integer digits in the mixed.

Radix and then sum them.

Okay.

These are last couple things in case we had a little bit of extra time. Just some more miscellaneous GPT suggested functions that are reasonable to implement.

I think Pro range is pretty silly.

Yeah, it seems pretty trivial. I mean.

And I'm not even sure it's the right thing. I mean, I think it will be more like I give you two primes or I give you two numbers, right? And I guess this is gonna give you two numbers and you give me primes between them, right?

I mean, isn't this just prime of nn, right?

Or.

Or some such.

No, it isn't. Prime of nm. Look, look, it's. It's the value. It's the.

I guess. I guess this stuff. I guess that definition is. Yes, just doing the sword. But one might have expected just from the name prime range to. To it to be.

Well, whatever.

But I think it's.

What does this. Does this do that? Okay, What, the other way around.

I think you need a large secondary.

Yeah.

Just use 2 to 1000 or something like that.

Right.

So it's going up by powers of 10. Or I can say go up by powers of 2.

Yep.

Okay. I mean, put prime range in the function repository at least it doesn't strike me as.

All right.

Unless I'm missing something, it doesn't seem like the kind of thing.

So this is an operation which could be useful, but I think it needs a generalization. It's a bit like an interpolator. We could have two points and then we say the fraction of the distance from the first, for example.

Well, what is that for an ordinary line? I mean.

There is this concept of lerp, L, E, R, P and. And. And it's spherical version.

What is lerp?

So this is a linear interpolator and there is this standard name. And then there is the sl erp slerp for the spherical version, you give two points and it gives like 0.4 of the distance to the first.

I think we have a function that does this, but I've now forgotten its name, which is a really bad sign.

Blend or.

Yeah, but it's like blend, but it isn't blend.

So blend does it for colors. But I don't know the general.

I mean, this is a very useful function to be able to get. Maybe we don't have it. I mean.

Rescale is similar.

It's similar, but isn't the same thing? Yeah, Look, I've wanted this function a zillion times. Where you. Where you go, I've got one point, I've got another point. I want to go a fraction, you know, x of the way from one point to the other. I've never heard the term. Look, maybe I'm just out of date. It seems to be mostly related to the structure produced by larvae of psyllid insects.

So I. I added the Wikipedia page 4 for the spherical version.

Yeah, This does seem like a useful function. Look, even in this linear case, this is definitely useful. I mean, it's. It's obviously easy to do. It's x times the first thing plus one minus X times the second thing, Right? That's the definition, right? Yes. But the spherical case is probably not so easy. Is that true?

It is relatively easy. You just need the angle between the two points.

Well, it's easy depending on what input you're given.

What do you mean? You're given two theta phi values, right?

Yep.

Right.

Isn't there a separate geo slurp and spherical slope then.

Your direction?

Well, no, it's not geodirection.

No, no, it's. It's more like a geo interpolate, like a geoposition interpolate or something like that.

Like geointermediate given points P1 and P2. I don't really like the name geointerpolate because it suggests we're dealing with interpolation. True interpolation, which is more like make a. Make a line from one to the other. I mean, I could imagine. I mean, it's basically this. Is that correct?

Yes. I'm not sure whether perhaps we would want a list of points in the first argument rather than to separate. I'm not sure.

I don't think so, because I think. I mean, how do we specify the X value as a second argument. Yeah, I know that, but it's not a single X value. There are seven points there. What do we compute?

Yes. No, I wasn't saying a list of arbitrary length. I'm just saying that the pair. It says the first two arguments look too similar and too different from the second from the third.

Oh, I see, you're just suggesting that it look like this, right? Possibly, yes. Question. What other functions have done. I don't disagree that that's probably right.

The other function we. We have, which is similar. Geodestination. Geodestination might take the X somewhere.

I don't understand that. Geodestination. So direction. That's not right. But this is.

So just destination would take the first point and say, move such a distance.

To the second point to the second. Yes.

So perhaps there is a way of indicating.

Well, sorry, look at this. This has multiple. This would suggest another syntax which is x1, x2 here.

Right.

To give multiple intermediate points. Yep. I don't think we should try and merge these together because I think that this one, Although the XI could be greater than one true or not true.

Yes, one can extrapolate, certainly, or even negative.

Well, maybe they should be combined. Well, but D is an actual distance here. It's not a fractional.

Right, right.

So I don't think that. That doesn't seem promising.

So I was suggesting that perhaps there is an extension that you can indicate to geodestination that you want a fraction of the operation that it currently does.

Yeah, but look, what about the question of I'm going from A to B, but I only want to go 100 miles along that path?

Right, yeah, that's another possibility.

I mean, it kind of makes it seem like we want a symbolic thing that represents a path, which we've talked about before, a geopath type thing. And then we want to be able to take the geopath and say we want an intermediate point or a, you know, a geopath point, so to speak.

Right. I mean, that question exists already in other primitives. Like we have a spline curve. How do I get some sort of fractional point of a spline?

Yes, that's a good question. Well, yeah, these are. That's a good question. I mean, let's look at. Take a look at that. That is not very good auto completion there. Oh, God. So the question is, what do you. How do you get a point, some distance along a Bezier curve?

Right.

How do you get a point, some distance along any curve? I mean, we can define a curve just with sort of implicit computational geometry stuff. I think this is an interesting general question and I think we need to something. It's almost like given a symbolic thing like this bezier function. Well, wait a minute. What's this? This is finding a point on the curve. I don't know what that parameterization. I think this is. This requires more research. All right, we should probably wrap up here, but let's bring this back. This is a good thing. Let's try and understand this.

Okay?

Yeah.

All right. Wrap up for now here. Okay. Anything else? Nope. Okay, should be good. See you later. Thanks, guys. Bye. Bye.