Athens Community Call 5/23/2021: Fluree

All right, most anticipated demo of the year right here. All right, so let's get started. So the context for this is Blurry is a really cool graph database, semantic, backed by immutable ledger, has a bunch of query languages. So I took some notes on it before, chat on Twitter, on Discord, and they gave us a demo a week ago. Brian and Trey, and thought it was really cool technology, wanted to learn more. I thought it would be great if we did a second demo with the community, and Brian and Trey were super, super nice with their time and decided they would be open to that. Flurry, Matei said he wanted to moderate this. He has been a big fan of Flurry for a long time. He put it on my map originally, and so I'll let Matei moderate most of this stuff. I won't say too much more after this, hopefully. But yeah, if you have questions in the chat, post them in the chat. If you have questions, you could probably just say something too during the conversation. But really excited for this whole thing. And I guess I'll hand it off to Matei at this point.

Cool. Yeah, thanks, Jeff. I think you pretty much covered it. You know, Flare is a semantic in-memory graph database backed by immutable ledger. It's built with Clojure, it's open source, and this is part 2 of the demo. And I'm gonna go ahead and post the first part, the link to the one that you did with Brian and Trey, into the meeting chat. And then as for what's actually gonna be covered, I'll, I'll flip it over to Brian to talk about that. But just before As Jeff mentioned, if everyone can just kind of keep on mute and post any questions that you have into the meeting chat and, you know, upvote each other's questions if you want to make sure one of them gets answered. But it's mostly casual, so if there is something you want to jump in with, go ahead. Just want to kind of keep things clean. So yeah, I'll go ahead and pass it over to Brian if you want to give us a little overview of what this demo will be about.

Great, sounds good. Just confirm you can still hear me.

Yep.

Okay, good.

Confirming, confirming that someone is recording, or more than one person.

I am recording. I don't know if anyone else is. All

right, great.

Well, I still, still apologize for the video. I've tried a bunch of settings here, but it looks like I can at least share my screen, which will be the important part today. The goal here is to just give a good introduction to some of Flurry's capabilities. One of the things that we were able to do in the prior meeting that Jeff referenced is just do, you know, some simple transactions and some simple queries. The goal here is to extend it and demonstrate a few of the other capabilities. It still won't be completely in depth. I think there's a lot there, maybe your questions. will allow us to dive a little bit deeper. I don't know all the areas that will be of interest to this group, if any areas, which is fine too. So I'm gonna try and cover a lot of things with a broad brush. I'm gonna focus much more in Clojure world, 'cause I believe most everyone here is, you know, Clojure. Lots of times when we're our focus with Flurry, even though we're obviously huge Closure fans because we've developed it in Closure, our focus was not primarily Closure developers. It was more just a a backend data platform targeting probably a lot more even JavaScript than Closure. And so yeah, I mean that's. That's kind of a brief background. I have been in enterprise software for over 20 years now, built several companies in this, and Flurry is really a passion for me. I think I've worked on it now for 5 years, using all my own money for most of that 5 years to focus on getting a product that I was really happy with that could do the things I thought data platform would need to do. I'll start out by saying where Flurry's focus is. It's in what we call data-centric applications. So that's opposed to application-centric, which means, you know, you're trying to build an app and the data is kind of an artifact of your app, typically designed and developed by developers. In the case of Flurry, we really want data to be the first-class citizen. Doesn't have to be, you can still use it like a normal database if you want to, but it really shines in what we call a many-to-many environment, meaning there's many transactors, many parties transacting data, and many parties consuming data. And this is vastly different than a traditional database, which lives typically in a one-to-one environment, which is the app server in front of it is the only thing that can transact to it, and the app server in front of it is the only thing that can query it. And if you want to expose any of the data outside of that realm, you know, you're building APIs and you're doing some of these other things. So Flurry can actually, in fact I'll demonstrate it here today, can be run with quite literally no app server. It can embed data permissions directly into the data tier, expose multiple query interfaces, no app server required. Now obviously a typical application is gonna have some sort of app server needs, whether that's sending users registration emails or things like that. Flurry's not gonna do that sort of thing. But as it relates to data, querying data, getting access to data, getting data to frontend web browsers, et cetera, there really is no app server required. So with that, I know we don't have a ton of time, so I will jump in and just really try and do the basics here. Just let me know if you have any questions that do come up. I know Matej said to use the chat, but also happy to get interrupted.

Cool. And just want to make sure, are you— okay, there we go. You're sharing the screen now. All right.

All right. Can everyone see IntelliJ? Yep. Okay, perfect. All right, I'm gonna actually try and move Discord over here, so I'm, uh, it's my first time presenting in Discord, so. All right, so again, uh, the focus here is very Clojure, uh, um, so with that in mind, I'm going to be using IntelliJ here to do a lot of, uh, what we're building with Clojure. I'm, I'm gonna quickly do a recap of what we already showed Jeff in the last meeting. So I know some of you might have seen that on the video, some of you might not have. I won't spend much time there in case you have seen it, but it'll either be new or a refresher. So this is just a standard Clojure project. If you look at the deps.eden, I actually have a local development one that I'm using, but you can just use the Clojure version of Flurry the library. Flurry in this case is running as a server. The server is actually running at this port. I guess this is one important thing to point out is that Flurry by its nature, the backend or the ledger backplane can be clustered and has fault tolerance built directly into it. So I might actually be running 3 ledger servers and they're all participating in Raft consensus. And, you know, they might all be at different IP addresses, and that would be a perfectly fine thing to do. If a server goes down, you have zero downtime. It'll automatically rebalance. All the data's copied across all the backplane. So you have a full redundant system. And if you're familiar, or if you've used like HashiCorp's Consul or tools like that, then you're familiar with Raft. But generally you wanna run 3 servers or 5 servers depending on the amount of redundancy you want. A nice thing about this is that Raft, we actually have it— let me— right here in case anyone's interested. We actually have the complete Raft library that we have built, which sits as a library. So if any of you have had to run things like ZooKeeper to maintain consensus across apps or anything like that, I would encourage you to try this Raft library. You can literally embed the equivalent of ZooKeeper, you know, right into your application if you want. That's what Flurry's using kind of under the hood in the consensus. So I already created the connection here in the REPL, but what we'll do is we're gonna create a new ledger or a new database, if you will, and I'm gonna call this Athens test 3. So here I just did a def, I haven't created it yet, but I'll go ahead and use the flurry.db.api namespace, which has all the kind of Clojure APIs in it. And I'll go ahead and create that ledger using the connection and that name. And once again, the servers reach consensus, meaning at least 2 of them, in this case, you know, if there were 3, I'm actually just running 1, 2 of them would have to acknowledge and durably store all the information related to the new ledger before you ever got the response, but you will get the response once Consensus has been achieved. And so the first thing we're going to do is create a schema. I think you're all pretty familiar with this. I actually put the schema down here, but I'll walk through it real quick. So what we're going to do is we're basically going to create a couple buckets. We call these collections to store things in. This is, I guess, inspired by Athens. It's certainly not your exact schema that you're using right now, but we're going to create a bucket called block. Not to be confused with Flurry blocks, which we'll also talk about, but we're also gonna create this bucket called org. And there's already a couple sort of core schema elements in Flurry when you just create it, but then these are gonna be all of what we call predicate and not just us. So we very much follow the RDF standard. So I'm sure a lot of you are probably familiar with RDF, or if you're not, certainly you've probably heard of different triple stores. And TripleStore is an RDF database, so semantic graph database. And in this case, we're going to be giving these predicates, these columns, if you will, names. Like for example, this one's a ref and this is going to point from a user record to the blocks that they've authored. So it's a ref multi-true, meaning multi-cardinality. And we've got this restrict collection in here, which is going to make sure that anything you point this reference into is also So of type block, that becomes important. It's not mandatory, but if you're gonna use GraphQL, for example, Flurry natively exposes a GraphQL interface. If you're gonna use GraphQL, like GraphQL is strictly typed. So anytime there's a reference, for example, in GraphQL, it has to be one distinct type that that reference is, but it would be optional if you didn't want something like GraphQL. So it depends how strongly typed you want it. In here, we're gonna create, you know, an email and some other block string, which I think is what you currently use, etc. So make sure we have this, and we'll go ahead and transact that, the schema. And once we have the schema, we can start putting some data in. So a few of these things I'm gonna do are kind of basics. And I think it's important if you try and design a better mousetrap that you make sure it can catch mice, you know? I don't know if you've heard that adage. So while Flurry can do some interesting things, it can run decentralized, it can do a lot of federation, it's got blockchain componentry in it, if you can't run it like a normal database, then, you know, we haven't hit the bar. So what I'm showing you right now is forget about all of that stuff. It's kind of sitting in the background waiting for you to use it if you want it. You don't have to use it. If you wanna run this just like a backend typical database with an app in front of it, you really don't even need to know about any of those things if you don't want. So here's the result of our schema transaction. And what we're gonna see is what we call flakes here. Flakes are basically, you know, an extension of RDF data, an extension of triples. I think a lot of you are familiar with some of the, you know, Clojure technologies out there that, are based on Datalog, et cetera. So you'd be kind of familiar with some of the same concept, but basically we're writing out all of the deltas, all of the changes. Importantly here, everything is cryptographically signed, everything can be provable, but I didn't set any of that up. And that's because it's our opinion, if you don't care about that, we're kind of, we end up generating basically a default private key that's used to sign transactions. And if you care about it later, you can sort of open this capability up But if you don't care about it, it's just there. It's in the background. But all of these blocks are going to have a hash. They're going to have signatures around the original transaction, which allows you to prove identity. So there's all this cryptographic proof that exists if you ever did want to go back and get it again. It stays out of your way if you don't care about it. Another thing that we output in Flurry, and this is important because it is designed to be a many-to-many environment, like I said, which means That means you don't always have control if there's many different entities updating data on usage. We have this concept we call fuel, and everything you do in Flurry, every query reports fuel, every transaction reports fuel, and fuel is the amount of work that the computer had to do to fulfill the query. And we'll do, you know, some real high-level smart function stuff here, but for example, as you're running code inside the database, it's consuming fuel. You know, the operations are consuming fuel. As it's touching data, it's consuming fuel. So in this case, this transaction consumed, you know, a little over 3,000 fuel, took about 10 milliseconds for a brand new ledger. So, you know, this is running on the JVM and it's not warmed up at this point. Okay, we got our train, we got our schema in there. So let's just go ahead and create some users. Feel free to jump in if there's any questions. Okay. So here's a user transaction, and I'll just explain a little bit what's here. So we use this _ID as a unique identifier. Ultimately, the _ID is going to point to these long integers, which are going to be permanent identifiers within the ledger. This, in this case, is telling us what sort of collection or class to put this data in. So we're going to say, you know, put this in the user collection. And here's just some information. So Trey, Trey's email, you know, an ID stuck on it. And importantly, Flurry has this ability to have and manage multiple identities, multiple auth records for an individual user. In fact, every auth record can even have different permissions, and this can even start allowing things like, you know, if I wanted LinkedIn to be able to get access about me, I could actually set up an auth record because now LinkedIn is, you know, authenticating as me, but I can set up specific permissions that says, you know, LinkedIn can only see these 2 fields. Or if I log in, you know, I have a different auth record that I'm using and I might have my full capability and permission. You can also do things like key rotation really easily if you are using private keys when you have multiple auth records. You know, every time you transact, for example, if you wanted to, you could delete your old auth record and put a new auth record in if you have the permission to modify your auth record. And all these permissions, you know, it's up to you to set up how you want. Again, we'll cover those here in a minute. But so here's a few users. So we'll go ahead and do that. And we're actually gonna put in a few cryptographic keys here. So this ID is really a derivative and I think I have 'em here. Yeah, so Trey, here's a public key, here's a private key, and here's the auth. So the auth is the piece that we're putting in here. And using this private key now, I can actually interact with Flurry by cryptographically signing queries, cryptographically signing transactions, and I need no identity server in between. So Flurry will have the ability to, to figure out that, you know, anything signed with this private key must have come with this auth. And now because we transacted it here, we know that this auth is associated with Trey. So we have identity based on the cryptographic signatures. Again, only if you want it. We're kind of running Flurry here in, in root mode or like you'd run a typical database. But we'll show a little bit of this as we get on. So we'll go ahead and put it in here now. So I already did that. So let's transact users. All right, so we have users transacted in it. Good time to issue a couple simple queries. So databases are immutable. So in this case, we're gonna just create a database. We're gonna use the connection and the ledger to create one. If we don't specify time, It's gonna use the latest database that's available. And so I thought we would start out because everything in Flurry is really at a core based on the W3C standards. SPARQL is the query language of W3C semantic graph. So I figured we'd start out with a SPARQL query. So this will probably look very, very familiar to you. With SPARQL, any variables you always prepend with a question mark. You specify sort of the predicates, or probably in your language, entities. And then you can also bind variables really anywhere in here. And then you're gonna produce these variables in your select statement. And of course, you can do aggregates, you can do all kinds of functions in here. This is a very simple query. It's saying basically find me all the subjects that have a user email binded to this variable, and that's what I want to select out. And we will see based on those 3 users that we put in, here's our response right here. So we've got Jeff, Brian, and Trey in there. Now we have a way of querying Flurry that takes Sparkle and it puts it in a data structure. So this is the identical Query is a Sparkle query, but at this point we're actually using a data structure to represent it. And I think a lot of you are familiar, and I won't go into why we see value in representing queries as data structures. Importantly, this can be JSON. In this case, I'm representing it in Eden, but identical query should have the identical results. And there you go. And then I thought, just because I know you're using DataScript. you will see that the difference between the Flurry query here and the difference between the DataScript query here is there's virtually no difference. And that is because this is entirely based on Sparkle. So Sparkle was kind of the origin of, I think, much of this thoughts, the thoughts that went into this. So all these queries are sort of the same thing. We do allow you, if you want to crawl the graph, you can certainly do that. So we're going to use the same query that we just did here. Instead of binding that email address, which we don't really care about, we're just going to have a blank sort of node here. So this is going to find all the entities or all the subjects using RDF vocabulary that have a user email. And here, because we wrap these in a map form, We're going to say bind that subject and then crawl the graph out. And I do want to point out that you can use keywords here. So we're using a lot of strings, but you can use keywords. Here's a good example of using keywords. So we're going to crawl all the users. We're going to pull their email, and then we're going to crawl all of their auth records, and we're going to pull out their auth record ID. And of course, we get this nice graph that returns. So again, you're probably pretty familiar with those concepts. The whole point of all that is, again, kind of just showing that all of the basics, all that core, are there with Flurry and ready to use if you want to. Okay, any questions?

I've been— a couple questions in the chat, Brian. I don't know if you wanna go back to some of those, you wanna keep doing the demo and circle back to them at the end.

Maybe Matei, I know you were gonna moderate it, so you tell me. I'm looking to where the chat is. Is this in a specific room?

So it should be in the meeting chat. And I actually, I just realized I had my— I muted, tried to mute everything for the recording, so I wasn't seeing the questions that were being posted here. Trey, do you see any that stand out that you think should be tackled first?

Do you wanna delve into a little bit more on the LinkedIn example you were using, Brian, for the author? A couple questions about it, a little bit more advanced. Authorization.

Yeah, I mean, the whole idea is that because, you know, this is a data platform that can exist like on the public internet, it doesn't have to exist behind a secured web server. We call this, or an app server, we call this, you know, Flurry has this ability, we say that data can defend itself. So you can only see things you have permission to, you can only transact things you have permission to, and we'll get into how to do that. Because we need this sense of identity then baked into the core platform, it really makes sense to offer a lot of flexibility in how we validate users. And so the idea that multiple auth records are there are simply that, you know, a user might maintain multiple private keys. That's a fine thing to do. They might do key rotations. So they might have 2 different auth IDs to authenticate while they're in the middle of a key rotation. But every auth can be tied to certain permissions. And once we get into the smart functions and how we sort of link the smart functions to this, you'll see how that happens. But yeah, there's no real limit that, you know, a user, in fact, you don't even need a user. In this case, we're using users, but technically all you need is auth records to define who can do things in the system.

One other one was—

Yeah,

I

have just a quick question. Uh, so I just want to get— I'm, I'm not, I'm not that like experienced. I'm just new to the idea of decentralization. I just, uh, I like, and I understand the logic of what you did and what you show, but I still like don't get the grasp of the idea is that this this mechanism does not need a specific server. Like, where does the database gonna live if we don't use a server and we want it to be reliable that everyone can access it?

Yeah, so, you know, Flurry, importantly, it's— decentralization is one of those words that's, uh, very loaded. Um, so it is important, I think, to define exactly what you mean. Uh, Flurry cannot be decentralized like Bitcoin, for example, can be decentralized. Bitcoin is a public blockchain. It has, you know, for one, it's slow. It's never going to work fast as a database, but it's not what we designed the platform to do. So there's a number of concepts around decentralization that Flurry can do. One is that, as I mentioned when we were coming in here, that these ledger servers here There can be multiple of them. They actually don't have to be run by a single entity. So, multiple corporations, for example, could all be running them. We're doing a lot of work with Department of Education. In fact, in California, we're hopefully about to start a project where they have kind of 3 main divisions in California. There's Southern, Central, and Northern, and they're working on interconnecting all of the student records so they can guarantee the privacy, some of the things, you know, that Flurry is able to do for them. Mm-hmm. But they want all of them to kind of balance each other out. So they just don't want Southern California to run all the servers. They want someone in sort of each of these areas to do that. So that's certainly an option, something you can do.

But yeah, go ahead. When you have like multiple like links or URL to the server, does that mean that each server will have like identical snapshot?

That's right, because the Raft protocol that's running behind this is guaranteeing that the servers are arriving at consensus.

So if the one, one of— so you don't just depend, like, it is on the architecture of this technique or technology that you don't have to depend on a specific server. It is like by design you can like spread it out.

Absolutely. And in fact, here, this query string, what it'll end up doing when you try and connect, if there's multiple servers, it'll actually try and connect to all of them. And whichever one responds first, it'll end up forming a connection to. But if that server ends up dropping, it'll jump to one of the other servers in the connection string.

And if someone else is doing some transaction with a different link, it is like those servers will Will sync between each other.

Absolutely. Yeah. Yeah. In fact, they won't respond that the transaction was successful until more than 50% of the servers have seen the transaction and durably stored. Yeah.

Yeah. Yeah. I get it. And that's what the essence of ledge and the blockchains coming, right?

Well, I mean, Raft is a pretty common protocol. So if you've if you've used like HashiCorp Consul, for example, you can. So Raft is a private consensus. Protocol. Something like Bitcoin uses a public consensus protocol, which has completely different trade-offs.

Yeah, yeah, yeah.

But yeah, we're using a consensus protocol to make sure the servers are always in sync

and consistent with each other. Yeah,

and consistent with. Yep.

Yeah, makes sense. Thank you very much.

All right. Cool. Well, I will, and I'll try to get to—

I wanted to ask, I asked in the chat like how you're feeling, like if they, if you wanted to go into more code or more into like just like demos of what's possible, like user journeys, like examples of like what would LinkedIn look like or like what would Athens look like sharing TVs or like sharing blocks and pages with one another. I'm not really sure what people want, but I do know that some people here are probably doing code and some people maybe less so.

Okay,

good show.

Well, let me— I— so good feedback, Jeff. So what I'll try and do is kind of balance it out a little bit. I will show some code, but I'll, I'll definitely explain why and how some of these things are there as we go through.

Yeah, thank you.

Okay, all right, so we have some transactions, um, and, uh, we have, um, transacted some block data in there. Uh, I believe the last thing we did was we started to— or I don't know if I transacted. No, I didn't. Okay, so there, I just transacted, um, all of this data right here, which is the block data for, you know, the sample Athens kind of block data here. It's got a bunch of blocks and sub-blocks, children, et cetera, in it. So now we have some block data sitting in there. And of course I can query that out as the graph. And just pointing out here that, you know, time travel is fully there as well. If I wanted to issue that identical query, but instead of using the latest database, I'm using a database, in this case, you know, I'm using a database 5 minutes ago, whatever that database is, I query, there's no data because there was no data 5 minutes ago. This, by the way, is the ISO 8601, you know, standard for duration. So anything to the right of T is like seconds, minutes, hours. Anything to the left of T is days, months, years, etc. So here I'd say 1 minute ago, and we didn't have data in there 1 minute ago, but, you know, in about another 20 seconds, obviously it would be the latest version. You can also lock this into very specific moments in time. I can say I want to do this as a, you know, block 4, and there the data was in there. But if I go back to block 3, it's not in there. So anyway, mutable databases, time travel, you can issue the query across them, things I think a lot of you are also familiar with. And of course, you can read all the block data as well. And like I said, that's always going to have all the cryptographic proofs if you ever did need to prove out any of the data. If you don't care about that, you don't have to worry about it. It's just kind of sitting there in the background. All right, so let's jump into smart functions. This is kind of where we were heading, I think, the last time I was talking to Jeff. So the idea is that data should be able to defend itself, and if you're able to expose data more directly onto consumers who need it, as opposed to them having to go through custom API development to get it, They can then start using very expressive query languages to just ask for what they want. Flurry supports GraphQL, like I mentioned. We support SQL. We don't support all of SQL, but if someone knows SQL and doesn't know Sparkle or these other things, that's fine too. They could still get some queries going. Of course, we support Sparkle. What I was showing you was what we call FlurryQL, which is really Sparkle in JSON, or in this case in Eden. But let's create a smart function now that starts to lock down this data. So I'm just kind of again using this like a normal database. I'm— it's— Flurry is configured right now with what we call OpenAPI on, meaning that it will allow kind of a root access. If you turn that flag off, Flurry will not allow anything unless the query or transaction is cryptographically signed. So we're going to play a little bit with both of those here. But what I want to do is I want to create a smart function. This smart function is going to say that you can only see your own blocks, right? So if these are my notes, I can only see my own notes. I can't see anyone else's notes. And this is actually the smart function that I want to put into the system, and I'm going to show how this is done. Um, you write smart functions effectively in Clojure. You are going to have all your normal sort of closure functions available to you, so you can do quite a bit in this. We have a special out-of-the-box function we call relationship, and this is just because it's really common. And what we're doing here is we're basically bookending whether or not someone can see data by following the graph. So remember, every time anyone's interacting with Flurry, there's always an identity tied to it. So on this side, we're getting the identity. We know that identity is tied to a user. We know that the user is referenced from the block, from the notes. And then this would end up being basically the block, the subject ID. These are functions that actually have a hidden context that's passed in as they're headed in, as they're being executed. That's why you don't see any arguments on this subject ID or auth ID. It's because this context of the transaction is sort of injected into the function. But it's basically gonna make sure make sure this relationship exists. So, you know, here's block user. So it's basically saying go from the auth to the user, the user up to this block, and then can the user see this? So in order to put this in, I'm going to transact this into our function collection. So I'm just going to take this smart function code right here. I'm actually going to create a function out of it, but our function's now in Flurry. But it's not doing anything. Now we need to hook into how do we trigger this function to actually do something if we want. There's really 2 big things you can do with smart functions. One is you can do data validation, and we call these specs, not to be confused with ClosureSpec, but they allow you to set up any type of, as complicated as you want, criteria for what must exist in data. For example, you could say A, you know, a new block can only be created if it references a user and that user has, you know, 5 friends. That's something you could say if you wanted to and express as a smart function. The other thing you can use smart functions for is to enforce permissions. So that first example might be just a generic, you know, data always has to look like this. This example is data is gonna be tied to identity. So our smart function's in there, and how do we tie this now to an identity? And that's going to be this next piece. So we are going to create a, what we call a role, and a role is really just a grouping of rules, but it's this rule that ties this smart function that we created into the logic. And you'll see here this rule we're going to create, we're going to say here just in a docstring that the user can only read and write their own blocks. Meaning their own notes, and we're going to reference this function that we just created. That's smart function, and we're going to say this is going to apply both for queries and transactions. So this can be two distinct operations. For example, I might have permission to transact something, but I can't look at it, or I might be able to look at something, but I can't transact it. In this case, we're saying you know the user, this smart function can enforce that. How we trigger this to only look at blocks as opposed to any other data in Flurry is right here. We say this is only going to apply to blocks. And we're actually going to set this up as a default. So individual predicates or entities can even have their own rules. So you can get very, very granular in this. So we'll go ahead and set up that smart function. And now the only thing left is to tie the identity to this role. So we call this role Standard User. This is presumably what we would get of every standard user who signs up to the service, this role. So we never have to write any other permissions besides this one. And in this case, I'm just going to attach it to Trey. So I'm going to update Trey's auth record. And again, I had his auth ID in that previous namespace over here. So here, auth Trey, here's his auth ID. So that's a unique identifier. You can use 2 tuples to uniquely identify anything in Flurry. So for this ID, we're saying for the auth ID, get that auth value, and we're gonna set this role up, this standard user role that we just created here. So now Trey has this associated with him, and what we can do is we can create a database, call it TreyDB, which creates a database just like we did before, but we're actually saying this database has an auth attached to it. And it's that auth, the Trey auth that we just created. Now again, Flurry will automatically attach auth to it by decoding cryptographic signatures if it's open. This is kind of assuming that you're using Flurry from a frontend service. And if I use Trey's database and I issue that query, that's fine, I can see everything. That's because all these blocks were actually written by Trey. But I'm going to also here get a BPlats database where I'm going to actually attach the auth to BPlats, and I'm going to issue that identical query, and I see nothing. So what happened here is now Trey has basically a virtualized database that only has data in it that he can see, and he can issue any query he wants. Brian has his own virtualized database. It only has the data in it that Brian can see, and he can issue any query he wants. And when I mentioned before, you don't even need an app server. This is partially why, because usually we're using the app server to enforce identity or permissions. And again, you can still do that. There's nothing preventing you from doing that. It's a perfectly fine way to use Flurry. A lot of people use Flurry like that, but you don't have to. You can actually open it up, and this is going to open up anybody to write apps or anything around the data that's being stored, but always guaranteeing the permissions in place.

So, Brian, just to clarify, so you have the core ledger, which is where you transacted all the data in the first place, and then you created 2 essentially query servers that are TradeDB and BrianPlatzDB, each of which are attached to certain roles. And based on those roles, you kind of have a virtualized database for each that can either access a set of transactions in the core ledger or not, right?

That's correct, yeah. So whatever sort of smart functions are tied to them is going to govern what data they can see. And again, the powerful thing here is that you can use the relationships in the graph. So, you know, there's nothing preventing you from, for example, introducing a new predicate called friends. And you could say, well, if I'm linked to a friend, it means I can see their notes as well. And then the next time you say select star from notes or blocks here, you're going to see not only your notes, you're going to see your friend's notes because that relationship is there. And as soon as that person unfriends you, they're going to disappear.

Awesome.

So you're using relationships in the data to do this, but no one knows what data is there that they can't see. This is very important to Flurry that we never even allow a hint of data that might exist that the user doesn't have permission to, even in its evaluation. So I just want to show another sort of extension. So we kind of showed kind of virtualized databases with permissions, et cetera. That can be a whole topic in itself. I'm just trying to touch on some of these concepts here at a real high level. So let's do one where let's create a couple extra schema elements. So we'll create a new collection called artists, like a painting For example, and say we're creating notes in that note. We want to link to the artist because it's talking about an artist, or that note is doing something else. So we'll create a new reference off of a block called artist, and we'll allow you to store, in this case, an artist string. It's just an arbitrary example. So what I'm going to do is I'm going to create a new block. Here's my block UID. Here's my block string. You know, hello. And in this case, I'm going to reference, you know, this note talks about Gustav Klimt, which is, you know, a fairly famous painter. Probably I said his name wrong, but it just— this note happens to talk about this entity. We don't know anything about him, but let's go ahead and transact it. Here's an example. So you can actually federate queries and use different data sources. In this case, we're going to use Wikidata because Wikidata also happens to be a triple store. There's triple stores coming all over the place that are housing data, and I would bet that Wikidata knows something about Gustav. So what I'm going to do now is if I'm rendering my block, for example, I can connect not only the data directly in my local database, but I can start dynamically connecting it with external data sources. So I'm going to issue the same type of query. I'm going to, you know, pull out this particular block ID, which is the one we just created. Then I'm going to crawl the block to this block artist. I'm going to get out the artist name, and then I'm going to start to federate this. So there's another element that's introduced into the, you know, triple declaration in the query. $wd, we already know in Flurry $wd means Wikidata, so we already made that connection. But this could be another ledger, this could be a different Flurry server, etc., that you're combining it. And we're going to use Wikidata to pull in any information about our artists here. That it might exist, and what we're going to try and do is hey, I want to get a list of artwork. Of course, we didn't start the artwork here in Flurry, so we're connecting databases now to bring this information together. And there we just issued a query for a block that we stored here that connected to an entity that we just combined with Wikidata, and now we've pulled out from Wikidata all the artwork that Gustave has ever made. Along with an IRI. So these ideas of federating data and connecting data across sources are really powerful. I did not get into this here, and we're not going to have time. Happy to do it again, but we're using kind of standard JSON here. We're using more of a like data script like format, a kind of closed world format. You can also use JSON-LD, and instead of identifying like these underscore IDs and kind of this closed world notion of things. You can use an @ID and use IRIs. IRIs to not only identify uniquely what you're talking about, but IRIs to even identify predicates, which means you can start having shared schemas. So like schema.org, of course, is a very popular shared schema. Google uses this. If you've developed webpages, you've probably embedded schema.org data right into your webpages. You could tag instead of us having our proprietary, you know, user username, we could use the schema.org vocabulary. Anyone using schema.org can now dynamically integrate with our data. They don't need to write integration scripts. They don't need to say this is really that. It can automatically happen, which is a really cool concept.

I just have a quick question about the Wikidata, if possible. Yeah. So, so I know, like, I have been— you, like, I'm a little bit aware of Wikidata and their logic. If you go back to the code here, where did you get the B178? B178, it does mean that you are looking to this entity in Wikidata. Where did you get data from?

Um, which, which part are you referring to? This?

Yes. B17. Isn't this like an identification for Wikidata?

Well, Wikidata, yeah, Wikidata is actually not a pleasant one to integrate with. So they very focused on being international, so they never use any language, any terms within it. So if you wanted to find something about, you know, Madonna, You can't find it by the name Madonna. You actually have to find their internal tag. So this happens to be the internal tag that Wikidata uses. But yeah, if we, if we just took one of these URLs too and just came here, right? So here's the, here's the painting that that was referencing. We're on Wikidata. You know, all this data is available to me.

Yeah.

Yeah. Yeah.

And but no, I just was like, But you knew beforehand to use that tag, right?

I did, because here, well, for Wikidata specifically. Now again, if I was using like schema.org vocabulary, I wouldn't have to do that because schema.org, you know, everyone's going to use the full IRI.

Yeah, yeah, yeah, yeah, yeah. Got it.

Yeah. So if I wanted to define a person but use the schema.org vocabulary, I could use this vocabulary. Now anyone else using this vocabulary, I can link and integrate How would

real-time collaboration work with Story?

Real-time, actually, you know what, that's a— I think will allow me to show the last thing, Jeff, and this isn't gonna 100% answer it, but I think it's a pretty cool thing and will at least partially answer it. So what I'm gonna do here is I'm gonna use Create React App. And there's already a template out there, a Flurry template that just puts some of the Flurry plumbing directly in there. But I'm just gonna create a frontend JavaScript app, no app server again, no anything. And what we wanna do is show how we can start streaming data events and real-time data quite easily. And I think collaboration is going to require at least a couple things. One is streaming data data that multiple parties can understand. The other part I think is locking in time, because how can you collaborate about data that's constantly changing? It's difficult to say, you know, I know the same thing as you when the data might have changed between when we asked. But here I just created a React app template. Open this up in VS Code, and I'm just gonna go into the app server here, and we just have, we're importing here actually the Flurry library. The entire Flurry database is in ClojureScript. We can compile it into JavaScript. In this case, we're actually gonna run it as a web worker, so it's not even gonna interfere with our main render thread. You can, of course, embed it in Node.js apps, In fact, we did a recent webinar where we did auto scaling on AWS, where we were just you know hammering load against a Node.js app that had the entire database embedded inside of Node.js, and then used auto scaling. And of course, the example that I was just using, we're running literally the entire database inside of Closure, inside of your own app. So you have this ability to embed the database as a library. In this case, we're doing it in JavaScript. So here we're gonna point it at that same server. Let's make sure we point it at the right database here, which we're calling Athens test 3. Oops. And I'm gonna run yarn start. All right, so here's Flurry running. It's connecting. It's actually the entire database again is running in a web worker. It's just using this out-of-the-box template. But let's say, you know, we have this sample. You can just attach queries right to React components if you want, but we want to pull out blocks. And we want to pull out your UID and the, I think, block string is the name. Go ahead and do that. Right, so here's the blocks that are currently in the system. They're sitting here in the browser, and the entire database is literally running in the browser right now. So one of the things, and I'll keep that there so we can see it, but let's just go back to here and let's create another new block. Blah blah, just so we can see it. Hello, new transaction. Go ahead and transact this. So this is an entirely different process. Transact it and you see all the data is instantly streamed to the front-end UI and automatically re-renders in real time. So what's cool here is that if you want to use Flurry like this, You can actually embed it, no app server. You can, when you connect in JavaScript, you can connect as an identity. Again, everyone's gonna have a filtered database and you have for free a permissioned real-time database that can automatically trigger re-renders in the UI. And you had to write, literally here we wrote zero plumbing to do that. So these concepts of streaming data, scaling data, embedding data into applications, They're really all there. The Flurry wrap or the React wrapper that we wrote in this particular case is literally a single index.js file. It's about 600 lines of code. But if you wanted to do this for Vue.js, if you wanted to do this for Angular, you know, it's almost all the work here is Flurry. It's just sort of wrapping how you re-render components in these different frontend frameworks. But Flurry's actually triggering all of this. I know we're about out of time, but I wanted to at least include that. So yeah, if someone was typing notes or anything like that, I would literally see the updates here in real time.

One last big kind of thing to cover, Brian, is do you want to talk about the licensure updates that we made? I know there's quite a few people that are interested in that.

Yeah, so Flurry is licensed under Eclipse Public License 2, including the libraries. The ledger server, the backend server, is registered under AGPL. The backend server is not designed to be embedded inside of any app, so, you know, only the database library is. So you would never embed the Flurry ledger server inside of an app, so the license there is, I think, less relevant. And if it is still relevant, anyone can download the version that we already compile, and then it has a— it's not even an open-source license, it's just a Flurry free-to-use license at that point. But normally that ledger server, you would, you know, throw in a Docker container, run it on Kubernetes, run it on EC2, run it however you want, but that wouldn't be what you would ever embed in code.

Yeah, because I know a lot of people have seen that We were AGPL a couple weeks ago. We just made the update, so I wanted to make sure that we brought that up on this demo. I

think there are specific questions like in the chat.

Yeah, like they feel like—

Lauren did ask one. All right, is that just background noise, Jeff, or someone asking a question?

Oh,

it's background noise.

Oh, gotcha. Uh, I think Lauren was, was asking about about the Ledger server being able to run on employers' hardware. And so I don't know if you have any thoughts about that or, you know, what that has to do with like the commercial license, um, like actually purchasing Flurry, like how would that work?

Um, the first part of that question, uh, Matei, Flurry running on, on what?

Uh, she was saying on Her employer's hardware, which I guess would be an enterprise that doesn't allow AGPL software?

Yeah, I mean, so you don't have to use the open-source version of Flurry at all. You can just go— in fact, if you want, you can just go flurr.ed. There's a link on our homepage. Oops, it started. You can literally download it right here. It's already compiled, it's already in a JAR file, and you're just using the Flurry, you know, custom agreement terms. You're not even using the open source license. So the only time you're always using the open source license is if you're downloading it from GitHub. or you're compiling it directly from GitHub, that's where the open source license applies. Again, if you're running the ledger server, you're not embedding that with any or combining it with any other code. You could also just run our Docker version too. So there's no reason you— I guess there's no reason you have to run AGPL code to run Flurry is the short answer.

And we also have the on-demand version. So we have a version that we're serving on AWS that you can spin up a ledger on and then connect via HTTP calls. So it all runs on our servers. You're just building the client that's connecting to it. So I don't think that would invalidate any sort of employer contracts.

Yeah, and importantly, the ledger server does run independently from the query server. And so the ledger server, you know, you have a lot of flexibility where and how you run that, but you can always run your query server You know, close to your code, in your AWS environment, etc. Well, in fact, you know, we were just showing it here a second ago, we're running it inside of a web browser.

What's— how much does it cost to like run Flurry? Do you guys have pricing?

Yeah, it's free. So Flurry itself is free. The reason you might want to pay Flurry is For one, if you wanted services. For two, if you wanted a guaranteed service level agreement. So, you know, if you wanted like 2-hour or 4-hour minimum support for severity 1 issues, that sort of stuff, that's when a typical corporation. But we have a large number of startups that use Flurry. And, you know, I think we only have 2 or 3 of them that actually pay that. So most of them don't. And that's fine. I mean, That's part of what we want. We just want people to use the technology, help out, provide feedback. If you like it, you know, tell your friends. If you don't like it, tell us.

So for Athens, what would the implementation look like? Let's say we have some people that want to self-host us. Maybe they're more technical, they have to self-host their own data. And then there's also some users that it's not as important to them, uh, to self-host, but maybe they still want some sort of collaboration. Up until now, I was thinking more in the typical model of software, which is like there's the self-hosted ones, they're kind of their own thing. Maybe like the entire company uses it internally, so like some people We'll use GitLab completely internally. It's all self-hosted. And then it's kind of like the SaaS thing where it's like we can just collaborate really easily with anyone, but it's all this like centralized server. So that's like the Notion, like you just add me as a collaborator, I already have a Notion account, it's really easy for me to edit. What would, uh, kind of like the collaboration model look like?

Yeah, I, yeah, yeah, I would, I would probably, you know, Well, I don't know what your customers are asking you to do, but I would probably run an Athens hosted version, give people the option of running maybe a packaged version that can have the data locally if they want. But you're kind of managing the backplane, you're managing sort of all the core ledger data and all the changes there. You could bundle it up with your offering if you wanted to, to allow people to run the whole thing in their own environment. Or, you know, if people want to— when we have people nowadays who are trying to run Flurry in their own environment, lots of times they want to run it on Kubernetes. So that might be another option is you could just set up a, you know, some Kubernetes scripts that allow people to kind of spin up the different services real easy that are required to do it. Importantly, so the data backplane is going to have to run as a service. But the entire database and the app can run, you know, can be completely embedded. So you're going to want a backend server that's like durably storing the data and all of that sort of stuff. But the app itself, I mean, you know, it can be anything. It can even be a JavaScript app like we were showing here. It doesn't even have to be an app server.

So, so users could still self-host this backend? the backend server, right? Like they would just have to figure out how to do that. Like it doesn't have to be Athens that does that unless Athens wants to do SaaS. So could users still kind of like get the bundled version of all this and run their own server with the backplane you're saying?

Yeah. Yeah. And I, like I said, you know, how you want to do that is up to you, but I'll just show kind of what we've been doing because this has been happening to us is that we have a we have this K8s directory in our source code repository, and this just has some different configurations. And if people want to, they can literally take this. If they're running you know Kubernetes, they can take this Kubernetes script, put it in, and say you know I think it's I don't use Kubernetes a lot. I think it's cuddle cuddle is the command line cuddle up. And, you know, the whole thing spins up and, you know, it's running. So you might want to do something like that with Athens where you kind of have this stuff, you know, sitting already in containers like on Docker Hub or somewhere else. And all someone has to do is take this script and say, if they're running Kubernetes, say, you know, cuddle up and, and that sounds like cuddle up and I know it's not cuddle up. It's kubectl. I don't know, somebody on this call, I'm sure,

Kubectl.

Kubectl, thanks.

So I still don't understand how this would work to build a fully decentralized solution. So I have my own Appian database just for me. It's only hosted on my computer. Because I don't want it to be anywhere. I don't want anyone to be on anyone's servers except for mine. Maybe there's no new server, actually. It's just my local desktop app, similar to what we have now. Then I also want to connect to a shared Athens database where I have access to it, and all the contributors have access to it. All the employees at Athens have access to it. Maybe there's a Flutter page that the Flutter team has access to. access to? Like, how would that be implemented?

So from a query standpoint, let me see if I have an example right here. Well, I don't have one right here, but we'll do it really quick. So from a query standpoint, you can have multiple sources. What we can do is we can actually start pointing to different databases. So we're going to call, you know, my work database or ledger, and this could have a URL, you know, http://flir.ee/athens, you know, I don't know, whatever, whatever it may be. In your query clauses, if you're trying to return Like a consolidated view of notes, for example, you can do bindings directly with different data sources. So when we're querying this database, the primary database is whatever database that we created up here, but we can actually specify different ones. And then you can prefix statements in your queries and say, okay, you know, combine these entities, but bind, you know, subject ID with my work subject ID. And then in this case, we'd be talking about the identity. Entity, but maybe they're both storing different data. So my work one stores block coworkers and my local one doesn't. So that would be a way of binding those directly. If you want to use IRIs, and we didn't actually show using, you know, IRIs to uniquely identify things, then you're really golden because every sort of entity, no matter where it exists on the internet, ends up having a unique identifier and you can always use that as a binding. As opposed to using something like user email, you know, bplatz@gmail.com is trying to uniquely identify me, for example, you could use IRIs to uniquely identify things. So that would be another way. So if you're trying to consolidate query results, that would be the way you would do it. You're starting to have statements in your queries that are binding to different data sources. And I don't know if that answers your question.

You can already do that with I don't know what Datascript and Datomic— right, so like where is the server hosted? I mean, you can do what Datascript does for sure. You can like have multiple connections, you can query multiple connections at the same time. A database is just an argument. So I get that you can query multiple databases. Like where are the databases hosted in the case where I have my I have my personal Athens database on my computer, nowhere else, and then there's a shared Athens database between all the collaborators of Athens. Where is Flurry hosted in this case?

You would have one in both locations. You'd have one in the central location and one running locally, and this is how you would combine

Who manages the authorization and permissions for the Athens database?

You can actually set up smart functions so only the users can manage their own authorizations if you want. You could be an entity or a service as Athens that helps people manage their identity. There's a concept of authority in Flurry where you can actually delegate other individuals to be able to perform operations on your behalf. We call that the authority model. Sometimes people, you know, front-end an authentication service in front of Flurry, OAuth, or, you know, Keycloak, for example, is another common one that a lot of our government work uses. So I think there's a lot of options. I think the question is, what do you want and how do you want it to work? And I could probably, if I'm not answering it well enough at this point, it's maybe because I don't understand where I'm missing the mark there.

Is there someone who creates the first Flurry database that has like permissions and they can say like, these people have access to my database? Yeah.

Yeah. And in fact, that's what we did here. So when we created this new ledger, the very first thing we did was A, transacted schema, right? And then at the point you have a schema, you can start layering in smart functions if you want to control permissions. And again, that's only if you want to. You don't have to use those. Then people have a cryptographic identity. If you really want to decentralize it, you would end up likely using private keys. And this private key can prove who I'm at Any instance, they've never seen my private key. They've never seen my public key. All I need to do is tell them my identity, and when I sign with this private key, every Flurry instance can figure out who they are and if they have any sort of standing or permission on that server. So the queries end up getting cryptographically signed to do that. Now with Department Education, we're also doing a lot of. Work with VC DIDs. And I mean, this to me is the future. This is where I think everyone should be looking. But, well, that was the wrong page. But here at the W3C, this allows people to individually manage self-sovereign identity. And you can start setting up permissions, which is really cool, without even knowing the user upfront. So you could say, this user can transact this, or this user can can see this if they're a US citizen, they currently work for Microsoft, and they had at least a 4.0 GPA from an Ivy League school. And different credential providers that you end up trusting, and it can sort of be down a chain, will assert these things. And as long as up the chain you believe these entities, that would be an example where I wouldn't even need to know about you ahead of time to give you permissions to do certain things. Certain things. And that's super cool. And that's really where I think, you know, the world is going to be in a few years. Ironically, places like Department of Education are sort of trailing the ground there. But this allows people to manage their own identities however they want. And then you get out of that business entirely. You just set up what we would call trust chains. But that could be another thing

So, so Brian, just going back to Jeff's question and how this could run in the context of Athens, I think, Jeff, correct me if I'm wrong, but what it sounds like is if you had, you know, just a self-hosted graph, right, it's individually owned, you set it up, you set up the permissions, it'd be equivalent to just running like a local instance of Atlas. local private graph. And then you also had a shared graph that was, you know, at first say centralized. And it's kind of how we're trying to do with Athens right now with real-time collaboration, where it's a central server, everyone's writing to it and the app server handles the reads and writes. Or alternatively, you could, in that same scenario, that shared graph, decentralize it and set up multiple servers that run on Raft consensus. And then, you know, in either of those modes, you could federate queries, you know, as you were showing here with SPARQL. And you could just like, just kind of put the database as a prefix and still keep your individual graph privately owned, not shared with anyone except through smart functions. And then just federate queries to these either centralized servers or decentralized servers. So you kind of have the option. Yeah. to run it in these multiple modes. Is that— does that sound about right, both for Jeff and Brian?

I'd say it's right for me. And then, you know, the thing, and maybe I wasn't understanding this part of your question, Jeff, if you're gonna connect 2 different servers together, you need to know like the IP address or where those servers are. So there's no like this magic way that, you know, it knows where Flurry Work is. Like, I'm gonna have to have an IP address or something like that that's gonna be pointing at that server I'm also going to say, hey, I'm running one on a localhost, and that would be enough for me to start actually combining query results, if you even want to combine query results. You know, so that's, I guess, sort of a product decision. The product might even be better if you didn't try to combine query results, you just allowed people to navigate between these things, you know, via some sort of UI. But, you know, whatever, either of those work. And yeah, any of those server addresses that you're pointing to could actually be decentralized if they wanted to. So, um, and Athens could run one if they want. I would say we have a good number of people who are actually hosting all their customer data. They're not even running it locally. But because they can wrap the smart functions and the cryptographic proofs around it, uh, anyone who contributes data can I can prove that Athens never manipulated my data because all the cryptographic proofs are there. The smart functions prevent anyone from seeing it, and I control my own data even though it's physically on your servers. So that may also be enough. You know, those are a few different thoughts.

That's a good summary, Matthijs. And so how would real-time collaboration work in this model? Say we're federated for like 2

So just maybe give me a quick use case. What, what are you collect— what's the collaboration doing?

So I'm taking notes right now on Flurry on my personal database. I'm learning about the Raft protocol. And then I meet with a few of the contributors here working on the backend, and we are sharing each other's notes. On the backend, distributed systems, architectures. I want to link to their notes for my database. I want to link to some of my notes in collaborative database.

Yeah. I mean, that would— the answer is I don't know. There's probably a lot of ways. That would be a good use case of using IRIs, which, you know, I didn't actually show here, but we can— I could try and show you here really quick as long as I don't fat finger things. But instead of using, you know, these internal identifiers like we have here, there's no real good way, guaranteed way of linking across entities because my internal identifiers might be differently. So this is where JSON-LD comes in and, you know, actually linking data in the web the same way that web pages link data. you know, across the web, or web pages link web pages for humans. Link data links data for machines. And so I would probably, if you're going to connect, go to this format here where you're using an @id and you start using IRIs. And the IRIs, you can make whatever IRIs you want, doesn't really matter. There's a lot of things already identified on the internet as well. That you can use those IRIs for, but it creates a surefire way of linking objects, the things you're talking about, without having to coordinate some sort of other method. So I would think about building something like that into the app that when you're creating notes, it's creating an IRI. That IRI could be hosted anywhere, so you don't necessarily have to tie that IRI to like an IP address or anything like that. But you do have a guaranteed uniqueness around that IRI that different things can now talk about the same thing and be guaranteed that they will resolve to the same entity.

So basically you could, for each block, give it an IRI and then each block is globally accessible, still defended by the smart functions. And so if Jeff and I were both had our own private graphs and And I wanted to like embed his block into my graph, I could do that easily with an IRI and I could see his block embedded in my graph. That wouldn't be necessarily real-time collaboration because you still have to like issue the query. So, you know, there's this response time, but that way you could embed other people's blocks into your own graph and vice versa. And I think Jeff on the real-time collaboration collaboration, like if you wanted to do that, like the best bet would still be having a centralized server and, you know, for the real-time, if you need the real-time collaboration and those speeds, you'd have a centralized server, but then that could act as a node where like, okay, for my individual graph, I can now just embed the blocks from that real-time server into my own graph, which that's not a real-time thing, um, but I may not need that, right? You don't always need real-time. Sometimes you just need to be able to access information from other places. Um, does that make sense?

That makes sense. How slow would that be if it wasn't real-time?

You know, I, I mean, it depends on the query. Um, simple queries typically respond in about a millisecond. So, you know, really complicated ones take longer. If you're using smart functions to protect data, that's going to add some additional latency. So all these things, you know, are factors to think about. If you want to run the Flurry library directly, you have the ability of registering listener functions directly on those data events. And then that is how I would trigger real-time, you know, updates typically. And in fact, I don't have it up here anymore. anymore. I think I closed it. But when you just saw the React app that I had up and you saw it updating in real time, all that's happening there is it's running the Flurry library where you can register a listener function. And basically you're saying, call my function every time there's new data. And your function will get called and it can do anything. It can re-render a frontend UI, it can send an email, it can, you know, do anything you want. So that would be a way of listening to data events even across servers if that's something you wanted to do. You don't have to. You could, you know, issue a query every 30 seconds if you wanted to. Um, you know, I, I, I don't know. It's, it's sort of, it's sort of up to you.

But I just had a question here. But when you just showed us what the React app, it did that already, there was like something like a socket listening to the database. And when you issued a transaction, it is already updated. So there was like some sort of mechanism that you did, is that you already implemented a listen function in that case?

Yeah, um, I did. So, uh, right here, right in the Flurry API, you see is, uh, flurry_db.listen, and you just give it the connection, the what you want to listen to You know, and you give it any sort of you know key so you can close the listener if you want. And now every time there's an event, we actually put two variables in here. Every time there's event, my function is going to get called. I can do anything I want: send emails. I can re-render things. So in the JavaScript version that we were seeing, that's all that's done here is basically. We're registering event listeners, and there's a WebSocket connection. So you're right. So basically, when I created this connection right here, whether it's in JavaScript or Closure, when I create this connection, what's actually created is a WebSocket connection in the backend. And that WebSocket connection is automatically getting fed data events in real time, and they're permissioned. So if a data event happened that I can't see the data for, I would never get it. It wouldn't trigger my listener. So it's only gonna push through data events that I have permission for based on my smart functions. And again, you can write your own code to enforce your own security if you want. You don't have to use smart functions to do that. You can run this like any other database. But if you wanted to, then those smart functions will only allow the data changes through. And then based on the data change, you can do, you know, whatever you want.

That might be a way. And that's, that's really, really cool. Um, so let's say there's— so how does this work in the blockchain example where there's just like one gigantic Bitcoin blockchain that is globally accessible? So let's say that we— there are, you know, millions of people that want to read King James Bible 4 and they want to reference it and they want to collaborate on it. Like, how would that work in kind of like this Bitcoin example, where there's just one global blockchain? We all want to reference it. Yeah,

I don't know if I fully understand the question. So for one, the easiest way, if multiple parties are not collaborating, to reference an entity will be From some unique identifier, we build in the notion that you can use IRIs. That's a great way of doing things. It doesn't have to be. You could use things like email address if you were talking about users. You can have multiple unique identifiers. So I guess that would be the first thing is deciding how you're going to uniquely identify something so that it can be collaborated around. And you know, Flurry is not going to solve that for you. It's going to allow you to use any of those. Identifiers real easily. But, you know, figuring out how you're going to do that would be the first step. And then, you know, you have the option of either creating multiple records about those items. You could even create multiple predicates if you want, or sort of attributes, I guess, in your language, where different companies are even adding different attributes that maybe only they can see to a specific And in that way, they're collaborating around it.

Yeah, sorry, it wasn't a clear question, but I'm still trying to understand it myself. But I feel like there's a use case where there's like one— we want one canonical knowledge graph. We want it to be— we don't want it to be centralized, but we still want anyone to be able to

Yeah, so the complication with knowledge graphs usually, you know, attempts to get resolved with ontologies, and we didn't even get into ontologies here. And the reason we need ontologies is we need to start to create effectively a class hierarchy, and that is because the only way of really resolving different things talking about the same thing, because they're all going to talk about it a little bit differently, is you need to abstract a little bit. So with— we didn't show it here, but you can create different class types. Classes can have subclasses, and it can allow people, even if they're talking about, you know, I'm talking about a bicycle, you're talking about a car, but it could allow a third party to come in and say, I want to find all vehicles. And we automatically know that a bike and a car are vehicles, and so they both show up in that query. And I may be manufacturing a specific type of bike, a tricycle, for example, but someone wants to find— well, I guess that's not a bicycle, bad example— but a tricycle is also a vehicle. Like, so these, these sort of cascading class hierarchies are typically how people end up resolving that problem because it's a hard problem. I mean, it's not— it's less a technology problem than it is probably an ontology and sort of taxonomy problem.

So

I didn't mean like, how do we create a shared ontology or a shared knowledge graph, like in the scientific sense. I meant like, could we have a global learning instance that anyone can read and write to?

Yeah, so, um, you might have just helped me, I think, clarify something. So, um, you can run any number of Flurry Ledger servers, but Flurry Ledger servers are only connected together if you explicitly have them participating in validating the same ledgers. The combining ledger data, combining multiple instances, doesn't happen at the ledger server at all. It happens at the query side. So what the query side does is it allows you to start connecting to different ledgers or instances and bringing those together and doing joins inside of the query. So that those joins aren't really done in the ledger side at all. Now multiple parties can can participate in the same ledgers if they want, but they're all going to get visibility on the backend to the data because there's no way for them to validate the data if they can't see it. You know, I can't validate a cryptographic signature if I can't see the JSON that was signed, and the JSON that was signed has data in it, and I have to be able to see it. If I can't see it, I can't validate the identity. So, so those backplane ledger servers are going to have really full visibility into all the transactions coming through. And you can say, hey, these 3 ledger servers, you know, that are participating in the network are only responsible for this one ledger. Or maybe it's Athens running it and they're responsible for 1,000 ledgers. You know, that's fine. They're all just ledgers running in this thing and they're all validating and working together to have redundancy and, and all those aspects. But if me as a user need to connect to completely independent, like organization Athene servers, for example, that's happening at the query level. I hope that helps.

Yeah, but my question is, like, could everyone that's, like, using Athene connect to— I get the queries and joins are just done at the application level, but, like, could there be a single Flurry instance that everyone could connect to?

Yeah, and you could set up smart functions so that you can allow other people to connect to it too and even issue GraphQL or SQL queries and they'll only ever see the data they can see and they can control who has access to their data. Absolutely. I mean, that would be a cool way of running Athens, I think, which is, hey, we're Athens, we're running it, we're running it using this technology that is allowing you to control your permissions and how you you share your notes and your data. We're keeping it secure and we can actually even have proofs that if we ever wanted to expose them, that can prove that we never manipulated data, that data that Jeff submitted really came from Jeff. There's no way we could ever change it. And then if you really want to run one inside your own company, you can. It's up to you, but we provide an immense amount of protection if you don't. I mean, I would think that That would be a pretty good message. But—

So just to maybe clarify a use case, like say that Athens wanted to run an Athens network. And this is like the global Athens network that everyone who, you know, if they want to participate in the Athens network as a global graph, would be sharing the same ledger, right? But maybe with the query, They only have access to certain parts of it or just to all parts of it. But that one ledger, that global Athens ledger would be distributed across all the participants. And, you know, at the same time, me as an individual Athens user could still have my own individual graph and I can combine my queries with, you know, the query server with the global Athens ledger. And those are 2 different ledgers. They can be combined if I wanted to, but Athens Athens could have just its own global ledger that allows any participants to join and, you know, uses the Raft consensus, right? So you could do both. Or if someone wanted to set up a different global graph, right, aside from Athens, you could still do that with the same platform. So like if a different organization that was creating some sort of like maybe a research coalition was setting up their own global knowledge ledger, same exact thing as Athens doing its own. So would that work, right? Like, maybe I'm getting—

Yeah, and, and yeah, so you could make— so say Athens is running the main sort of backend and people have this option of running their own if they want. You have 2 choices at that point. One is, is do I run everyone on the same ledger? That actually works really well. Everyone would have visibility into the ledger, except they would only see data they have permission to. So doesn't matter how they try and query it; they're only ever going to get results of their sort of virtual database inside of your larger ledger. If you wanted to, you could actually set up an individual ledger for every user, but you're still managing the backplane. That's a fine thing to do too. That's going to make your queries a little bit more complicated because now you're going to have to get into the syntax. Where you're pointing at different ledgers as opposed to just, you know, giving it one ledger that it's doing these queries, but it's filtering out data. But absolutely you can do it. Or you can combine multiple ledgers running on completely different servers. That's going to complicate it a little bit more because now you got to point at multiple sort of servers. You got WebSockets open with more than one server now that are getting event updates. But certainly if that's You know, an important feature, that is also possible.

I just have a question regarding if you want to have all people on one ledger versus each one have their own ledger. Does it have any like scaling implementation? Like, is there a limit? Or if like I have like X number of users connecting to this ledger, then at this point I would need like to split up or like use multiple servers like To balance the load? Is there a limitation?

Yeah, good question. So, on the query side, you really have no limitations. So, because you can spin up query servers.

Yeah, yeah.

But on the transaction side, you will have limitations. So, that's something to consider is, you know, if everyone is going to be making very, very frequent updates and you have a lot, lot of users, you might want to run it across multiple ledgers. And that's gonna, um, you know, one nice thing about, uh, the Flurry and the Raft consensus as well that we do is that we'll actually load balance your ledgers across servers. So, you know, for example, um, uh, like Datomic, at least it used to be like this, they, they have one ledger server and that's it. Doesn't matter how many databases you have, and you can have like one redundant one. So after a few seconds, if your primary fails, it'll go to your Flurry is very different. You have 3, they're all actively running, they all have a hot copy. Anyone can go down at any time, it will stay running. But importantly, it'll actually distribute all the ledgers running on that server to all the different servers. So server A gets, you know, 15 ledgers, server B gets 15 ledgers, server C gets 15 ledgers. And if server C goes down, the 15 ledgers that it has gets redistributed to the 2 servers that are left. And then if server C comes back, it rebalances again. So it's always balancing the ledger load. Now, with everything in a single ledger, no more than one server can transact it at a given time. So you are going to create a bottleneck by having it on the same server. And part of that's just thinking about, you know, how much transactional volume you have and you expect, and, you know, thinking through some of those.

What are you saying? You can't have the same ledger on different

There's copies of the same ledger on different servers, but one ledger is only responsible for processing updates at a given time.

Because you don't want— because you want to confirm that transaction being consistent, right?

That's right, strong ACID consistency.

But, but, but if you are creating virtual ledger, does that resolve it? Because if like 2 users are creating like private transaction. It does not matter that one— it, I mean, it makes sense in theory that you don't necessarily want to have one server that have to listen to them because they are like contributing to private one. But then again, you would lose the sense of why you need one ledger.

Yeah, so on the query side, you know, it doesn't matter, but on the ledger side it still will. I mean, maybe in the future we try to, you know, build in some sort of locking mechanism that can even distribute load for a single ledger across multiples. We actually have a notion of sharding already in the schema of Flurry with the idea that based on IRIs, you can set up namespaces and break them into shards, but it's not in the product yet.

Oh yeah, yeah, makes sense. So, so yeah, so it would be that that each copy of the ledger would be responsible for a specific number of ROI. That's right. So, that's like— yeah. So, that's— if 2 are contributing to different ROI, it does not matter that they have to be handled with the same ledger, but eventually they will be merged on— or like synced across different ledger. That's right. Yeah.

And then the restriction is that within a single transaction, you really want all of the transit— all the things getting transacted to be in the same shard. And as long as they're in the same shard, that's great. If they cross shards, then, you know, it gets really complicated really quickly. But again, this isn't in Flurry yet. You'll actually see it in our schema. We have the foundation and everything there to do it, but it's not, it's not live at this point.

This is like work in progress. Yeah,

work in progress. That's right.

All right, it's been an hour and 40 minutes. I don't have any other questions at this time. People can hear us on the phone.

Yeah,

okay. And I need to run too. I just want to say, hey, thanks for everyone. We've still got a pretty big audience here even after that amount of time. If you have any feedback or anything like that, let me know. And, you know, I don't know that Flurry's right for you or not. And, you know, if it is, we'd love to help. If it's not, that is 100% fine too. You know, I'm not trying to convince anyone to pick something that doesn't map what you're trying to do. But yeah, any questions, feedback, if you wanna dive into other topics, you know, more later, we can set something like that up. up. But, uh, really, thanks so much for your time today.

Thank you, Brian.

Thanks, Brian.

Take care, everyone.

Thank you.

Thanks, everyone. Take care.

Thank you.