Intelligent systems and knowledge graphs

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by Digital Ocean, the simplest cloud platform out there and we're excited to share. They now offer dedicated virtual droplets and unlike standard droplets which use shared virtual CPU threads, their two performance plans general purp CPU optimized They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, cicd, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers or active front end web servers that need to be full duty CPU all day every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive starting at 40 bucks a month. Learn more and get started for free with a 100 credit at.co changelog again.co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're @practicalai FM. And now onto the show.

Welcome to another episode of the Practical AI Podcast where we make artificial intelligence practical, productive and accessible to everyone. I am one of your co hosts, Chris Benson. I am Principal AI Strategist at Lockheed Martin and with me today as usual is my co host Daniel Whitenack who is a data scientist at SIL Introduction International. How's it going today Daniel?

It's going great. It seems like the past week or so has been the week of messy data for me, so I've been dealing with a bunch of missing rows and weird data issues it seems like for the for the past week, which maybe that's like typical for every person in AI and everyone's like oh, that's my week every week, but seems particularly to have hit me this this last week. But what about you? You're at GTC, right?

I am. I'm at Nvidia GTC, which is their GPU technology conference in Washington D.C. it's going on now, although right now I'm hanging out in the hotel room so we can do this. But a lot of fun. Came to Washington at the beginning of this weekend for the Alpha Pilot race and you know we've had recent episode on Alpha Pilot and that was the second of four. Super cool doing that and had a lot of fun. Did some various things on stage. And then today at gtc, I've got a session coming up that I'm leading. It's kind of a fireside chat where I'm kind of both moderator and panelists together with a couple of other really, really smart people.

Yes, that sounds great. I hope that maybe some of that will be available at some point where people can access it.

Yep. I think they put it all online afterwards.

Awesome. If you want to follow up on that or interested in other things related to Nvidia, you can definitely connect with us on our Slack channel. If you go to changelog.com community, you can join us on a public Slack and or on LinkedIn and ask some of those questions and follow up on guests and all of those different things.

Well, today we got a treat. We have a guest by the name of James Fletcher, who is principal scientist at Gracken Labs. And I think we're going to talk all about intelligent systems and knowledge graphs in the in the minutes ahead. Welcome to the show, James.

Hi guys. Thanks so very much for having me along.

So I noticed on your LinkedIn as we were prepping for the show, it said a couple of things and one of them is a little bit the first one it said is it says that you're presently leading research on machine intelligence and cognition at Gracken AI. But it also, and anyone that listens to the show much knows I'm an animal nut. I just own that moniker. It says that you are an entrepreneur with a background in computer vision for automated veterinary diagnostics. And I just before we got into the main topic, I just wanted to ask you about that. If you could take just a second as a tant and tell us what that means.

Yeah, absolutely. So that was quite a fun project. And that was my first foray into machine vision, which actually started when I was studying. I was studying general engineering at university and ended up in this specialization in machine vision. And I really didn't see that coming. I always thought I was going to head towards mechanical engineering or something like that. And then when I saw the capabilities that were coming out in machine learning at the time, I was like, okay, wow, this is really good stuff. This is disruptive, right? You can really do something new with this. And no one's using this. This is clear in industry and studying under Professor Andrew Zisserman at the time, who's quite a big name in computer vision. And we got on well and coming out of that course, I said to him, you know, is it okay if I look at actually commercializing some of these algorithms? This stuff is clearly enough to warrant a whole company around it. And so off I went and started doing that. That was actually a family business. My dad is also an engineer. And so the two of us decided, you know what, actually, let's give this thing a shot.

How was it? Because I know like, the transition of research out of university into the commercial world can be kind of an interesting journey. Was that awkward in trying to convince the right people?

That's a good summary of the journey.

Awkward, you mean?

Well, no, I wouldn't say. No, I wouldn't say it was awkward, but we weren't knowledgeable on IP and all of that kind of thing. But I mean, at the end of the day, it was released open source by the university. That was actually really pretty trivial. No. So, but that actually formed, that was an interesting conversation also because it had been implemented and released open source in Matlab. But you know, that wasn't actually commercially useful to us. So that was a rewrite job from the start to put it into Python so that we could actually, you know, productionized that. And then it was really happenstance and things that put a lot of things together for us. We, we had these generic algorithms, so we wanted to find a place to use them and as a family, actually there's a hobby farm involved here, which my parents have, and we happen to have connections with the veterinary college nearby. So we went to them and we said, you know, we need a vertical, we need a special, a specific task that we can hone in on to actually, you know, prove the usefulness of these algorithms and what they can do. And so we were looking at veterinary science and they said, yeah, that's exactly what we need. We don't have anyone who's actually being able to help us at the university do this stuff at the moment. So we launched this whole research effort with them. What was interesting actually, as that developed was, and this is a lesson in being an entrepreneur, I guess, is that the core value of the business actually moved sideways from the AI algorithms that we were working with, from the machine vision and into the actual hardware and robotics that we needed to actually fully automate the process. Because it's all very well having a machine vision algorithm that automates, you know, the skill of looking through a microscope, but if you don't have a machine that puts the microscope slide on the microscope, essentially, right, I mean, that's, I'm really simplifying it. But I'm sure you got the idea then, you know, how many samples can you actually run? Like, what's the actual improvement? You get to that whole system. And so actually, that was the area that was much harder. Once you had an image on a computer, you were kind of laughing. But getting to that point was actually a little bit. A little bit more tricky. But yeah, the end goal was actually trying to control parasite burdens in animals, particularly grazing livestock. But the. But that translates sideways actually into human health, because rough statistic is that 2 billion of the world's population actually has this parasitic worm infection. There's a number of different reasons why you might want to work on this particular problem.

And there's a lot of samples to run.

There's a lot of samples to run. Exactly, exactly. You hit it in a nutshell.

Well, that's pretty fascinating. And I just. As a way to close that off, I run an American nonprofit charity called the Animal Institute, which brings technology like AI and computer vision and such to solve problems in animal welfare. So if you ever have any interest in discussing these topics further, I definitely have a playground to play in.

Well, absolutely. Sounds like we should definitely go there.

I was just thinking while you were talking about it, I mean, the application is definitely interesting and valuable, but I also think it illustrates. I get asked all the time, and maybe you do as well, like, what should I start working on to get into machine learning or get into AI? What kind of problems should I start looking at? And I think, like, the best thing that you can do is start working in an area where you have some connection or where you're passionate about. So for you, this was kind of a connection between what you studied at university and worked on in research along with your family and engineering, along with, like, this hobby farm and the connections you had with the veterinary school. So it made a lot of sense to go into that vertical. So, yeah, that's what I think, you know, people should consider is trying just try something out that you're passionate about, because those are usually the things that you stick with long enough to learn and to experiment and to level up.

I totally agree with that. I think that's a really good point, because what you're. What you're really saying there is that you will exceed yourself better in things where you are motivated. Right?

Yeah, yeah, definitely.

Not just learning machine learning, but everything. So if you've got that motivation, the more motivation you can summon and put in the one place, then, like, absolutely, you'll. You'll double down on It. Right. You have the passion will get you through the hard times. Right. When you're missing those rows in your data set. Right?

Yeah, yeah, for sure. Thanks for the extra motivation this week.

I was going to say this has turned into completely a motivational show. Totally unexpected in this area and we haven't even hit the main stuff we were expecting to talk about, so.

No, there you go.

Well, speaking about that, I mean like how do you get from robotics and microscope slides to knowledge graphs? What's that kind of journey like?

Yeah, well, unfortunately I don't have some twisting rollercoaster to tell you, only that what I wanted to move out of doing the technical work on that project and I was looking around for the next challenge. I suppose one of the things that I really like to be is sort of like impact driven in terms of the choice of where I wanted to work. I wanted to see something where you can, you know, where you get that value actually disposed. And so you could see that project was the same like you had, like you could see where you were going to actually make some impact. And looked around at all the roles and had this really great conversation with Heikal Prabadi, the CEO here at Kraken. And we were a really over excited conversation when we first met where he was explaining to me all of the ethos about Kraken and the vision that the company has. And I was pretty sold to work here straight off the bat from that conversation. So really just a pivot. His ethos is to take on people that have demonstrated themselves within the scope of what they do. Not necessarily that they have to be people who've worked on, you know, knowledge graphs or graphs at all in the past. Right. So he's very open minded about which field you're coming from. Coming from robotics himself actually. So there's a bit of a resonance there. Cool.

Well, maybe you could just define. So if I go to like the Gracken website, which is Gracken AI, we'll put it in the show notes. You talk about a couple of things which you've already mentioned and I think it'd be great to kind of dig into those terms a little bit more. So one of the things you mentioned is intelligent systems on the website and then you just mentioned knowledge graph. So maybe you could start out by just kind of sharing what Gracken means by intelligent systems and what sorts of intelligent systems people are developing out there.

Yeah, absolutely. So the terminology that's being used at the moment is an interesting and kind of hot topic of Its own. Naturally you're going to get a grack and bias spin while you're talking to me. But the general ethos, I think it's better to start with knowledge graphics.

Okay.

It's good if we also start with how we describe Gracken and what that does for people. Right. So Gracken itself is a database. Right. And typically when you're talking about knowledge graphs, that's what you're talking about. You're talking about some sort of actually large store of knowledge. Now, knowledge graph itself is essentially totally synonymous with knowledge base, which would be like the mathematically correct terminology that's been abused on the web a lot for other things. So we tend to go with knowledge graph. It's a little bit sexier and also immediately gives someone with. Without experience in knowledge basis an idea of the shape of the data, which is, you know, graph in the computer science sense. So, but what do we actually mean by like knowledge graph as opposed to just graph? So there's all sorts of different graph types of format all over the place. But what we're trying to build here is a system which takes you from. You want to make that leap from a graph full of data to a graph full of knowledge.

Yeah, I was just going to jump in and say, I think that's maybe the part where I struggle. I think a lot of people have dealt with databases and maybe some people are familiar with graph structured data. Like, oh, I've got this node which is a person and another node which is another person and they're connected by, I think the terminology is some edge that is like this person is friends with this person or something like that. When does a database or graph data go from being just a database to being a knowledge graph? What's the idea around that?

Yeah, so the idea is that the way that we built the system up is how can we capture all of these different kinds of knowledge. Right. And so what we have is we've built a knowledge representation system. Right. So Grack in itself is actually everything that's in grack and is actually built on top of a graph database. That's actually the start of the innovation, I think, that helps people understand what we're doing. So we started. If you start with a clean slate and you're going to build a project, we started with a graph database and then we built other things on top of that. Right.

Can you talk a little bit about what the difference? When most people probably think database, they're probably thinking of a relational database kind of more the classical Postgres and those kind of databases, as you explain here, could you differentiate between what a graph database and a relational database are so that people can, if they're not already familiar, they can kind of make that jump.

Yeah, exactly. So as we were already talking about, right, like, so we've got a graph in the, in the computer science sense as opposed to in the like the XY plot sense, in that we've got nodes and edges interconnected, right? So in a typical graph, a node might represent anything. For instance, I like your example, from one node which is a person to another node which is a person, you'd has like, has friends as the label of the edge in between those two nodes, right. So what we can do is rather than a relational database forces you to store everything in tables, right? That's what you've got. You've got a set of filing cabinets and each file in those respective cabinets may have a reference written on it that links you to a file in another cabinet, right? That's the kind of structure of the data that you've got available to you. But what we find is that, well, as soon as we're dealing with data that's more representative of a network, then dealing with it in those kind of tables gets really messy, really, really fast. Because as soon as you've got like one thing which is connected to eight other things and eight different file cabinets and all of those are also connected to eight different things, you know, you get into a big mess with that.

Starting structure doesn't scale well there across laterally.

Exactly. Well, the idea is that when you're actually trying to build some kind of application with those things, the complexity that you as the user of the database has is enormous. Right. Suddenly you have to try and control this structure that wasn't really designed for the data that you have. So then you go a layer up and you say, okay, now I need a graph structure to actually more naturally represent my data. Right. And so that's where graph databases are kind of born.

And when you say kind of more naturally, other than that it reflects the data, the relationships between the data very accurately. Are there any other advantages? Big going graph. If somebody is trying to make that decision today and they're looking at that, maybe they're looking at grack. And is, what are the benefits of going graph database versus relational database?

I mean, I think you kind of say it in a nutshell in that the idea is to be able to naturally represent network data as it is.

Is it easier to get to the data though? In that way and not having to do giant SQL, Classical SQL.

Exactly right. And we go a level more natural again when we actually come to the knowledge graph element that grack and builds on top. Right. So once you've got your data in like a graph form, now you want to be able to concisely refer to and search your data and reference what you're looking for. Right? So the major innovation, I would say there's two major parts that you need to understand to figure out what gracking is and why it helps you. The first thing is we've got this knowledge representation system and we have this flexible model. I don't think we want to talk in like technical depth on, on all of the intricacies of that.

Yeah, yeah.

You can basically make entities, relations and attributes. We make these three things, these three kind of characters, right, that you have in the story of building a grack and schema. And entities are, you know, things like people, things like companies, even things like abstract concepts in the world, right? But then when, when someone references an entity, you immediately know roughly what they're talking about. Relations are the kind of glue that sit in between these things, right? So that's what you would use as edges in the graph that we were talking about before. Right? But relations are probably the most standout concept in terms of what we do because these relations allow you a huge, huge volume of flexibility. They say that not only can I have a friendship between two people, right, and say that person A is friends with person B, but I can say that they're also friends with person, circumstances, person D, person E. I can do that with one relationship. We used to know that as an edge. So in this case, what we're saying are these relations are hyper edges, right? And you can see there. So immediately we're starting to introduce like big concepts at the low level of the structure that we define, right? We say basically we want to upgrade how you can represent your domain. We want to give you this toolbox which we're calling the schema in Gracken, that lets you model your domain in all of the complexity that it has, right? And that then means that you've now got this format, this structure that can govern your data, that can look after your data for you. It can make sure that you haven't done anything that's logically invalid. It can make sure that everything is cohesive within your database. So when you start adding facts, right, you now know also what the context of those facts is, because we heavily label all of the all of the elements that go into the graph, for instance, you could insert a company, a charity and a university. All of them. All of those types that we've described that we can describe have inherited from organization. Right? What that now means is that when I want to search my data, I can search for either companies, for charities or for universities, and I could search for those individually or I can just ask more generic questions and I could say, just tell me about organizations in my data. Right. And so what we're trying to do there is to get this really natural way to actually interact with your data so that you're using your own domain terminology to actually access what you're looking for, rather than having to say to sort of imagine what are my nodes and what my edges in my graph and how do they fit together. Right. Instead we try and bring that to the user and reduce what the burden on them when it comes to assessing what's going on in their knowledge graph.

What is up practically I listeners, we're working with Infinite Red to promote their free AI mini course. It's called AI Demystified. Learn more. Enroll@learnai.inf RED. This free 5 day mini course is a great introduction to the most important concept types and business applications for AI and machine learning. Each day of the course includes a lesson, a quiz and an assignment to submit your learning. And after you've completed the course, you'll also get a certificate of completion for your LinkedIn profile or for your portfolio. If you've been feeling lost in the world of AI and hearing lots of buzzwords, then by the end of this mini course you'll be able to speak intelligently about AI, AI and machine learning and their practical business applications. Again, this course is completely free. Learn more and enroll at learnai.infinite.red Again, learn AI.infinite.red.

So jm, I appreciate kind of where the conversation has landed in that there's natural ways of representing your data and that can be modeled well on top of a graph. I've tried kind of graph databases in certain scenarios with more or less success and some have been really useful. But something I always find is like it seems really hard to build a quote unquote knowledge graph in the sense of kind of developing your schema can be hard because you may know what entities you have, but might be multiple ways to represent them. Or you may have just a bunch of unstructured data and you're not totally sure what entities to choose. So how do you recommend if people are interested in creating this sort of representation of knowledge? Where should they maybe start thinking about the data that they have and how to develop a schema?

So that's a really great question. I don't have a short answer, but essentially that has been a huge part of what I've been doing here at Gracken and what we do overall with members of the Gracken community. We try and help people to actually understand the principles of what is an entity, a relation and an attribute. How do they best fit together? And actually what's super interesting about that is that that's a really great meeting of philosophy and technology, which I found incredibly interesting. And that essentially my thoughts on this is that we now see knowledge engineering and knowledge representation as entire careers that are actually coming around now. Right. That you actually have someone who's, you know, a specialist, an ontologist, I've also heard them called. Right. The body of knowledge of the best way to do this is not yet set upon. And we have our own ways of doing that here at Gracken. And those ways and how we think that things should be done informs the design decisions that we make in the language that we provide for the knowledge graph. So at the moment, it's actually been on my to do list a long time to actually write some best practice for knowledge representation and building your schema in Gracken. We have snippets here and there and we have examples here and there, and it's very difficult to give really generic guidance, but we do have some that we would give out that's a little bit long winded for here, but maybe we can link to that in the future.

Yeah, no worries. I actually want you to extend that just a little bit. I'm kind of curious, what can you do with a knowledge graph that you would not be able to do if you didn't have one? As you're talking about kind of design and thinking about, you know, what best practices are, what comes to mind.

So the main thing that anyone who's interacted with me in a professional context will know is that what I harp on about is trying to get to the point of true to domain modeling. Right. What I really want is to see people building a knowledge graph where they start with a schema, where one person who builds the schema could show it to their colleague and their colleague will immediately understand what elements of data are where in the knowledge graph.

Right, that makes sense, yeah.

And just to clarify, to make it super clear for listeners, when you're talking about the schema, you would basically like, we Gave the example before of like person is friend with person. So like there's a person type entity in this knowledge graph, but there could also be like country type entities or organizations or like different metrics, websites, resources, all sorts of things. That's the sort of schema or ontology that you're talking about, right? The definition of what things are we going to put in our knowledge graph and how are we going to label them? Is that the best way to think about the schema?

That is absolutely correct. And what I think is also really nice is to make some analogies to oop. So your object oriented programming, right? So anyone who's familiar with oop, and there's a lot of people out there, I imagine you have quite a lot of listeners who are familiar with OOP then. And what we're saying here is we're defining the class. We define a class, right? And those are our schema elements. And then when we actually insert data, we're inserting like instances or instantiating objects.

Of that class and just quick interjection for those who don't know what OOP is. He's talking about object oriented programming. It's a technique for representing real world concepts in code as well. Just keep going. I just wanted to let anyone know that didn't know that.

Yeah, yeah, yeah, absolutely. So, so the idea is that all of the elements that we would have, as you say, we have this like this schem and you can, you can update that, that over time. But that is the map for your data, right? That tells you what things are present in our knowledge graph and how can they be connected to one another. So for instance, we can immediately say in that example where you had like a person entity and also an organization entity. We can then also define the friendship relation that you talked about, right? And we can say, okay, a person can be in a friendship with other people. That makes sense. Can a person be in a friendship with an organization? Now maybe that's philosophically debatable, but I would probably say the answer is no. In which case that should not be permitted by your schema and you should write a schema that disallows that. And what that means is that takes some weight off your shoulders. Because when someone tries to add some piece of data inadvertently that says that there's a friendship between a person and an organization, then gracking can automatically reject it and say, no, that's rubbish, that can't exist.

So I think maybe there's a bit of a misconception and maybe parts of Time that I've been thinking about knowledge graphs and maybe other people too, where there's kind of this sense that, like, when you hear about, oh, Google's knowledge graph or something, it's just like information is all over the Internet. And like, if you create a knowledge graph, then all that, you just suck in all that information and then you automatically know a bunch of stuff. But there is actually a lot of work in terms of developing a schema that represents the types of things that you're interested, the types of knowledge that you're interested in. It's not just like automated thing where you just like crawl a bunch of websites and then you have a knowledge graph on a certain subject. Would that be accurate?

Yeah, I mean, absolutely. You can go at it any number of ways that you want to. So you can start trying to scrape information from the Internet, but, you know, the quality of the information that you get may not be that high in terms of, you know, kind of can I ensure validity of the kind of facts that I've pulled from that? And there's plenty of people that are trying to do that. So that would be automatic, like entity recognition and this kind of thing. Our focus is more on building these things from the ground up so that, you know, if you. Someone's got proprietary data or they've got a particular data set that actually they can realize an enormous amount of extra benefit from just managing the data that they have very carefully, rather than maybe trying to augment it with just all data from the Internet. You know, probably you can take a more targeted approach and just bring in elements where you're fairly aware of what that information even is. Right.

So wanted to kind of delve into a different area, given that we're an AI podcast. And so I wanted to ask, you know, how is artificial intelligence related to knowledge graphs? And are knowledge graphs a source of data that might be available for AI models, or is there some other connection there?

Yeah, I mean, where to start? So, I mean, the way we see it is that knowledge graphs are going to be central to the effort towards, well, intelligent systems, as we put earlier. Right. So that's our nice way of trying to avoid using AI to make systems more intelligent than they are today. We want to empower them with as much as we can. And so the idea here is to, you know, much of the world is still using relational databases. And as we talked about before, you know, structurally they present themselves, present us with some challenges when that format isn't natural. So instead what we want to do is we Want to actually be able to capture the full complexity of the world, actually capture all of our knowledge in one place and then be able to present that to, for instance, learning models for them to learn over it. But what we also provide is actually the artificial intelligence of the 80s and that is automated reasoning. So what we have at Gracken built into the open source core product is a automated reasoner that allows you to infer new data based on the data that you already have and sets of logical rules that you know must be true. So this is super interesting, right? Because in the day to day we all use our deductive logical skills any number of times and we essentially just don't notice because it's so second nature to us. But if you actually try to point to any tools that anyone technical is using right now, about the only thing that people have heard of and they did like a week on it at uni or something, is Prolog. That's about the only tool out there for logical programming. Right. And it sounds like something computers should be able to do easily. Right. Like a small set of facts and figuring out a new fact based on a rule just sounds like an if else blocks, right?

Sure.

But when you actually try and scale that and make that work and be able to have any number of possible rules that you might want to be able to write and bring that into the database level, that's when things start to get a bit interesting there. Because now we can say when A and B and C are true, then D is true. And what's nice about this is that your database, then whenever you ask for something that fits the bill for D, it's going to give you that regardless of whether or not you ever even stored that in the database.

So I just had, it's almost a tangent of a question. Would you talking about, you know, prologue and using automated reasoning, which was kind of before the days of machine learning as we know it today. And I just want to ask, you know, is there any tie in maybe today? I know you were saying that you're kind of including that in your approach, but you know, today, I guess if we were going to tackle that with the current set of technologies, we'd probably use things like generative adversarial networks and along with natural language processing to try to create things new from what you already had. Is there any tie into that? And just as a random side question, is there any similarity maybe in the two.

Well, great question. So I think our ethos is when you have facts, if you can write a rule that definitively tells you that a new fact must be true based on what you have. Like that's absolutely fundamental, where you can use that, then you should use that. Because why is that true? Well, because firstly, it generalizes perfectly, right? Any new set of A, B and C, and you know that D will be true. And secondly, it's explainable that when you see D, then you can say, well, why did I see D? And the database can tell you, well, because A, B and C. Now what's really interesting, and this is the crossover space that's happening right now is as you said, how do we see that complementing the other tools that we want to use? How do we see that complementing, you know, any other machine learning approach? And so essentially the border for me is to describe it as well. You're either if you were a human approached with a particular problem, you would probably decide whether to use one of kind of two major skill sets that you have. Either how you deduce things and your logic or your intuition. And so essentially what we need is we need to start figuring out, okay, when do we need to deduce things logically versus when do we need to use a machine learning approach which gives us some kind of intuition based on experience. Right. And so that's actually the center of my work here at Grackham is how do we actually build learners on top of a logical reasoner on top of a knowledge graph in order to like get to the next level of intelligence of our machines? Right. How do we make an iterative process between those two that ingest new facts that have been learned and then reasons over them? Or how do we reason over facts and then learn from them? Right. So this is very much an unsolved region and it's super invigorating at the moment to be in that space.

And what do you think are the sorts of tasks that are kind of low hanging fruit for learning on top of a knowledge graph? For example, one thing that comes to mind is question answering sort of tasks or something like that. Are there other tasks that have been explored in AI, maybe in a non knowledge graph way that you think are particularly relevant to explore on top of a knowledge graph?

Absolutely. I mean, as I said, that's actually kind of the whole remit of the research division here at Gracken is to try and fulfill those end user problems. And what are they? Well, actually wrote a whole blog post on all the kinds of problems that we see there. So you're absolutely right. Question answer systems is, I mean that's what that 80s logical reasoning AI systems were all about was, were building expert systems, but they didn't really work because you had to hand code everything. Well, now we can maybe use machine learning to derive some of it automatically. Right? And we do question answer systems. And you see that with Google's Knowledge graph and this sidebar that they have, right. When you type in a search, it may just directly find the thing that you're interested in, not just links. But then besides that, we see a lot of applications in, for instance, well, we can talk about knowledge graph completion. So that's maybe I want to find new links in between elements of my graph that I'm interested in. So for instance, if I ingest a lot of biomedical data, then maybe I want to try and predict new links between a drug and a disease, right. Want to infer new treatments or maybe I want to, you know, enrich my whole graph before I try and make those as well. So I can, you know, find other relations, interactions between genes, proteins, et cetera. Right. But then there's other tasks on a totally different spectrum. So what about NLP systems and computer vision systems when you apply background knowledge to them? Right. Well, as humans, when we approach understanding a person who says a sentence we have behind us, however many years we've been on the planet of experience of hearing people say sentences, we often don't really bring that. We also have more than that. We also have our knowledge of the world. Right. We often hear someone say something and we, we mishear what they say and what they said sounded ridiculous, given our knowledge of the world. And so we correct ourselves or we nudge them and say, did you just really say that? Because that, that doesn't like, align with my understanding of the world. That's what we hope that then the knowledge graph can do. And we've got, you know, I've had a number of conversations with people who want to improve, for instance, their company's customer service platforms where they know the body of knowledge, they know quite a lot about a customer, they know a lot about their products and the kind of things that they offer. And you know, if a customer says my connection is broken, can we immediately infer what they're talking about? Because we actually know products that customer has, okay, they have a home broadband connection with us, so they're probably talking about that. Right? In machine vision, as we've already talked about a little bit from my past, then often we just present a learner with a flat image. We try and get it to guess what's in the image based just on the pixels. But you know, again, if the learner starts to see things that are nonsensical in the image or things that go that are often seen together, that would be a, you know, a big help for it to be able to understand and identify when it might be wildly wrong based on the other things, the other context, the surrounding context of the of the problem that it's trying to solve.

Foreign. This episode is brought to you by Brave. Big news from the brave team. Version 1.0 is official. That means our favorite open source, privacy focused, blazing fast browser is ready for prime time. Their brand new iOS app landed just in time for the announcement and the Brave team is celebrating by granting 8 million basic attention tokens to the community. That means when you download the iOS app, you get 20 bat absolutely free. Put it to good use by heading to changelog.com hitting the triangle icon in the upper right hand corner and flipping us a tip.

So you started to get into a little bit of the the details of where you think certain tasks like computer vision or other things could be augmented by a knowledge graph. And it seemed like in some of those cases it was a matter of like, okay, you have the image and you have this other information that goes along with the image that helps you reason about the image or predict something. Is that where you see kind of the near term of knowledge graph, augmented AI? I don't know what the proper term for that is, but is that kind of where you see the near term? I know that there's also people exploring or doing AI with graph structured data itself, rather than just kind of extracting features from the graph as new features in a model, but actually using graph structured features or sub graphs or other things in AI models. Are you familiar with that at all? Or how do you see, maybe as a person who says, okay, well this sounds cool, I'd love to try to augment some of my AI systems with knowledge from a graph. Where might they start looking in terms of methods and next steps?

Right. I mean, great question. So I totally agree. What we don't want to do is just stick with the status quo of sort of taking essentially sort of square shaped data as inputs to machine learning pipelines. Right. That's like the status quo at the moment. Right. Our data is stored in these filing cabinets. And so what do we put into our machine learning model while it's data that looks like filing cabinets? Right. And what do we get out? Surprise, surprise. Right.

Yeah. And I think it's probably confusing to people. Sometimes it has been for me, where like TensorFlow talks about a graph, right? So it's not a graph of the data, it's more of a graph of the computation and how it's executed on a certain, certain architecture or the logic of that computation. Whereas what we're talking about here is actually data that is structured like a graph. Being processed through one of these systems as a graph would be different than just putting a tensor in, right?

That's absolutely true.

Yeah.

So that's one of the fundamentals that makes learning over, well, anything except just a matrix or vector representation difficult, is that all the frameworks are set up to take those things in. And as you say, in the case of these pipelines, the shape of the processing is a graph, but we don't really need to worry about that compared to the input and output. And as you say, what we're here, we're saying is, what do we do? How do we move from these square inputs to something else? So that's actually a big body of work that I've been doing over the last year, has been looking at one of the approaches that have been done around that. And some of the first approaches, which is still quite common, is to do, for instance, walks through the graph. Like I'm interested in some particular entity in my graph, so why don't I start there within my graph, and then just walk randomly and see what I encounter, record what I encounter, and then maybe I use that as like a row in a vector or something and feed that into my model. That's one way of doing it. Right, but you're kind of hoping for some serendipity there. You're kind of hoping that I'm going to encounter things in my graph that are important. Right? Because I'm just sort of walking around random. I'm literally randomly walking is what is the approach through the graph. So, okay, so the next thing. And so from this, there was a really nice piece of research came out of Stanford. They called their paper graphsage, or at least their approach was called graphsage. And we actually implemented that here over the knowledge graph. And the idea of that was to essentially not just take these single walks, but to actually look at all of your neighbors. Take a subset, a random subset of all of your neighbors, but then also look at their neighbors and their neighbors and their neighbors and sort of have this more like spider web shape of the graph that you would analyze. Right? And sort of in some way, without going into all of the technical detail that basically roll that information Inwards towards the point the entity that you are interested in. So you kind of gain some information as you move from that, like the outer radius of a, like outer circumference of a circle, like inwards. And that's also really nice. Right. So what that's also doing is still kind of putting our data into a box shape because you're still dealing with a tree now. So we've gone from, from a line, which was the walk, to then a tree. And we still didn't find what was really difficult about this. So we tried using this. But what it doesn't manage to capture, say, say we are trying to do something really difficult. We're trying to find a new drug for a dis. To treat a disease. Now if we try and do this, if we just look at like generally what does a drug look like and what's nearby to it, and also generally what, what does a disease look like and what's near to it when we then try and match those two things. We haven't actually looked at any of the common connections that exist between a drug, that drug and that disease specifically. Like, we haven't actually figured out what are the, like logically, what are the paths that actually connect these things. We should probably be interested in in those. Those are probably like the most important features in this graph. Instead, we've just looked at roughly what they look like. And then you end up with just like some generic answer like paracetamol treats lots of diseases because lots of diseases exhibit pain. Right. So what we want is again a more targeted approach. And that leads us to. No, we have to do the hard thing. We actually have to learn over a graph shape. Right. We actually have to take in graph data.

Yeah. So I'm kind of thinking about natural language processing because that's the world I live in. And some of what we've learned recently is that it's very useful to have your algorithm learn the proper representation of text, taking into the context of context around just like a single token, for example, in order to actually learn a good representation of text for a certain task. It sounds like what you're saying is it would be useful to do similar things for graphs in that we need to learn how to represent graph structured data in a neural network. Because it might not be like if we just take all the nearest neighbors and put them in kind of standard row structure and use that as a representation, then we might miss that. Actually the predictive thing is beyond the nearest neighbors. Right. And like a bunch of links away. Even Though it's not a nearest neighbor. That's, that's like the thing that's indicative of the thing that we're trying to predict. Is that kind of along the right track?

Absolutely. What I see you describing there in NLP is definitely what we're aiming for here. Right. And not just in graphs, but I think in the industry in general is where we're now seeing like beyond curve fitting it's called. Right. And like how do we move beyond where we are right now to a point where the machine is actually understanding, it actually learns to understand what's going on. So like we already talked about that with like NLP based on a knowledge graph. It understands the context. You were just talking about that there context in machine vision problem. Also understanding the context of what's actually in the image. All of these things mean that the learner can not just sort of learn by rote or learn by exact examples, but can actually understand what's going on. What's really interesting in graph is that you have exactly that you might have like one particular feature that you find. Like if I see some particular thing that's in some particular way related to what I'm interested in, that's a huge indicator. But you might also just see a general structure that occurs that when the, you know, I want to have these five elements, these five entities all connected together in a particular way, they all have particular types. That is a very typical structure for a really effective drug. Right. That those combinations come up again and again but in like a generic sense. And maybe we want to learn that we want to learn some kind of structure. So then what we were faced with was we were faced with the problem of okay, no, we actually need to learn over graphs. And to our luck we're not, you know, we don't have the budget to do like, and the manpower to do these huge research efforts ourselves. But neighbors over here in London, DeepMind released a paper last year and they also released a library to support what they were doing where they've generified a lot of the concepts of, of graph learning and how to do learning over graphs in this really neat way. Given they were acquired by Google. I mean it makes sense that they also figured out how to do this in TensorFlow. So what they've got there is a pipeline that now actually lets you input a graph into TensorFlow as the data and get that same graph back out as an output, but with updates made to every element of that graph. So that means that essentially what we can use is we can Use that as a little toolbox that allows us to perform any number of different tasks over a graph structure. And obviously we've tailored that here at Grackham to work over the knowledge graph. But what we can do is we can just carefully frame the kind of problem that we have so that this toolbox can help us to solve that.

And is that the graphnets library?

That's exactly the one. Yeah, that's the one.

Okay. Yeah, we'll definitely link that in the show notes as well because it seems like they have a good usage example and notebooks and such that people can play with that.

So you've totally won me over and I'm looking forward to jumping in and playing with this, and I know Daniel is too. Could you start walking us through what it is like to actually build a knowledge graph with Kraken? And, you know, what do you need? What languages do you need to know? And also I noticed on the website you talk about. Is it Gracul? Am I pronouncing that right? And if you.

So that's Grackle. That's Grackle.

Grackle. I'm sorry.

No, no worries. So, yeah, so yeah, I can give you the whole, like, overview of what you would do.

Right, Fantastic.

Actually, to close down what we were talking about just there, the whole learning approach that we've been building and all of the research that we do on top of knowledge graphs. Right. I'll emphasize that is all. We release all of that as code available via our GitHub. Specifically, we have a library called Kglib. So that's our Knowledge Graph library for machine learning. So kglib is the center of those projects and the main one that we're running right now is Knowledge Graph convolutional networks. So that's how do we apply these learners on top of both the Reasoner and the Knowledge graph shapes data. The starting point is how do you actually get a knowledge graph? Right. How do I actually get my knowledge graph together? Now, the components that you have there, as you pointed out, is something we should start with. So you have Gracken itself, right. So Gracken Core is released open source. You can Download that from GitHub or install it with a package manager. And that's a database which is going to run. You can, you know, install that on your local machine and get it up and running or put it in the cloud. And so you need that backend service running. Now when it comes to actually accessing that, we have three officially supported drivers at the moment. We have Python Node JS and Java. So we make sure that all of those are up to date and working with the latest Gracken. What's really interesting there actually is the communication protocol between those clients and Gracken is called grpc. So that's something from Google, Google's remote procedural call that has replaced using REST services. So what's really nice about this, the actual end goal that that gets you to is it means that when I'm accessing the database in Python, with Python, I get to actually use native Python functions. All I have to do is import the package that talks to Gracken, import the Gracken client in Python at the top of my script, right. And then I can just instantiate a communicator that will talk to Gracken and make queries to the database just out of my native Python. I can just launch them straight from my application. And it doesn't feel like you're talking to a database anymore, right? It just feels like you're making function calls which comes back with information that's pertinent to your knowledge graph.

That's great. And would you use that, so would you use that client tool to help you build your knowledge base? Like let's say that I have a bunch of text data and I'm like pulling entities out of it that I. Or classifying that in a certain way to store it as a certain type of entity. Would I kind of be doing that in Python and then push that to Gracken via the Python library? Or they're like bulk upload techniques or like ways to get data, let's say from relational to graph. What's the sort of range of what people do?

Yeah, absolutely. Great question. So basically you're absolutely on the money. The idea is that we give the users these clients in their native language because that's their strength, right? We already know that they know how to speak that and they get all of the freedom that that language offers. And then the way that you're actually interacting with Gracken is through Gracken's query language, Grackle, right. You can probably see where the name comes from, right. So the Grackle's got this query language called Grackle and the idea is that that's this really concise, really expressive language. But then what you would do is that is your one stop shop for how you actually talk to the knowledge graph in terms of what your intentions. So if I want to either retrieve something, then I make what we call a match query. If I want to insert something, then I use an insert query. And if I want to, to wherever I see a particular pattern, insert something that's a match insert. I'm sure you get the idea, right? So you have all of these different ways that you can read and write from the database, and you do all of them in the same way through your application. You just, you know, you call, ask the client, you say dot query, right? And make it. Make this query. And then the response you get back will be the answer, right? Either you inserted something or read. Now then, what we've got, we've got a number of. We've got a repository of examples so that people can have a look on there. You know, very typically people are, as you say, they're migrating from either SQL data or from CSV data, in which case it's a matter of just writing what we call an ETL pipeline. So something that will just traverse over all of that data that you have and make the appropriate queries in Grackle to get that data shifted over into Grackle itself. Now, one of the questions that people ask me really often and definitely comes in on our community Slack quite often is, can I like, automatically build my knowledge graphs? And we kind of talked about that a bit earlier in the call. The problem is that, like, it's possible to automatically ingest a relational database into a knowledge graph, but the problem is you just end up with the same structure that you had in your relational database. But in the knowledge graph, you know, you still end up with something broken because. Because you need to apply that human understanding that you have of the data that you have in these table formats. You need to say, what's that actually mean? What does my domain look like? So what you do is you first. Well, it's an iterative process, of course, like a lot of engineering. But you're going to start out by saying, here's my schema, here's what I think my domain looks like. Okay? Now when I go over this file, what parts of that schema can I infer from the particular row I'm dealing with right now?

So I guess if somebody wants to get into this, I know we're both very excited about it, and I've learned a lot that I didn't know before the conversation. Where can they go and learn more and actually start digging into using Gracken themselves and Grackle. Any specific links that you want to recommend?

Well, we have the docs available on our website. People seem to think those are quite fun. And we also have so there's quite, there's also some in depth examples there on for instance, how to do data migration into Kraken so that you got, you know, get that knowledge, knowledge graph up and started so you've got something to play with. We then have an examples repository on our GitHub and also for those who really like to jump in at the deep end, then the KGLIB Repo is quite a good place to if you want to see immediately from the top how you're going to then do the machine learning over it. And then I suppose the other thing to majorly encourage is to check out our blog. So that's blog Kraken AI. So we have a lot of stuff there that will give people an idea and give them a flavor of what you can achieve with the knowledge graph and how succinct it could be to get you motivated to actually move your data over and give it a try.

Well, James, thank you very, very much for coming on the show and just kind of schooling us in all this. It's been really fascinating and we appreciate it. So thank you and we'll talk to you soon.

Thank you very much for having me, both of you.

All right, thank you for tuning in into this episode of Practically I. If you enjoyed this show, do us a favor. Go on itunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you got to do. Share the show with a friend if you enjoyed it. And bandwidth for Changelog is provided by Fastly Learn more. Com and we catch our errors before our users do here at Changelog because of Rollbar. Check them out@robot.com changelog and we're hosted on Linode cloud server servers. Head to lino.com changelog check them out. Support this show this episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@changelog.com when you go there. Pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.