This is a presentation (video/transcript) introducing the problem of evaluating knowledge graphs and proposing three direct metrics—K score, I score, and C score—derived respectively from science of science, information theory, and causality perspectives. The presenter argues existing evaluations (case studies, downstream tasks) are unreliable or hard to implement and reports a human-centered pilot study to assess metric effectiveness and surface evaluation challenges.
Thank you for stopping by. Today I'm presenting my paper the Measurement of Knowledge in Knowledge Graph. So Knowledge Graph has emerged as a comprehensive tools to enable knowledge discovery, knowledge representation, learning learns Knowledge graph embeddings which are extremely useful feature out input for a wide of different prediction and graph analytics tasks. So due to the diverse and subject nature of Knowledge Graph, Knowledge Graph evaluation becomes an important and open problem. Existing evaluation methods are often based on case studies or downstream tasks like information retrieval and question answering system which is either lack general reliability or hard to implement. To address the challenge evolved, we propose three direct metrics for the nudge graph, the K score, the I score and the C score derived from the science of science, information theory and causality perspective respectively. We propose a human centered approach to evaluate the effective of the metrics. Through a pilot study we share insights on the complication of human centered evaluation and motivated future work.
