Generative AI in the Real World: Measuring Skills with Kian Katanforoosh

Foreign.

So today we have Kian Katan Furush, CEO and founder of Workera, which you can find@Workera.AI tagline is verified skills Intelligence. So, Kian, welcome to the podcast.

Thanks for having me, Ben.

So I guess let's start at the beginning for our listeners who are not familiar with this domain and space. So it looks like on your website you have a catalog of skills and a way of verifying skills. So can you give us a sense of how big this market is for skills verification?

Well, it's a newer market that is extremely large. I would say that anything that touches skills data today is on the rise. Oftentimes people, when they think about skills assessments or measurements, they think about, I've taken the SAT or I've taken the TOEFL exam, or I've taken the GMAT when I'm applying for my MBAs. You know, companies that build those exams are very large. And. And when you extrapolate that, let's say, concept of university admission to someone's career or an employee at work, you realize that there is countless points in time where they need to validate their skills or to demonstrate a certain competency, whether it is before getting a job, where they need to qualify, after getting onboarded, and they need to be matched to a project, when a manager wants to mentor them and orient them in their career, when they need to choose where to spend their time learning or. Or what book they should read or what podcast they should listen to. Measurements come very handy in those situations. And so because of all these touch points, the market for accurate skills verification is extremely large.

So, roughly, Kian, what's the breakdown between B2C and B2B?

From a worker standpoint, it is exclusively B2B and federal.

However, there are assessments that are also focused on the B2C. Workera has free assessments for consumers so that they can go and test their skills in artificial intelligence, in cybersecurity, in software engineering, etc. But the business model is exclusively enterprise and federal.

But there are people who are signing up. I guess one can imagine something like Duolingo. It's like a skills assessment, right?

Absolutely. Duolingo offers an English test that is growingly popular and that's considered a skills assessment as well.

Yeah, and so obviously this is an AI podcast and we will discuss the role of AI in skills assessment and skills identification. But rewind, let's say five years ago, obviously there were tech companies in this space. Maybe they were using some form of AI. So what were the prior solutions before.

The rise of let's say generative AI in skills specifically.

Yeah, in skills assessment. So what were people doing before? I'm sure they were using some form of technology, right?

Yeah. I would say that historically assessments have been used for summative purposes. Summative means pass fail, high stake purposes. So you're a candidate, you want to get a job, you get an assessment and you don't get a ton of feedback. It's not meant for learning or personal development. It is meant to admit you to that job or to reject you from that job. The same happened in university admission. So until, until maybe work era, most assessments were exclusively built for high stake pass fail assessments. And what we provided to the market and sort of reinvented is the use of assessments not only from a pass fail perspective, but most importantly for people to know where they stand, compare themselves to the market, to benchmarks, be able to make the right decisions in their career or simply to decide what to study next. And that takes a lot of different technology because you can see how you're flipping putting the importance on maybe the company trying to hire to actually putting the importance of what value you're going to provide to the person from a self awareness standpoint, from a time saving standpoint, from identifying the right projects for them or the right jobs for them, or the right courses and classes for them. The purpose is different and the technology is different as well.

And let's see. So obviously generative AI became much more prominent in November 2022 with the rise of ChatGPT.

So what changed in the area in the assessment space?

Many things to create, I would say a few things first to motivate the need for assessment. But.

What changed is that skills change much faster than ever. Now with everybody needing AI skills, people talk about prompt engineering. You need to update your skills much more frequently. And there is a metric that the World Economic Forum shares on a regular basis called the half life of skills. And it is reported that the half life of skills used to be over 10 years. You didn't need to update your skills for a decade. Today the half life of ski is estimated around 4 years. For digital areas around 2.5 years. Which tells you that with the rise of these technologies, people in every job need to update their skills much faster. And so AI has influenced to a certain extent the human resources and education market because of the half life of skill going down. On top of that, writing a quiz is very easy. You know, anybody can write a quiz. Writing a really good assessment is extremely hard. Validity is a concept of showing that what you intended to measure is in fact what you're measuring. And it is extremely hard, impossible to prove 100% validity. But AI can help in a variety of steps in the workflow of assessments. First, before you develop the assessments, AI can help with modeling the competencies you want to measure. Before I decide how to measure, I want to decide what to measure. And making sure that this is relevant to the employee's role or this employee's future role. On top of the competency modeling, you need to create the assessment which comes after. And AI can help streamline not only the creation of an original assessment, but also the conversion of that assessment to different languages, to different scenarios, to different contexts that are industry specific, geography specific, et cetera, et cetera. So that's before the assessment, if you will, during the assessment, when you start testing the assessments, AI can help you with synthetic users. For example, you can resend synthetic users through the assessment and actually collect feedback from those synthetic users, sometimes called LLM, as judges. And that can help streamline the calibration process of an assessment. And then you have all the things that people do post assessments, you know, monitoring the fact that certain questions are going to show up to be bad. You know, just people find them too hard or people find them too obvious, or one of the options in a multiple choice gives away the answer to another one. There's a lot of things that can go wrong. And the ability to monitor an alert with AI is drastically superior to the old way, let's say, where humans had to run models and collect and update, you know, now you can do that much better with a monitoring system with AI, and maybe even you can have an agent that updates the assessments with a human in the loop, which is much faster, you know, so countless places for innovation of AI in skills assessment.

Now, actually, as you were talking there, I was thinking about the start of the pipeline, right? So let's take programming and coding, right? So five years ago, there were already tools that people were using for filtering people out coding tests, right? So coding skills assessment. But now, I don't know, Kian, the nature of coding and programming has changed, right? So I would imagine if you're telling me that I'm going to take your coding test and I'm not allowed to use a coding assistant. Well, wait a minute, when I do the job, I am going to use a coding assistant, right? So in other words, when nowadays when I'm doing programming, I am using three or four different coding assistants to help me. So why should I not be able to use that when I'm doing the assessment?

No, I think you should be able to use it. I think the assessment has to change. If you look at a lot of the previous generation coding assessments, they focused overly on syntax. But do you care anymore if you forgot a comma or a column at the end of your sentence? Probably not, because not only you would be able to debug, but also you will already have used an assistant to code and it would not make the syntactic mistake that you're making. So instead the assessment should focus on other cognitive levels, let's say such as analyzing information, synthesizing information, creating new information rather than purely syntactic details. And so this is an important trend, you know. Yeah, I agree.

Thanks for listening to this episode of the O'Reilly podcast, Generative AI in the Real World. If you want to listen to the full version, please log into the platform@o'reilly.com.