254 - Unlocking the Role of AI in Enhancing Software Performance Insights with Flatiron Software's Kirimgeray Kirimli

Welcome to Listening with Leaders. I'm Doug Noll, lawyer turned peacemaker. I teach executive leaders how to listen to emotions rather than words so that they can become the leaders everyone wants to follow. And I teach those same leaders how to be authentically present, available, and connected to their families despite being insanely busy. I have learned that we are 98% emotional and only 2% rational. Learning how to listen to emotions is, in my experience, the foundational skill of life. Stick around to the end of the show and I'll reveal how you can be on our next guest in 15 to 20 minutes. So let's get started. Karim, Karim Lee, you are— first of all, welcome to Listening with Leaders. You are the president of a company called Flatiron Software, which is a technology software consulting company, and You are, you've also got some other companies. We'll talk about that in a moment, but welcome to the show from your, and you're talking to us from Turkey right now, which is impressive.

That is correct. Great to be here. Thank you very much for having me.

Yeah. First of all, let's start with a little bit about your backstory. I know you went to university, you went to, I think New York University, is that correct?

Studying computer science?

Yeah.

I actually did a triple major at NYU.

Wow.

I studied computer science, math, and French. French literature.

French literature. Liberal arts and science. That's pretty interesting.

I wanted to make sure that I had something that makes it a bit more interesting. I started off as a computer science and math, uh, double major. And then at some point through my third year, I thought that I was getting too technical and I needed something else in the mix to make me a more well-rounded person.

That's right.

I went with French, studied in Paris for, for a semester, did some, took some classes in opera, French literature, various arts, and then came back to New York. NYU had a great program.

Wow. I was a liberal arts major at Dartmouth. Uh, I was an English lit major. I'm curious, and I went into law, which would not be surprising, but for a, a math computer science guy to focus effort in liberal arts is really interesting. And, and I'm curious, how did that study affect the course and trajectory of your career? I'm— as I know it did.

It didn't have a direct impact in the sense that it didn't help me get my next job, but it actually made me a much better person later on in life and much more successful because before, obviously, I knew how to read, but I didn't— but studying literature actually taught me how to understand and how to communicate much better, which became a very useful tool, maybe sometime around the 5th to 10th year of my career. Obviously, in the beginning, when I started off as a software engineer, My main problem was understanding code and didn't come in very handy that I knew how to read in French and spent a bunch of time reading Albert Camus, but eventually it became very valuable and it paid dividends further down the line.

I would assume that as you moved up and started taking on leadership roles, that's when you started seeing the benefits. Definitely.

Like the very first time that I've noticed the benefit was when I was writing ticket descriptions, task descriptions for myself and others. I was like, oh, I can actually do this much better now than I did before. That's convenient.

That's pretty cool. I'll tell you a quick story about how the liberal arts education affected me. As I said, I studied English literature and one of the classes I took was in Shakespeare. And in those days, back in the late '60s, early '70s at Dartmouth, we had a big mainframe computer, and the English department was using— was one of the— using computers at that time, technology, to analyze the works of Christopher Marlowe and William Shakespeare, because there were some scripts that they had. They didn't— there was a big question over who authored what part, and so they were doing computerized linguistic analysis. I took courses around that stuff and who would ever know? 10 years later, I'm a lawyer and I have a client who's been defamed by his business, former business partner, and with a 3-page unsigned letter. And I remembered the linguistic analysis. And so we ended up hiring a linguistic expert who did an analysis on the defamatory letter and other known writings of the former business partner. I ended up recovering $10 million for my client. Wow. Because of that.

That's incredible.

1986. Amazing, huh?

That's incredible.

You just never know how something like a liberal arts education, whether it's studying French literature or English literature, how that will have an effect on you later in life. And today everybody pooh-poohs the liberal arts education, and I think it's the one of the most important things we can do for ourselves when we're, when we're growing up.

It's a great foundation whenever we're hiring. Obviously, we're like, we're mostly hiring for software engineer roles, but one of the main things that I look for is for someone to be able to solve problems and understand and read well and be able to research. And that's more important than any other skill they might have because Obviously they need to understand some code because we're hiring them as software engineers, but especially for junior roles, that cognitive ability, that foundation in most cases is more important.

Absolutely. You graduated from NYU and then you worked in the corporate world for a while. At what point in time did you branch off and start forming your own company?

I I was working as a vice president at one of the media companies, vice president of engineering. And then my good friend from a while back was working on a consulting company. He had his consulting company and he asked if I'd like to join him. And I've always wanted to do my own thing. I always thought that I, it was in my best interest to have an ownership in the business that I'm working on so that I could have a bigger impact on the business. And I wanted to— I jumped on the opportunity, joined him. We started growing the business a bit, and the consulting business is still going. In the meantime, I always, in the back of my mind, I was thinking about building our own software, having our own successful software business. And that's something that we wanted to do ever since— I think I started like thinking about this when I watched Social Network back in 2011. But ever since then, we've been wanting to have our own software business or our own software product. And at some point, we started asking ourselves, what can we build? What is something that we can build? And we Obviously we're thinking it needs to be something that, you know, well, it needs to be in an area that we're experts in. And one day we were talking and my business partner, it was, I think it was the end of the year, raises season. And then he gives me an Excel spreadsheet. We had about 100 employees and then he tells me so-and-so are gonna get raises. And, and then I'm like, all right, if that's what you say, so let's do it. I didn't know half the people on that list. I've known some of them, but I didn't know them well enough to actually make a comment on their performance. And then I was like, hold on, surely this can't be just me. There must be others thinking that as well. So I started talking to some of my mentors and some of the executives in the industry that I know. And they were all like, yeah, I have the same problem too. Whenever I would talk to a CTO and they would say, Whenever the raise or the salary review period comes, my VPs would bring me an Excel spreadsheet. I would not know anyone on that list. And they would say, this guy gets 10%, this guy gets 5%. And I just have to take their word for it. And then I was like, this is not cool. And there's, there's an opportunity for us here. There's something that we could improve. And then I started thinking from another perspective, from the engineer's perspective. And we've had very successful engineers on various projects, but because of some reason that's outside of the engineer's performance, their roles would get eliminated in our clients. And I was like, that sucks for me, but also it sucks for you. And you don't even know about this because this is a very successful, very good, productive engineer that you're actually cutting off. But obviously, as the, as a vendor, what me making that argument doesn't have any substance to it. Of course, I'm going to say that. And then I was thinking, if only I had some data to back this up. Then the two, I started connecting the two issues. I was like, there must be a way we could measure the performance of these engineers. And we could also at the same time provide visibility for the executives, justify their budgets. And we were, we started off with the, toying with some analytics, building some software, building a software as a service tool that allows you to add 360 reviews, sprint retrospectives. Then we started adding GitHub data. So the actual code that the engineers put in and we started ingesting Jira information. So their task management, so all the tickets that they work on. And at that point, it was just a bunch of graphics, a bunch of fancy widgets. And then one day, after talking to some of the, some of my mentors, and then I was on a call with one of our engineers, and then he was demonstrating something to me, and then all of a sudden, a lightning stroke. We could have an LLM that ingests all of that information and essentially become a know-it-all for your organization. So we could have an AI that has access to different levels of information, different, from different sources in an organization that previously would not be easy to access. And we could build a tool that allow, essentially, we could build a companion for the CTO, for the VP to ask any questions that they would want that they otherwise wouldn't be able to ask. This AI could then go and estimate tasks because it has visibility of previously created tasks that were done previously, the description of those tasks, how long they took, by who, and what was the solution they've done because we actually have the source code. We could have, we could ingest transcripts of Zoom meetings so that whatever discussions there that were had, we would already have them. And that's how our product came to life.

And today somebody can acquire this, feed, you feed the data into it and you basically got an in-house ChatGPT that allows you to make queries about anything involving the technical side of the business.

That's correct. And obviously you don't want to necessarily give this information to ChatGPT. That's why we actually had to build our own, uh, guide—

it's an internal private platform.

Exactly. It's a guarded instance of an LLM. It's not ChatGPT, but it's a guarded LLM instance that you will have access to, and you can actually ask anything you want.

Wow.

I find working on this project very satisfying because of the technology aspect, but also makes us feel like we're actually creating some value. We're, we're helping other engineers and we're also providing management and executives objective information, objective data. I'm not saying that's the only data point that they should use to make decisions, but we're making information available that was not previously accessible.

Hmm. And how do people, the customers, how do they like the product?

So far we've been having, to be honest, we've only quite recently launched a product. So we're rolling out to customers very slowly. But the impression that we get from all the demos that we've done and the early adopters is Phenomenal. They love it.

How long does it take to feed the data into the LLM?

For a small organization, it takes about 11 hours. For bigger ones, it takes— What? I don't even want to know. It's, it takes a lot, a long amount of time. And once the side of the business grows, one of the very first things I'd want to do is probably to improve that. The learning curve, learning curve of the LLM, right? But it's a very interesting, it's a very interesting thing and I haven't seen other examples of it and it makes us very excited. One of the other reasons is the core philosophy of Flatiron is, well, let me take a step back. As a software engineer myself, I think that being a software engineer working as a software engineer is great. Uh, working in tech is a great opportunity for anyone who works in it. Like, you get to oftentimes work from home, and you get to be creative about what you do, and it's very intellectually stimulating. And oftentimes the work environment in tech is quite merit-based, and a lot of the politics you'd be shielded from. So it's a great job to have. And then what we've noticed is, and the reason, the idea behind Flatiron was that in the US you have access to that. You have a lot of good engineers, but oftentimes not enough. And as Flatiron, we wanted to democratize the access to this job market. And what we do is essentially we have, uh, software engineering talent from all over the world. That allow companies to hire. And Flatiron is facilitating that and ensuring that we have great teams. Now, what I'm going to go with that is Snapshot, on the other hand. So Flatiron is the entire philosophy of Flatiron is to have remote software engineering teams that are hopefully, and I believe so, very successful and very productive. And Snapshot is a tool that enhances, that allows companies like Flatiron and our competitors to actually, uh, be more productive on remote employees. So you can have an entirely remote team. You can hire Flatiron and every other competitor we have, or you can just have them, uh, in any way you want. And then using Snapshot, you could actually manage your employees At least while you— it won't manage them all alone, but it'll help you manage them better than you were able to do in the past.

So Snapshot, Snapshot's the AI application.

That's correct.

And this makes perfect sense to me. You've got, you've got software engineers all over the world, India, Vietnam, everywhere.

For the most part, we're, we're in Latin America, but we do have—

but now with Snapshot, you can take all of the Everything that they create, including, as you said, transcripts of meetings, their code, their work product, all the notes they take, everything they do gets dumped into Snapshot. Yeah. The LLM digests it all. And now you're able to make queries, prompts.

About anything.

About anything. And it'll pull all the data together and give you and analyze it and give you an answer. Whereas before you wouldn't be able to do that.

And that helps me provide tangible evidence about how great my engineers are.

And give me an example of, of how that might work.

So one of the examples is, but you can use this AI to look at the work that's being done. You can use this AI, for instance, previously you could just, let's go back to the CTO example. You see a bunch of names in a spreadsheet and some of them have 10% raises, some of them have 5%. You can go to AI and you can ask, what has this engineer been doing? What has this other guy been doing? Previously, you couldn't do that. That you wouldn't be able to ask, like, you could go and look at your Jira and GitHub and find one piece here, one piece there. But you wouldn't have anything that actually summarizes. And you could actually ask the AI to list the strengths and weaknesses because we're recording everything here. We're recording the feedback they have, the review feedback they've provided as well as receipt. You could ask how productive is this person, how compared, you can even ask it from a perspective of delivery. You can ask the AI to potentially compare the performance on various tickets, various tasks.

You could ask what rank could they rank.

Almost like you could ask ranking. I don't, like it. I don't recommend it because there's no, like, direct comparison there. Like, you can't really, you can't compare one person to another. But you could ask for explanations. You could ask, why did this task took so long? You could ask, how can we make this better? What are the improvements needed on this team? What are the strengths on this team? What are the issues that are being faced?

Wow.

Compared to this other team, what is the issue that this team doesn't have? And all of these questions, all of these answers that you'd get from the AI, that's, that I think is priceless.

I absolutely agree. That's amazing. That's incredible. I'd really never thought about what all this is doing as a non-technical person. I learned how to code in college, but I'm a lawyer, not a coder, is What AI is gonna be doing for us as we move forward over the next decade or so, it's gonna be incredible, isn't it?

It definitely will be. And we're all, as everyone else says, we're only scratching the surface on this thing.

Right.

Many AI scientists say, I don't necessarily think it's gonna turn into Terminator just yet, but a lot of these applications that we have, the ability to analyze text, the ability to do things more efficiently. It's already here and the innovations are gonna come. I remember reading a quote from Sam Altman in the past couple weeks, maybe about a month ago. The, the most successful, and he was saying, I think something in the lines of pick a niche in the, and then apply AI on it. And that's what I imagine what most successful AI startups are going to do. Pick a niche. And I'm sure, like, in a law example, I'm sure some company will pick a niche in one area of law and they would apply AI on it and specialize in that area. There will be other others, and there's going to be so much more.

It's already happening. You can go to Thomson Reuters or LexisNexis and they've got AI applications all over the place. I don't practice law anymore, so I don't use that stuff, but I've used ChatGPT just to ask it to draw up a simple contract for me under California law. And it did, it did, it saved me hours and hours of work. Yeah, I had to go in and tweak stuff, but it was pretty impressive what it came up with.

One of the, one of the applications of Snapshot that we foresee is that eventually code will be written by AI in one way or another. You might either have an engineer ask ChatGPT to write it, or you might have, or you might just have programs that actually generate code for you.. But if the code is getting auto-generated, then it's gonna get out of control.

Right.

So you actually are going to need AI to monitor AI. And in fact, ideally, you're gonna want a completely different AI with a different LLM background to monitor that AI that you're using. Because if you're auto-generating code, then it's going to— the amount of code that gets generated is gonna be so much that no human will be able to actually read it. So you're going to need an AI, another AI that actually monitors it and tells you what it says or tells you what it does. And it needs to be objective and it needs to be independent, compartmentalized enough and independent enough to assure that it's, the other AI is not tricking you.

Right.

And that's what I think the role Snapshot could have in the future. Ah, and so the one of the concerns we used to have about Snapshot was that we're monitoring performance of developers. What if AI automate? What if they become unemployed after AI becomes a developer? But then we realized we're not actually monitoring the performance of developers, we're actually monitoring the code that gets created. And that's always going to be needed.

Right. Interesting. So you're in a leadership position making all of this happen. Tell me about the importance of listening as you've been developing all this new technology.

All of the great ideas I've had come from listening to others. Sometimes it— the whole idea of having AI on this came from listening to others. The code came up in conversation, and it was only after asking more and more questions and seeking advice from others, seeking advice from people who have more experience in the industry, that's where the best ideas and best practices came. I've also seen that listening in another way, communication is very important. Making sure that other people listen to me, uh, when I try to convey this vision. But it's, it's been crucial for our success, especially when communicating with engineers from all around the world. You have to ensure that whatever message you receive or convey is very clear. And that you can avoid any misunderstandings. Because English is not my first language. English is not— I don't— there are very few people in my company whose first language is English, but everyone communicates in English. And we need to be extra careful listening to each other, making sure that we understand, because oftentimes misunderstandings could be costly.

That's right. And it could be costly to have a misunderstanding. Yeah.

Hmm.

One more question and I'll let you go. This has been fascinating. What's one thing about yourself that we would never know about unless you revealed it to us?

That's a good question. I never, I never thought of this. What would that be? Oh, I never actually liked coding. I never enjoyed writing code. I actually, as a software engineer, I said that all the time. I never actually enjoyed writing code, but I love creating code. I love the end product of code. I love thinking about the architecture and how we could do it. I love understanding it, but I don't actually enjoy sitting in front of an IDE and writing code.

That's pretty interesting. Good for you. You wouldn't know it, but I'm a jazz violinist. Kerem, it's been a great conversation. Thank you so much for taking time. I guess it's late evening for you in Turkey.

Almost.

12-hour difference almost, I think. Thank you so much, and I hope our paths cross again soon.

Oh, I hope so too. Thank you for your time.

Doug Noll here. Thank you so much for listening to Listening with Leaders. If you are a successful executive leader who would like to be on this program, please visit podcast.dougnull.com/podcast. If you got something out of this interview, would you please share this episode on social media? Just do a quick screenshot with your phone and text it to a friend or post it on the socials. If you know someone that would be a great great guests, tag them on the social media to let them know about the show and include the hashtag #ListeningWithLeaders. I love seeing your posts and guest suggestions. We are regularly putting out new episodes and content. To make sure you don't miss any episodes, go ahead and subscribe. Your thumbs up, ratings, and reviews go a long way to help promote the show and mean a lot to me and my team. Want to know more? Go to my website, DougNoll.com, or Follow me on LinkedIn, Facebook, and Instagram. That's @DouglasEnoll. Thanks for listening, and we'll see you on the next show.