#29: an AI-based engineering mentor

So who are you?

Hey, I'm Aisharya. I'm the founder of Intelligence AI.

And what is that?

Yeah, so at Intelligence, we're working on building artificial engineering intelligence. So a little bit about me first and like how this came about. Yeah. So before, before starting at Starting Intelligence, I was at Uber for about four or five years and what I realized is that like there's a lot of tools right now at Uber. We were using Copilot for a while and realized that all these tools, they are trying to replicate what junior engineers do. And if you look at some of the top performing engineers, what they were really excelling at was this kind of domain knowledge. You would say that there are people who are domain experts in different areas, which means that they had a comprehensive understanding of like how previous things were built, what different teams are doing, what teammates were doing, actually review everyone's PRs and have like a good understanding of the entire architecture and system. And that simply isn't the case with current AI developer tools. So I wanted to set out to kind of build this kind of a system so that it would be able to help all engineers actually with the context and building, building the things that they were doing at these large companies.

And AI engineering is different ball of wax than like writing C code in the old world. Right. This is a new world. Tell us a little bit about what you're seeing and what your company helps engineering teams with.

Yeah, yeah, like I think so. There's been a few phases, I would say with, with kind of using AI in engineering. Right. I think as soon as ChatGPT came out, everyone was super excited that you could, you could stick a chunk of code into GPT and get something else out or you can ask it to, to write a basic app, et cetera. It was able to come up with something, something remotely helpful as well as you could ask it, hey, this is like some error message. What can you do about it? Right. But all of this, I think people quickly realized, like doing these things in isolation are only, it can make you, it can like be a better stack overflow and that kind of a thing, but it's not going to actually help you build something in, in your system. Right. So that was kind of, I would say the first phase and then we saw the whole kind of hype cycle around Autocomplete solutions and Copilot and everything. And what that was able to do is now it's like obviously in your editor it's able to kind of fill in a little bit about like predict what you're trying to do understand some, some things around the before and after for that code. But what people also started to realize with that is that in order to actually write like real software engineering code, you need to understand a lot of things about how the system works. And none of these solutions right now are actually able to do that. And I think now the next kind of wave of these solutions are, as I said, kind of how can you replicate how engineers are actually thinking about these solutions? Which is I understand how this architecture works, I know what these different components do, how different people are collaborating to it. And I can not only help write the code, I can help design the solutions, I can help review it and kind of make sure the entire system's working together. So that's kind of the, I would say the next quote unquote phase of how AI can help engineers. And yeah, that's what we've been working on.

I was talking to Dylan this morning. He runs R and D or runs an R and D group at Unity, and he's saying that this new world requires a different architecture, different thinking about architecture. Do you agree with that at all?

And yeah, I would.

And how does your system help get people into building? Like, we're having to learn new, new ways to do things like rag, right. Retrieval systems. So give us a little taste of what, what, what, what you think of as new architectures.

Yeah, I would say there's really two large problems that pretty much any company building in the devtool space has to learn and to solve. Right. Those would be one, this context awareness issue, which is there have been approaches with rag. There's a lot of improvements, obviously you have to make on top of it to make these systems work on top of code. We've seen a lot of things. One is integrating LSPs, being able to actually understand the code structure, being able to do graphical representations. If you're trying to draw diagrams, how exactly are you doing the different layers for these code bases? So that entire thing works into this context awareness and it takes, it goes a few steps beyond just code. Right. Again, if you look at current solutions, a lot of them do try to take in the entire code base for context, but they're missing very critical things like time series awareness. Right. If I am working on a code base, I go work on some other come back, I want to know what all has changed and how that's going to affect my future decisions. So getting recaps on the code, not only that, but understanding when you're working in a team Right. It's very important to know what other people are doing. So is your system actually aware of individuals, how they're contributing, what's changing? So all of those things kind of figure like are a part of this first part of the problem which is context awareness. Like do you understand how the code base, how the org is working? If someone mentions something in a PR or in Slack, does your system actually take that into account when it making decisions? That's what engineers at engineering orgs do today. But most AI systems have none of that information. I think the second part of the problem is really planning. Can you take that information that you now know about the system and use that to actually decide how to build things in a smart way? I think both of these components are super critical to make AI systems that can actually work somewhat similar to how engineers are able to actually build things. So yeah, I would say anything that you're trying to build, if it's going to be meaningful, it would need to be a combination of these two areas you're building.

Like a pair programmer that would sit next to me, an AI that would interact with me. Tell me what that pair programmer would do compared to a human pair programmer.

Yeah. So as I said, we really model this off of. When I was first looking at this, we realized that at these large companies there's a few key tech leads that do this. They understand the entire system. They're the ones giving people feedback, mentorship. They're kind of like the ones who review, sign off on different design decisions, et cetera. And I was thinking how can we actually replicate something like that? Because if you think about it, there's a few key things that they're able to do and obviously how can we work towards being able to provide that experience? Right. So we don't want just another human, we want a person that has like an AI, that actually has this full system awareness. Right. So there's a lot of simple low hanging fruit there. Right. If you think about it, a lot of times engineers that work at these large companies think about, I want to build this feature, probably 10 people have done this before. Half of the problem there is just trying to find where those PRs are, who to reach out to, who to ask. Right. That's one of the most common things that happens when you're trying to solve a new bug is finding who did it, asking someone, figuring out how to orient. Right. Right now, yes, a tech lead who's been there tech six, eight years probably knows that information. But if you can build a AI system that's actually robust enough to understand the entire time series of how this code base has changed. What are those similar issues in PRs? It can point that out to you, right? So that's like one of the things that we support now is you can search for similar issues or PRS in our platform and it will help you kind of walk through, okay, this is something that's doing something similar to the ticket you're working on. You can use it. Other low hanging fruit, as mentioned also, I mean this isn't necessarily low hanging fruit, but also being able to generate visualizations or diagrams. A lot of coding, I think in my first CS class in Berkeley, they really highlight that coding's a lot about abstractions, right? Like if you're trying to look at the system as a whole, you do not want to understand how function A is implemented. If you're trying to, you know, there's kind of a lot of different abstractions and when you're thinking about the system, you need to abstract away all the details you don't need to know so that you can focus on what's important and kind of go back and forth between these layers of abstraction. So that's really, that's something we also need to get our AI systems to be able to do. Right? So we've thought a lot about this when we kind of generate diagrams is how can we use LSPs, which will give you all the granular details about this function, calls this with this class name under this thing and actually zoom it out to a level that humans can understand overall, the big picture and then kind of go in as needed. So those are some other things that we've really been focusing on. So we don't want a pair programmer that just kind of like will walk you through, okay, this line of code doesn't seem right or that we want actually something that can give you that like overarching understanding of these are relevant things. This is how the system's been changing. If you want kind of an overview, this is where you should look. And that's kind of what we think humans would, I mean, engineers will want as they, as they interact with AI and engineering.

It sounds like I use Limitless on my computer to record all my meetings. And it really didn't become useful for a few weeks. Right, because you had to get enough things in there for you to start seeing the usefulness of the search features and talking to it and stuff like that. And it sounds like the same thing is true here, but also you're trying to your AI system is trying to really understand the code base over time. So first of all, how long does it take to get set up for a team? Let's say I have five engineers. And second of all, how long does that team really have to give your tool to really get into it and understand it and for it to understand your code base and really start working?

Yeah, no, that's a good question. As I said, what we actually do is we work with most companies that are not starting from scratch. So we actually go in and ingest everything historical. The great thing about GitHub, about code is that you have pretty much an entire historical roadmap of how this code base came to be. So we're actually able to take all of that information into play so that if there is an issue in the past, we have access to that from day one. So a lot of that kind of just knowledge understanding of how your system came to be, we can just set it up for from the very beginning and then as you keep kind of developing on it, we keep our entire thing up to date. So yeah, but I think on top of that, just like everything that's in your code today, the other thing is you're actually able to also feed it in things like your style guide, how you prefer code to be written, et cetera. All of those things also factor in. So it's able to kind of understand kind of what you're looking for in your code based best practices. So this really factors into, into our PR reviews and kind of issue solutions is understanding how you as a team, what you're looking for so that it's able to guide you the way you and your. Org have already found the best.

Will this comment your code as well? Because you know, commenting is always a contention with developers. Some people don't like to do it, some people really like to do it, but they have their own style. And, and if you're working in a team, you really need to stick with a standard way of commenting your code because somebody's going to have to use that code later.

Yeah, yeah. So what we found to do is, so the way we've been thinking about it really is we've started off first just building those two core APIs that I mentioned, which is the context awareness and planning. And then now we have been, over the last month or two, we've really been plugging that into all the interfaces that engineers need the most. Right. So it started off with the website. If you're trying to onboard to the code base, we give you an entire onboarding guide, you can understand issues. We do comment directly on your PRs, but recently we've also been expanding more of our VS code interface. So within that you can have kind of all the functionality of highlight code, explain it, add comments. You can visualize just a section of the code. We have very robust commands now. So as I said, everything around search for an issue, search for relevant pr, et cetera are kind of all different commands that we've set up on our platform. So, yeah, within your code base, you can also ask it to do and you can also build custom commands. So if you want just something that goes in and comments every function, you can set up a custom command to go ahead and do that for you.

Sort of similar question, but a little bit deeper. What about test testing and making sure that the system doesn't regress or anything like that? Do you write test cases for us and hook into a test. Do you work with. With other test suites or. Because I know some. Some teams have their own test suites that they hook into and stuff like that.

Yeah, yeah, of course. How about I actually show you if.

Yeah, that'd be great.

Yeah. Let me just pull it up here.

And then does it. Does your par. Does your AI look out for security problems too? Oh, you left a. You left a problem here that could be exploited by a hacker, you know, kind of thing.

Yeah, you're. You're able to. As I said, like we. What we've done is we've. Our goal really has been to f. To focus on kind of those two core things and then use that to enable you to create kind of the customer commands or. Or even agents is something that we've been working a lot on recently too. So if you do want something for like security analysis, again, like, as I said, each company really has specific things they're looking for. You're able to give it those guidelines and it will. It'll audit your code and one thing. Yeah. And kind of give you those guidelines. So let me just show you some of those.

All right. We got our sharing working perfect. Yay.

Yeah, so I just. I just wanted to share kind of some of the things that you had just asked about. So, yeah, at Intelligence, obviously you can see our extension here. We've been working a lot with sweetbench, so this is just one of the code bases that we've been working with. So one thing we kind of do differently again from Copilot and from everything else is you can essentially integrate with any. Any other third party, either your other code bases in your system or an external code base you might be integrating with. So I've been using this platform for a lot, so I have a lot. We have about a few, several thousand open source already on our system. So yeah, you can put full agentless here. Let me go ahead and start a new chat. So we've kind of started things off with a few open ended questions. Right. So a lot of things, a lot of times when users first look at a code base, the first thing they want to do is get like an onboarding guide. So we've really been focusing like iterating a lot on that experience. So what we think when you think about onboarding is you want to first understand all the key areas or sections that matter the most to you and then kind of be able to dive deeper into any of them. So within this, it's kind of identified these are some of the top areas. And to do this right, we're actually synthesizing the entire code base and understanding graphically what are those key kind of layers of abstraction that you need to understand so that you can then go into anything in more depth. Right. So if you want to understand fault localization, how code repair works, text chunking, the structure, ranking, pre processing, these are kind of the main areas within the code base. And then you can obviously click into any of them and dive deeper. And then what I was mentioning in kind of addition to that is we support several different types of commands for engineers to work with and you're also able to create new ones in our ui. Also the way we do references is you can actually pull up any of the files. So let's say you're going through this and you want to understand where, where this actually lives in the code base you're able to kind of like directly open up the retrieval file. Anything else that, that you're looking at so that you can kind of orient yourself with the code as well as with, with the chat. Yeah, in addition, as I said, we kind of support all of these different things. So create test is kind of one of the commands that you can do as well as explain, search relevant issues, docs, PRs and then recent updates, improve onboard, etc. So let's say I have something and I want to write a test. Let's go ahead and find it. This is whatever, mostly a prompt, see if they can find. And then. Yeah, so I mean, and we've, we've also built like a lot of things. If there's this one code, you want to understand how what this is actually doing. You can hit intelligence AI explain. You can also kind of do it in parallel with visualization. You can write tests for this. You can also search like where, where else in the code is. Is most relevant. So we built again the. But the, the API that actually powers this kind of has full awareness of all the issues. NPR's et cetera as well. So if you want to do like recent updates, it will tell you kind of all of the latest changes or PRs that have been published in this code base. So you can also get a sense of okay, there was a repair eval enhancement, verified suite Quadrant, like essentially just all the recent changes. So you also understand who's contributing. If I want to go talk to him, I can kind of do that. So we've built our chat with kind of a full team awareness as well as being able to keep into consideration that it's not just your code base. There's all of these other things that kind of factor into you writing this code.

Can you talk to your assistant or your AI and ask it to write code like hey, I need a Twilio API call here hooked up to the button or something like that so I can make a phone call, my user can make a phone call or something like that. Can I write code like that or do I have to use cursor or one of the co pilots? You know, there's. Everybody's arguing about which one's better right now. Right. But cursor is the one that Carpathy just popularized this last weekend.

Everybody's going, yeah, no, I mean I think there's. Obviously they're, they're great too. But there's as I mentioned, I think they also are kind of missing this overall kind of just being able to. I don't think they have time series. I don't. There's. I'm not going to like do a head by head but there's several things that I think they aren't really building for an engineering team and are focused more on an individual, which is great. But I really don't believe that engineering is a solo person sport. So we've really been trying to think about how can we get it to work so that you're actually working well as a team and as an org. And we also, as I mentioned, kind of allow you to integrate very seamlessly with other code bases. Other, you know, so that you kind of have that, that kind of community and engineering team set up for you already. But yeah, that being said, yeah, we, you. You can obviously also write code we're not an autocomplete solution per se, but let's say if I want to take this and update it, the Twilio example is kind of not relevant at all for this code base. But I can say update this code to. Not entirely sure what this is doing right now. It's applying a diff to. Okay, I'm not entirely sure, but it will essentially do that for you as well. And we're also going to just update it so you can do that in line as well. So here, as you can see, it'll just rewrite that code with the changes that you have asked for.

That's cool. Anything else we want to see? Because I want to go back to seeing your face other than the screen window.

Yeah, I think these are the main. Oh, there's one more thing I actually wanted to share is, as I mentioned, like, code visualizations are also a big thing. Right. So again, something else that Cursor doesn't do is let's say there's this large file, this large function here. Right. Correct hunk headers based on content. No idea what this really does. We can go ahead and essentially copy this whole thing. Yeah. And then you can just. You can also hit Visualize and it will actually draw you a diagram of what that's doing. So as I said, like, engineers really like to both visualize things as well as kind of having to like read code every day line by line. So we've been kind of really thinking about how we can both pair together, that kind of understanding and explanation as well as viewing things in a more graphical way. So here, as you can see, I'll make this a little larger for you. Yeah. What it's actually able to do is explain how the chunking is working. Right. So it says it identifies the code element, element type, has each of these different ways that it actually tags the tree sitter, query, applies the definitions, do references, et cetera. So this way you kind of, once you kind of understand kind of graphically what's happening, it's much easier to then go in and kind of piece that together with the code.

No, that's really, really, really cool. That helps people really visualize even just a chunk of code. I mean, it's one thing that visualize the whole afternoon what it's doing, but just to. To be able to take a chunk of code. What the hell is this doing? You know, explain it to me a different way. Let me see it a different way. That's really, really cool.

Yeah, as mentioned, as I said, we really focused on that kind of abstraction thing. So you can also say, hey, draw a diagram of this entire directory or in the onboarding guide for each of those sections. You can, you can draw diagrams there. But yeah, as I said, like it. It really depends what level of granularity you're working with and how we. You can effectively help you understand or visualize that.

So could you. Anything else we need to see before we close down the code window?

Yeah, no, I think, I think those are the main things I wanted to highlight.

That's awesome. All right, so yeah, if you could close on your side, I think, or stop sharing. Yeah, there we go. All right, now we're back. Wow. All right, so some things that are like. Oh, a lot of enterprises are really nervous about intellectual property now. Obviously you're, you're giving it direct access to your intellectual property. So tell us a little bit about how you think about privacy and do you use this to train another model in the future? You know, are you listening? Are you watching my screens? You know, all the kinds of conspiracy theories that people have. Oh my God, I can't do this stuff. Not at work.

Yeah, yeah, yeah, of course. I think as I said, security is super important in the space. I think as you mentioned, as soon as you mentioned code, that's kind of one of the first things that comes up. So we actually built our entire system kind of from the very beginning to be fully self hostable compa. Self hosting compatible. So essentially the entire thing. Right now you can either we have an app on both AWS Marketplace and soon on G so that the entire thing is kind of containerized and you can run it on your. On your own servers.

I think our Internet went away for half a second. You said Amazon and.

Oh no, I said that we, we, you kind of, we host this on those marketplaces so you can containerize and like run it on your. On your own setup. Got it. So if you, if you are kind of working at a larger company, I think that's usually what works best. But in addition, we're also in progress of getting our SOC 2. So kind of how we look at security as far as like fine tuning models, again, those are all kind of only per Org. You get full access to the model. We do kind of tune on your previous issues and PRs in order to do the issue solutions because that is very customized per kind of code base, per. Org. To become a domain expert, you really need to fully understand that Org setup. But again, obviously there's absolutely zero kind of sharing cross Org across domain and if you're self hosting it, we don't actually end up seeing those models.

Very cool. Anything else I need to ask? Because I'm not a programmer, I'm not in the, you know, doing this every day. Like, like, like many of the people who watch this. Anything else I should be asking about or watchable about here?

Yeah, let me think. I, so I, I showed you the entire VSCO thing. I didn't really show app like website. So that's potentially one of the thing. In terms of other questions. Yeah, I'd be happy to talk a little bit more about how we've built the system. I think that's, that's awesome.

What's underneath the covers, but how many models you got running on. On my little Mac while I'm coding.

Yeah, yeah. So we, we do make it interoperable. I think as a lot of people have realized, Sonnet is really kind of a few steps above most other models. So we definitely use that for the kind of overall logical reasoning, for the questions, for kind of generating these diagrams and working alongside the lsv. But for some of the other things such as, as I mentioned, for issue solutions, that's one of the features we've worked a lot on. It actually takes into context all of your previous issues that you've worked in the code base and it tunes a smaller model to understand how each of those were kind of actually implemented into pr. So yeah, there's a few, as I said, for small individual tasks we found it most useful to actually tune it. But for our overall kind of being able to reason all the commands agents, those are generally powered by Sonnet 3.5.

Very cool. Does it look at the user interface or help you with user interface or is it mostly for the code that goes underneath the user interface?

Yeah, right now we're focused a lot on, as I said, just kind of understanding the code based architecture. One thing that we were working on a lot is, as mentioned, a lot of things aren't in your code base. So it's what you say to other people, it's in a conversation or it's stuck in Slack or Discord. So we've already built out Discord and Slack integration. So you can kind of chat with our entire product in Slack. I think a huge thing, at least at Uber and a lot of companies that I've talked to, they have these large support channels. Right. So you could, you could go into the code and try to figure it out yourself. But Generally speaking, about 80% of people would prefer to just ask the question in the support channel and wait for someone to answer it. So we've also been seeing a lot of strong adoption there in the sense that you can, if you set this up, our system on one of your Slack channels, you just identify which of the resources are most relevant to it. So it can be one code base, it can be a set of documentation for each of the code bases. You can choose whether you want to enable kind of PR issue access as well. And then it's essentially kind of just answers on autopilot, all those questions. And yeah, same with the PR review. We're kind of what the way we think about this is. We want to build the core reasoning and planning APIs that are foundational to making this entire system work. And then after that we're really just focusing on making it as streamlined to interact wherever you need the help or context. So currently that's in our interface in Slack and in your code.

Very cool. Last couple of questions, tell me a little bit about your business. Who are you working with, how many employees? How are you funded? The usual stuff about business.

Yeah, yeah, yeah, of course. All right, so right now we're a team of six based in San Francisco, so pretty small team. We raised our seed. We just raised our seed actually. So we will be disclosing like all our primary investors pretty soon. And yeah, we've been working with two design partners and we have, as I said, about several thousand engineers using the platform both on VS code and our ui. We actually recently kicked off a SWE bench hackathon. So that's kind of where this agent list came out of, is just helping engineers trying to understand all of these different code bases. And with that we actually also put out our first these two APIs that I mentioned for if engineers want to continue to build on top of it. So that's kind of a channel that we're also looking into. Yeah, but right now, as I said, our first design partners have really been focused on both the PR review. So we have one of our customers does about 30, 40 PRs a day using our system. VS code is pretty new, but we've been kind of adding all these new functionality and are looking to kind of start promoting it more very quickly. And then we've actually partnered with a lot of open source code bases, so we have a lot of other ones. Assistant UI is one of them. There's a few others based hardware, et cetera that we've been working with to just allow engineers to say, hey, this is my code base. I really want to integrate X into my platform with that code base. I don't want to have to dig through their docs, their code, all their different setups. How can I do this? Easily. So, yeah, that's kind of where we've been working with both of them for a few months now. And we have a few enterprise partnerships that we have also been kicking off.

Very cool. What else do I need to know? I'm out of questions. I think it's been great. Look, you know, I, I've worked at a programming magazine. That's why I want to talk to you, because the programmers are really running the. Here's one, here's the last question. There's 750 companies I've found for developers. How, how does any developer keep up with the amount of change that's happening right now? You know, and, and how do they evaluate different companies like yours? I mean, you're one of 750 that they have to choose from, right? And, and think about, you know, that's a lot. That's a lot.

Yeah, I agree.

When I worked at a computer programming magazine, there was like a few dozen companies going after programmers. Now there's, I mean, it's insane what the competition is to try to get developers to use new tools.

Yeah, yeah, I definitely agree. I think, I think there's a few. Yeah, it's. It's crazy to be honest. I think as I said right now, most, I think like until two years ago, most of these things you couldn't even do. So now that you can, everyone is like, how can I take a small piece of the sliver? Where are we going to get into? But I think, I genuinely think that there's those kind of two key facets that kind of underlay any tool in the space. If you are actually building something that engineers are going to use, if you're kind of more on the metrics like dashboard, more management side of things, it's not relevant. Again, you're just building more of analytics Dashboard, you can be on the more observability infrastructure. So I would kind of say this. The space is, obviously has a lot of different verticals, but if you are working on kind of helping engineers with their immediate developer productivity, you're not going to be able to do that effectively if you don't have good context awareness. Context awareness. And there it's a question of building the right interfaces, UX and being able to help an engineer throughout that process. I think for us, as I said, that's what we really focused on. Kind of from day one. And by being able to use that as kind of our core platform, we're able to help engineers both if they want to get that onboarding guide or if they want to draw a diagram or kind of understand, get like a review on their PR and get feedback. And I think by having this right, we're really trying to emulate what a tech lead would do and then provide insights to management on. This is how your team's doing. This is what we're helping the most with. This is what's the most confusing. These are the issues that have been taking forever to be resolved. These are the unhealthy parts of the code base. You kind of get a much richer set of insights and analytics by being that kind of a platform. So that's kind of what we are able to offer to larger companies to which as a product that essentially may only do like a PR review or may only do autocomplete, is really missing kind of the breadth of what engineers do day to day and how an engineering team what it needs to really be efficient.

Very cool. Thank you so much. Where do we learn more about you?

Yeah, you can check us out at our website. Intelligence AI. I can drop links for our discord. Yeah.

Thank you so much.

Thank you.