Sahil Lavingia introduces Shortest, an AI tool that writes tests faster as AI shortens feature development time. The team discusses the MIT-licensed launch at shortest.com, early pricing ideas of $10 per org per month, and plans to open-source Gumroad in 2025.
Chapters
Awesome. The other thing I wanted to figure out is, should I stream this or should I just record it and upload later? What do you guys think? Should I stream this on YouTube live or will that be— is that a bad idea? No comments. I say yo, just my comment.
I mean, it's a public meeting, so if we stream it, then you know why, why somebody would just not join in.
Yeah, that's true. You got it. That's fair. Good point. Good point. Yeah, awesome. Cool. Yeah, so I just wanted to chat a little bit about Shortest, uh, which is a project that I started working on a few weeks ago. And if you go to shortest.com, you can kind of check it out. It's a pretty simple idea, uh, which was basically I realized that as, you know, as AI has started making it faster and faster, specifically, you know, Cursor and v0 and Next and has made, you know, made it a lot faster to start shipping, you know, new product features. You know, this, for example, probably took me like 3 hours to build the whole thing or maybe 6 hours or something like that. Mostly just like dealing with like figuring out the GitHub API and things like that. Uh, I just started noticing like the stuff that used to take a lot of time really feels slow. Right? Because, you know, let's say it takes 6 hours to build a feature and 1 hour to write tests. It's not a big deal, but if you can build a feature in 15 minutes and it still takes 1 hour to write the tests or even 15 minutes to write the tests, it's like all of a sudden half your time is spent, right? Writing tests. And as someone who likes shipping product, like my, what I enjoy doing the most is I like ideating. I like coming up with new ideas. And like this idea, how long does it, you know, it took, takes 5 minutes to come up with the idea and then it takes 6 hours to build, right? Uh, and then it takes even more time to write the tests. So yeah, it just felt like writing tests is just a huge part of, of building stuff. So writing tests is probably not going to go away with AI. Some people think, well, why do you need tests at all? Uh, but I actually think AI will make it even more important to have a really good test suite because if you have a really good test suite, you can trust. You know, imagine an AI writes 30,000 lines of code in 2 years. You know, it's going to be able to do that. You may not be able to wrap your head around it. You're not going to read even all of that code, right? Right now, you know, if it's a few hundred lines of code, you can just read it. But if you have a test suite that passes, you're going to have a lot more confidence in the changes that the AIs are making on the codebase. And in theory, the tests can actually serve as guidance for the AI as a way of actually knowing what to build. So pretty simple idea, and I had this nice domain, shortest, which has the word test in it, which I think is kind of cute. So the shortest, you know, shortest way to write tests, fastest, whatever. And so, yeah. And then the last thing I'll say is one thing I wanted to play around with is commercial open source. So commercial open source is this idea that you sort of give away the codebase, you give away the code, you open source it, and you actually monetize something else. And so, in this case, Sortist, you can see the license here, but we use the— oh, I should probably change this. I copied it from Vercel. Kind of ironic because given, let me just change this to Gumroad. Not that it matters because it's an MIT license, I think this is basically MIT is like the most permissive. Basically anyone can kind of do anything. I think the only thing you have to do is that you have to basically, if you fork it, you know, it has to be, it has to be included in any of the forks. But, you know, you can basically without limitation use it, copy it, modify it, merge it, publish it, et cetera. Even sell it, right? Someone could take Shortest and just monitor, like, compete with us, which I think is really awesome. I guess it says it right here. So the things, the limitations are basically you can't sue us if you fork this and then you, this somehow leads to, you know, you, I don't know, losing a bunch of money or something like that. You can't come back to us and say, hey, you owe us money for that. So, and why would I do this? There's a couple of reasons. One is I'm busy, so I don't have a lot of time working on this. Great way to get help is to just open source it. But more strategically, I think that with AI, there's actually going to be a big push around this movement. The reason being that, like, you know, with, with open source, imagine, you know, imagine I have something like this, like, let's say 3 years ago, pre-AI, and I said, hey, I want to, like, add streaming test files, right? So right now, if you have, like, 10 files, it just takes a while, it doesn't do anything, and then all shows up. And with Vercel and Next, and you can, I'm sure you can do it without that. You can basically stream the inputs. All right. So as the, as the AI generating the text, uh, you can start to show like the test files that are changing. This, you know, pre-AI would take tons of time, right? 3 years ago, it would take a ton of time. And so the value of open sourcing, it's not really there because you're basically opening yourself up to competition., but you're not really getting any help. But with, you know, because which person is going to, you know, someone who can go make a living as a software engineer, why would they spend all of this time contributing? It could take a week to build this feature, right? But all of a sudden it can start, you know, imagine you have AI and you just say, hey, build this feature. And in 15 to 30 minutes, you now have this feature built. And so I think that the economics of commercializing your software and open sourcing your software are quite different 2 to 3 years later. I would say sort of 10 to 100x different. And so a lot of the math, I think, that went into keeping something closed source doesn't really math anymore. And I want to open source Gumroad. The plan is to open source Gumroad in 2025. And so I felt like this is a good experiment and I can learn, you know, by doing it. I'm sure I'll make mistakes and then we'll make less mistakes when we do Gumroad, which is much scarier and bigger. And honestly, I've been shocked at the, at the response already. And so I'll get into that. But before that, does anyone have any questions or concerns or feedback or anything they want to bring up? I want to go through kind of what's actually happened since, and then I'll go through some ideas so people kind of know kind of where I want to take this and know how to best contribute. Karan looks like he wants to say something. Nidish, do you want to say something?
I, I, I was putting it in the chat because it probably wasn't important to interrupt, but I'll just throw it out there. Is, is the, uh, text unclear and blurry for others too, or is it just me?
Oh, it might be just the resolution. Okay.
Yeah, this is—
It might be easier. Thank you. Okay, so yeah, I'll try to— Yeah, I'll do this. Awesome. Cool. Great. Thanks for that. And yeah, feel free to interrupt me. I think it's always more entertaining when you have more voices anyway, so. Awesome. Cool. So how many stars? Everyone should star this repo. We have 150 stars. I feel like that's what everyone goes for on GitHub. So stars. Star of the repo. Uh, but yeah, I just thought I'd go through, uh, kind of the stuff that's happened since. Uh, so you can see kind of when I started working on it. Pretty basic. Uh, if, if you go to shortest.com, you can also sign in and check it out. I should probably update this to actually the production mode thing. Let's see if I'm assigned to any pull requests, make sure that I can see this. Yeah, this one should show up. Oh, we added this. This is another, another thing. Okay, so this is what it looks like. You can see, why are my repos not showing up? I don't know. Anyway, this should show up and allow you to pick your repos, which I'll go in and fix later why that's not working. Um, but yeah, anyway, I wanted to go through what people have shipped and then we'll talk about, uh, talk about what, uh, what, what we want to do next. So, I mean, quite a lot of stuff, honestly. So this was, uh, someone from Vercel noticed this. Apparently this was an issue. This is a nice thing about open sourcing. Uh, so this route wasn't protected. So basically people, if they wanted to, they could just hit this route and basically, you know, hit us with a bunch of API tokens. Which are expensive. And so now, now that's protected, so you have to log in at least. So that's nice. What else has been done? Small stuff. Let me open all these up and then we'll go through them. Also, I switched to Safari because I want to try, try using Safari, so I might have some weird stuff happening. I will go through why I closed this as well. So I mean, this is a good example. I mean, like, look at this. This is like 600, 400, 500 lines of code diff, right? I mean, this is a ton. This would, you know, this would be hundreds of dollars, you know, to pay a software engineer to do. And because it's actually fun to do this with AI, it's like much faster. And so this guy did this, which is basically the issue was I would, I was adding Clerk, but then I wasn't actually using the Clerk to get GitHub. There's like a second layer of authentication that needed to happen because I thought that Clerk didn't support basically getting the repos, which is probably why it's not showing up for me is I have to like redo my permissions. And so now we actually use the same, the Clerk the Clerk thing to actually get the, to actually get the repos, which is pretty awesome. So all this code that we needed, you know, to kind of fetch that from GitHub is now gone. Great example of, you know, something that is help, is adding value that I didn't have to do. I just woke up and like, literally this is like the day or two after got merged. So we got that done. This was a small thing, small bug. I'm still new to Next, so I didn't realize you have to— you have to return it after a redirect. So that was added. Awesome. Another bug fix. Yeah, these comments are annoying. I actually added a cursor rules so that this cursor AI will stop suggesting these comments, which are not really useful in my opinion. I personally think with AI, you don't really need comments that much anymore because you can just ask the AI like what the code does most of the time. So you can see that there's like multiple layers of HTML, and so this HTML tag was kind of redundant. And so that was removed. The code is cleaner. Woohoo! Awesome. Oh yeah, this also div className main. It's kind of like an old school way of doing things, but now we have main in the HTML spec. Don't need to do that anymore. These are all great, you know, great things that like, if we had a team of people working on it internally, we wouldn't even do this because it's just not worth you know, the speccing and the, like, the internal dialogue around it. Uh, but these are all great, great improvements. This one I loved. Big fan of stuff like this. Uh, so I, you know, I built this myself. I had the, I had the README so that you would just do Vercel pull. You know, if you go down to this, I had it so you would just do Vercel pull that, and it would pull all the Vercel environment variables, which is great if you're part of the Gumroad team, right? Because our Vercel account has all these variables in it. But if you're not, it's kind of annoying. And so now we have a really, really nice README that hopefully everyone can use to get signed, to get set up properly, which is really nice and clean. And I didn't realize you could even do this in GitHub. So that's really cool. I'm learning a lot as we go. Pretty awesome. Pretty cool. Pretty cool. So that's done. This is another thing that someone did, a script file. So instead of— oh, this doesn't really need to be there. It's kind of redundant from that. But all of a sudden, you can just run npm run setup. And we now have this setup wizard, which is awesome. And good example of something I would have no idea how to do any of this, but all of a sudden you can, you can just run a single command and this person will actually, this little piece of code will actually tell you what to do. Oh, there's Docker. Someone, oh, it even has Docker in it if you want to use. Nice. Cool. I don't, I don't use Docker personally, but I'm not a fan, but Some people like it, especially if you're working on a lot of stuff. So that got merged. Small thing here, we added env/local to .gitignore. Someone wrote tests. This is a great thing to do. Thank you, Krabbist. So if you're looking to contribute, you can write tests for the app as it exists. Awesome. README got improved. Also awesome. GitHub OAuth section, more README stuff. This I closed. I'm open to it, by the way. But basically I want a few people to like thumbs up it. That's kind of, if people thumbs up it like this, then I kind of know, okay, this is something the community wants. But if there's like a, if there's like someone who disagrees, then I think it's probably not great. So yeah, I mean, basically if you have Linux or something, sometimes I think Docker is necessary. So that got done. This is like the first product feature, I guess. Why is this video not working? There you go. So now you can Select your repos. You can add, you can, you can just view these. I'll probably clean this up design-wise. I think this is good. I'll probably move build status over here, like make it a little bit cleaner. Or if a designer watching this wants to do that, feel free. There's just a little bit of space that I think could be cleaned up a little bit. One also thing which would be cool. Oh yeah, this is why I merged it. I was like, oh, people want this. So I'll merge it. But one thing that could be cool is if you refresh it, if it saves, if it's like persisted, so you don't have to— most people, I assume, only want to change these like every once in a while. And so I merged this yesterday. Oh, I also said what I plan on doing. So if anyone wants to do this, feel free. It's pretty cool. Pretty awesome. And this is a good example of of a feature. Yeah, these comments like don't really add much in my opinion. But all looks pretty awesome to me. We could probably use tests. So that's another thing we should probably add is some tests for this, for this pull request filter or whatnot. Oh, this is another good one. So before we were passing all of the tests. So the way shortest works, which you can, you can see in, there's basically a single file, which is all of our secret sauce, which is gonna be, I believe, in here. Is it here? Where is, no, it's in You are an expert. There you go. Awesome. Yes, this generate test route. So this is all the magic. This is the stuff that like if we were closed source, what do we know what we would be charging money for? And so basically what it does is the prompt is you are an expert software engineer. And this is by the way, AI written. AI suggested doing this, which I think is smart actually. Like basically you always want a sentence of like, who are you? Like, what is your role as an AI? And so there's two buttons, the writing and updating. And so, you know, let's go for updating. Update the provided failing test files in order to get the PR build back to passing. Make updates to tests solely, do not add or remove tests. And so there's basically these two inputs. There's the diff of the PR and then there's the test files themselves. And I tweeted XML is the new JSON and why, why it's basically because This is like, if you're talking to AI, this is much nicer and cleaner than like JSON. JSON's really nice, uh, if you're like building a REST API, but if you're talking to an AI, basically you want to know, like the problem with JSON is you have like these, you know, like dictionary with, with keys, but then at the end you just have a closing tag and you don't actually know, like imagine you have a really long prompt, like what is this actually referring to? And so this tells the AI very clearly, like, where this ends and what it is. It's kind of just a nice reminder also as a human reading it. So that's basically it. There's these two inputs, right? There's just the diff of the PR, the actual code changes, and then the test files. And before, we were passing all of the test files in, and now we are only passing the failing ones. So you can see that probably Where is it sent? Here. So yeah, you can see in the case of, of writing new tests, it'll pass the entire test file directory. But in the case of updating failing tests, you don't really, you only care about passing in the failing tests, right? Because those are the only ones you're going to make changes to. And so this is what that does. If the mode is updating, then get the failing tests. And then filter the test such that you're only returning the ones in which the name matches. Really cool.
There might be—
and how does it—
yes, there might be a case where there would be test helpers that are test utils that might want to be included for like editing purposes. Maybe stuff that's imported into the test gets automatically included as things that can be edited.
Oh, yes, that's a good point. Yeah, so you might have a failing test, but you don't have, but yeah, if it's importing something and the actual failure is actually a problem that could be fixed in some import, then that is a good tip. Good. Yeah, exactly right. Good to test. So yeah, you basically have to run through the code and get those imports. Let me see if this is actually any, if this is a good example. A really good example would be you have a page that imports a bunch of components. I assume that you may, you know, the page test may fail, but actually the problem is gonna be in like the pull request test or something like that.
Yeah, another example I was thinking of is like, you know, when you start having on a big codebase a suite of tests and you've started writing like some test util class modules where you have like you know, you're faking a particular type of data frequently in many tests and you kind of move that out to its own thing and that has like some kind of bug going on that's causing the PR. That could be also a case where you want to include that file.
Yeah. Yeah. So like in, in Ruby Rails, it would be like a factory or something like a fixture or factory that is included that Something like that. Awesome. Yeah, I agree. Let me make a note of this. I can flesh these out later, but just tracking. Awesome. Sahil Lavingia. Do you have plans to monetize? Is that even like a goal of this or is that just, you just want to build the best product possible? Uh, I would love to monetize this, uh, at some point. Um, I was thinking of charging something like $10 per organization or something like that per month. Uh, so, uh, but yeah, I'm not, I'm not in a rush. I really want to make sure that it's really good. And to be honest, like, what's actually really nice about it being open source and basically free to use right now is that it's not that useful, right? Like, it's useful if you're like, for these contrived examples, but because most codebases have like lots of code and there's lots of technical debt you have to think about, I think it'll take a little bit longer for AI to be good enough to really like one-shot a lot of stuff. But it'll, it'll definitely get there, no doubt. And so getting all of the UI and UX and all of the piping in place such that when it's ready, it's like amazing. I think it's a really good example of like basically you should build for stuff that the AI cannot really do yet, assuming that it will get smart enough, like the context length will, the context window will grow, the AIs will get better. It can do a lot. It just requires a lot of like human involvement, and eventually it will require a lot, a lot less. So this is— yeah. Sorry, that just made me think of a follow-up question. Maybe I missed in the beginning, but I guess it's the goal here to write that the, like, software developers have to write no tests. They get this tool, would just handle all the test writing. And, and I, I think I, I'm, I, I'm trying to think of, like, what does the software engineer archetype like doing, or at least the one that's focused on shipping product features. And I think it's not too bad to write the initial test suite, right? Because you're going to kind of want to set up, like, you want to choose, like, what pack, you know, what testing package you want to use. Yeah, you know, every, every ecosystem may use it, you know, there's like Minitest or RSpec. So some decisions have to be made, I think, by the human and kind of the architect, quote unquote. But once you have the overall, just the framework of the approach there, then I think the vast, almost all of the tests should basically be written. I think there's two approaches actually, either And it, it depends on the developer preference. One approach is that the developer is writing almost no tests. They're just building features and then when they're happy with the feature, or maybe even as they're building the feature, the AI is like committing tests that kind of align with that and then updating the test. But then I think the other approach, and I'm quite interested in this one, and this I think is the actual business here, is gonna be feature development by the humans writing the tests. And then the AI looking at the tests and updating the code base in order to fix the tests. This is an example of, I think the context needs to be quite large. You're gonna have to feed in a lot of context for the AI to really be able to do that. But imagine, imagine you go into the test, you know, and you, and you say, hey, I want it such that when I click a select, I'm able to, it shows me all of my GitHub repos. And then I'm able to, you know, use checkboxes and select the ones. And then when I reload, it stays there because I want it persisted. You might write that as a test, or you may use AI too, because at the end of the day, humans are still gonna be enjoying the process of building features. But then all of a sudden, the AI can just say, oh, the build is failing. The build is failing because this person wrote a bunch of feature specs that— for features that don't yet exist. I'm going to go build the features in order to make the test pass. And I think this is really well situated for how AIs work because AIs need test cases, right? If you think about any eval, what are they doing? They're kind of basically saying, can this AI pass the test? And how you train these models is often like the test fails, you know, these, this is the update that made the test pass, right? It's kind of like RLHF in that way. And so I think it kind of can go both ways. And there's this concept in programming called test-driven development, which no one really does because you don't really know exactly, like sometimes you don't know what you want to build. So basically you write the test first and then you write the code to make the test pass. But it doesn't really work because you're often like, you don't really know what feature you're building and you don't, when you write the tests, you don't know if you've written the test correctly, right? That's the other thing is like, what if the tests are broken because you just messed up the test? You don't actually know. But I think AI will be able to do that like much, much better than humans, humans will. The goal is that, you know, humans are able to like ship what they want really fast and anything they don't enjoy doing, they should basically be able to have the AI do, right? So for example, We use like linting. Humans don't like to lint their own code, so we have AI auto-fix all the linting errors, right? And humans can just build their feature and then they can actually just get pushes, you know, commits on their PRs that automatically fix all their linting errors and get their builds to pass. And imagine like it's more creative than that. You could actually like write tests, fix tests, write features, all that I think will be, will be possible. But I don't know exactly what people will want, you know, what will people use it for? And so once we know that, once we have enough traction on the product, then we'll monetize it in some way. And the cool thing about this, uh, you know, this model is like, if you don't want to pay money, well, great, right? Like, just run it locally and it's free. Like, it's, you know, all it's doing is making some API calls to Claude or OpenAI or anything like that. So, um, it sort of forces us to be really cheap and affordable because no one's going to pay like a lot of money for it. So yeah, that's kind of how I think about it. This is an example of something I don't think we should do yet, but maybe we should. And so this was— Junius wrote this, and I appreciate it, by the way. Thank you for doing this. Which basically the problem is sometimes the context isn't there. And so you don't really know what tests to write. And so this guy did a demo. We can watch it together. So you can connect an issue. In this case, Linear. I do think we should probably support connecting a GitHub issue because I think most people will probably be doing that. So you connect an issue and then when you do that, it generates. So this is kind of what I'm talking about, right? Where you're kind of feeding in some context from some other source and it's writing the tests. I don't know why it goes black randomly. It's probably his, uh, so I, I, the, the thing I worry about is that it, this feels like it may be a little forced. Like you're, I don't think most people are speccing things out so concretely in Linear or Notion. But maybe they are. And so maybe this is like an interesting approach where that you can just take a GitHub issue or a Linear issue or a Notion card or something and feed it into the, into the AI. So I'm open to this, but I decided not to merge it for now. And I don't— did it get any pluses? Let's see. I didn't get any pluses yet. So if you want, if you want this, plus it. But I think something like this would be cool. I think it makes sense to some degree. Just have to handle it the right way. And ideally, the developer flow is that— what I find is that you can ship so fast that I don't even use Notion. Like, I just want to ship, you know? I just want to go into the code and build the whole thing out. Because it's so much like if I, if I'm typing the Notion thing, why don't I just go into Cursor and just type the exact same thing in and have all the code written for me? Uh, so I feel like that's a, that's, that's also another reason is like, I, I think we may be moving into a world with AI that like a lot of these PM tools are not that relevant because you're like basically outshipping the, the PMs. PM is useful when like it, you know, the cost of shipping new products is so expensive that you want to kind of make sure you're doing the right thing. But like, if you can ship this in 30 minutes instead of 3 hours, you can just kind of go straight to, straight to code. Like even this, right? This wasn't even an issue in a discussion. Just went straight to code and video, and which is amazing. You know, 2 years ago it would be like, it would just be like this, you know, as a, as an issue, and then we discuss it for a while and then the code would be written. So it's a good example of, I think, how things are changing already.
That's an interesting point. Can I just ask one follow-up on that? So one thing I wanted to find out is how are you tracking like what things to build in the product or what things are wanted? Is it mostly around the issues that you've been showing like in the thumbs up and stuff like that?
Currently, yes. Ideally, we'll have some customers at some point and then, you know, they'll have demands, but Basically, I'm trying to get Shortest to be used internally in our repos, and we have sort of 5 repos now with Shortest, Iffy, Helper, Flexile, and Gumroad. Each one is like more complex and more technical debt, so the goal is to kind of get it used for Shortest, be like, okay, Shortest works for Shortest, that makes sense, uh, then Helper, then, and once it works really well for Helper, and then each one I think will learn and have to do certain things for it to be really good for that. And then Flexile and then Gumroad. And each one has a different codebase as well. You know, different tech stack. But yeah, and then, and also what the, what the kind of community, community wants. What, what people find it, find, would find useful. But my, my North Star is always save time and save money. So like whatever people are spending time on that they think AI could do. We should probably, probably try to automate.
Yeah. And the other interesting thought that came to my mind when you were talking about how the speed of shipping has become faster and most people are just jumping into Cursor and typing out what they want. Like there's, I think, good value in, in these prompts that people are using to come up with what they're building, right? Like those prompts can be kind of a lot of context around how the tests would shape up as well, right? Like, so I wonder if there's something over there where we're, you know, integrated into these workflows in a way where we're seeing what it took to come up with the feature, like the steps that it took to get the features, to almost ask the AI to organize how the tests might look like based on, uh, you know, what the feature set is, right? Like, so I don't know, just spitballing an example if you're gonna be a—
sorry, go ahead. I know, I think you're totally right. Sorry for interrupting. I think you're totally right. Like cursor rules exist, which is, I just created don't add comments. And you could imagine something like a cursor logs or something like that. You would have to figure out what's the right way to pass that data into the AI. Um, but I've definitely had that thought where I'm like, I'll build a feature and pass it off to an engineer to finish. There's some things I probably didn't think about, but they don't have the context of like, what prompts did I use to actually build a feature such that like, they're like, oh, Sahil wanted this and actually the AI didn't do that. And so that's a thing I should fix. So that's actually very interesting. Yeah, like it's, and you know, even, even like teams, right? It's like, I think it should be very easy to like for teams to share all the prompts they've been using with each other so that people can get help. And even, even like open sourcing this project, it's kind of nice to have, you know, that like generate tests prompt, right? In a way, because you're getting, where did it go? Generate tests., you know, because you're learning, oh, this is how you would ask an AI to do this, right? Basically. Um, and so, and yeah, having it open source, I think, will improve just the training dataset that AI will use anyway. Um, so yeah, I think, I think that's very interesting. Uh, I'd have to think more about, let me just make a note as an issue because I think that's a really good one.
And actually, sorry, just to add on that, it's, it's really, uh, interesting what you said about The, the PM process, right, of how we've been going from writing documentation and specifications and then discussing and then writing. This is actually, this is a movement of like reverse PM, cuz you're starting with all of the prompts that went into building the feature and now you can write some documentation for like, I don't know, internal folks, uh, who want to know about what features are coming out, uh, roadmaps that, uh, need to be released, change logs that need to be released. All of this stuff can be generated off of like the feature being built already. It's just pretty good.
Yeah, it's like imagine right now we have Git commits, which are kind of useless to me in my opinion, because they all get replaced with the GitHub issue description, all that stuff. But it's like almost like, yeah, what are the— what is like the Git commit log is actually like the cursor prompt log. And it's like, what are the AI prompts that you use to build this feature? And then effectively, you should be able to look at those and ask AI to turn that into a blog post, right? Of like what got built or a changelog. Like if this, if instead of seeing all these commits, which are kind of like, you know, if you go through them, you know, they look like this. But if you didn't kind of, it's okay, but like not that useful, right? Like you have to guess quite a lot, like what that is. Yeah. But imagine if it was actually like a human English that's like, build the thing that puts the thing, you know, and you have it all there, I think. And that makes a ton of sense that like AI would effectively use that as like kind of the scope/spec to generate the test file itself, right? Because you're effectively taking that and making it programmatic, which is what AI is one of the best things AI is good at is translation, right? So effectively, you're just translating from like cursor prompts in English to like a sort of whatever the testing framework language happens to be. So yeah, okay, one more to go and then we will go. Okay, yeah, one more to go. Yeah, awesome. We'll go through this one and then we'll go through the issues. Oh, this has some pluses, so we probably should do this. Damn it. I don't use GitLab, but I commented a little bit. So anyway, so this one is Basically, you know, I built it for GitHub because that's what we use. Everything is in the context of GitHub, but someone who doesn't use GitHub and uses GitLab, effectively it's the same software, same feature set, but should apply to GitLab as well. And so, uh, Udul— Udul Kaush did this one. He also merged some other stuff. And I think this is a really good one. There's a couple, couple questions I had. So there's no test for this. It would be great to test it. Ideally, we could use Clerk for auth, so we can kind of use Clerk for both GitHub and GitLab, which would be nice. Maybe we decide we don't want to use Clerk for auth and we just get rid of it. So maybe that's the right answer is we just want to allow auth directly to GitHub, directly to GitLab, but I like Clerk. I think it's nice to not have to worry about all the extra UI, especially if you have organizations and roles and permissions. It'll be nice not to have to reinvent the wheel there. And also Clerk is effectively free up to 5 users. So if people are self-hosting it, it should be free, free for them. And then ideally, it doesn't make it worse. I think one of the issues, as I commented here with this PR, Oh, this we can, if this is useful, we can just move it out into its own PR and merge it. Is basically, where did I comment? It'd be cool if there was like a way to see the comments right here. If anyone from GitHub is listening, the only real issue I had with this whole thing, honestly, is where's my comment? Oh, maybe it was already addressed. Did Prakash already fix the issue and therefore it's not even showing up? Oh, outdated. Yeah, it looks like he probably already addressed it. So never mind. I had some issues. The issue was just as a PM, you know, this is just a bit ugly, right? Because it's like fail to fetch pull requests and merge requests. But ideally, you're only, you know, you're only going to be using one of these. Maybe not. Maybe you're using both. But I think we should just change this to just say, if you're connected to GitHub, it should just say this one. And if you're connected to GitLab, it should just say the other one with merge requests and GitLab, just to make sure that it's not worse, a worse experience for people who are just using GitLab. This, just some whitespace that I asked to get removed. This, oh yeah, this is why I added the comment rule. So the cursor should stop doing this. If you're in this repo and using Cursor. But yeah, these comments just don't really add much. But another example, I mean, this is like 400 lines of code, not a simple thing. I mean, this is like a whole feature. You know, let's say this takes 6 hours. By the way, the other thing is this is free. No one's getting paid to work on this, which means people are optimizing for doing it really fast, right? But if you're paying people a full-time salary, let's say you're paying a software engineer $400,000 a year, Well, they're going to take 2, 3, you know, a lot. I'm not saying that they're like charging for billable hours. They're not billing. They're doing the— I'm certain they're actually there doing the work, but they're, you know, they're not incentivized to be able to, you know, to do it in 30 minutes. But if they have to ship to this, you know, before their real day job, you know, they're going to try to do it as fast as possible. So I think that's also— it'll kind of keep the pressure on. Other people. But yeah, I guess all that was fixed. Yeah, all the console logs are gone. Or no, I misspelled that. Yeah, so maybe, maybe it's ready to merge now. So if anyone wants to look at this code and let me know if you like it, we can merge this and we'll have, we'll have GitLab support. Nice. Cool. So yeah, I'll go through these and then we'll be done. Or we'll take questions when we're done. So yeah, streaming test files, basically, as I mentioned before, right now there's just that really slow thing. It just says, you know, generating or analyzing PR diff. And then, you know, 3 minutes later, it shows the thing. It'd be really cool to use Vercel's streaming objects, something like that it's called. And yeah, basically something like this where you're kind of getting the this, the test files are showing up, you know, much faster, which would be awesome. Uh, and one reason I like using Vercel and Next and all that stuff is it just makes it a little easier to do this. Not that much code. Um, so we should do something like that, which would be really cool and also serve as an example for people to implement it, you know, implement streaming in their own places. Streaming is like one of those things that like blow, you know, would blow people away as a UI feature. Like, oh my God, it's crazy. But, you know, will be expected, like will be default expected in 5 years, I think. This I think would be a really good one. Prakash thinks it's urgent. You know, right now I kind of built it assuming, you know, like basically the shortest tech stack, which is just Next, TypeScript. But, you know, I want it to work for lots of things. So the regex currently just looks for this like test. This actually, this is the regex. It's to be— it wasn't showing this because of that. So let's see. Yeah, there you go. So this is the one I use because in the code base, it is the one that— that's how these are set up. .test. .test. which I like. I think it's really clean. But in the Rails ecosystem, it's often like you have a specs folder or tests folder. And so we should be able to basically have a really simple shortest.config.ts that you can just put into the repo in the main level directory. And then all of a sudden, you can kind of, you know, you can kind of suggest, you know, different formatting. Like someone may want like spec. And it's just that, right? Like the entire spec directory contains all the specs. Something like that would be awesome. And then we can add more settings over time. And I like, you know, we could do this as UI. I was thinking, oh, this should live on shorts.com. You like have per repo settings. But I think just like cursor rules, like things should live in the codebase more and more and more. 'Cause that's, it makes it much easier for the AI to see what's happening, for other people to see what's happening. You have version control, you get all this stuff for free when it's in the repo. So similar to, let's see, like Tailwind. Yeah, something like this, right? So you would have, yeah, this is how I would get, see what AI would do. Didn't do it right. I don't know why I kept this. But this is a good example, right? It's like this prompt could live somewhere in the codebase and then the AI could use it to solve the problem. So just delete all this crap. So something like this would probably work pretty, pretty well. Well, you'd only want one of these. So in this case, in this repo, something like that. And then all of a sudden, this, this, the AI would, you know, would need to sort of like look for the, for this file if it exists. Git checkout new branch, git commit AM, git add. Awesome. Something like that. So that would be a good one to do. Pretty simple and probably pretty valuable. To get it to work on way more repos really simply. This is the biggest one, so I'm gonna do this at the end. Oh, this is kind of like the same one. Oh yeah, this is the one I mentioned before where how cool would it be if you actually had tests written and then the AI basically you could have a button that says like build feature or update codebase to get tests to pass. And then it builds, it builds this. Let me link to this so people know what I'm referring to when I say test-driven development. This would be really cool, I think, and keeps it in line in the sort of testing. There's going to be so many, so many AI coding startups and agents and IDEs. I think focusing on tests keeps us pretty pretty focused. This is the one we talked about today. Adding all the dependencies you may need to failing tests would be really awesome. And then, yeah, we'll think more about the kind of approach to this. I'm also going to DM the cursor. I have a cursor dev in my DMs, or mostly I'm in their DMs, but I'll see if what, how they're thinking about this, because I'm sure they must be thinking about how to do this. But yeah, these two are pretty interesting and related, which is right now you have a button, you click, it generates the new tests, you can see the diff, and then you can commit the changes and it'll push a commit. The issue is what if that is wrong, right? What if it actually breaks the tests, not fixes them, or keeps them broken? And so ideally there's some way to test the changes for a green build before committing. I don't know what the right approach is. Maybe it pitches them and then auto-reverts the changes. Maybe it opens a new PR with the head branch as the feature branch so you can build your feature. And as I think about this, this might be the right approach. As you build your feature in one branch, effectively shortest is like spinning up a new branch and like with, and then just like writing, keeping the tests up to date. And then you can, whenever you're ready, you can just merge that. Into your, into your main branch. This would be really cool, but also massively expand the complexity of Shortest. Maybe this is the right thing to do, but I don't really know how we would do this for all sorts of different code bases, right? Which is basically like, if you're using Rails, RSpec, then all of a sudden we have to like somehow spin up like a RSpec runner, you know, and Rails requires like big beefy servers. And if you're using MySQL and like, it's just a lot of stuff. So I don't know, I think I think the right answer is probably something like open up a new PR with the head branch as the feature branch. That way you can just kind of— sorry?
Sorry, how does the custom test runner solve this problem? I'm not following.
I guess he's saying that basically you have the whole test suite run as part of the generation of the tests. And then, like, when you hit commit changes, or when, like, it actually tests all those changes in a custom test runner. And then once it gets a green build, it commits the changes. So basically a server, oh yeah, here it is, a server where its primary purpose is to execute tests and report the results. So I think this is going to be tough. I don't know. But if someone wants to mess around with this idea, I think, I think it's really cool. It's not that I think it's a bad idea. I just don't know exactly how you'd pull this off or like, like I think you could do it if everyone, if you're just like, hey, we only support Next, 'cause then it's like, you know, pretty simple. But that's not most codebases, so I'm not sure what the approach should be. I do think this is probably a good one, which is basically, and this also kind of solves this other issue. Which is— autofix is really cool. Autofix is basically, you know, just kind of commits to your feature PR, and then all of a sudden, like, you just— your linting errors are just fixed. You don't, as a human, have to do anything. But, you know, with autofix, it's kind of deterministic what the fixes are, because you're just changing some text. With writing new tests, it's, it's a bigger deal. So I don't know if this is like the right approach. So what does this guy say? He says he may look at this during the weekend. Possible, but yeah, so some sort of polling and a Vercel cron. But if we did, so you're kind of watching, you know, you're watching a pull request branch. And sort of taking it and then auto-committing. So I think, I like this. It should come from our own accounts, but maybe not. Maybe it's the right thing. So yeah, I think the right thing to do, if we did that, we'd have to Yeah, I think the right approach is probably a new branch, and then you can auto-commit to that branch. So I'm gonna— I'm actually gonna merge this into the other one because I think they're very related.
I think it might be worth also looking at some of the GitHub bots, right? Like they have Dependabot and a couple other bots that are following this kind of UX where they are keeping an eye on the dependencies of an npm package and allowing you to auto-merge if something has vulnerability or it has become outdated. Yeah. So, like, you know, keeping within that workflow might be worth staying familiar with people's UX today.
Yeah, I, I agree. Something like that, I think, would be really cool. Um, the other, the other thing that this would allow us to do, uh, which is I think very powerful is do this kind of the self-learning agent, agentic behavior, which is what people call it, which is often the AI won't get it in the first shot. But if you have like a multi, like you have 01, maybe look at what Claude 3.5 Sana has done. And you could say, hey, this is a series of commits that Claude has, you know, that an AI has made to try to do X. But it hasn't worked yet, you know, what changes would you make further in order to fix the build? And so you could have, you could see, you know, I mean, it's not too different from like when you look at a PR that a human makes, right? You often have, you often see stuff like fix build, right? So you could imagine that like an AI could sort of have one commit, which is like write tests or update tests. But then it's gonna fail and it'll, you know, you can see that this has the check, but this doesn't have the check. And so if it doesn't have the check, it can kind of keep running. And I think what autofix does is it does like 4 commits total. So you can just say, we're gonna, it's gonna try 4 times to get it to a green build. And if it fails, then it fails. But this kind of is a good example of like maybe the AI could have done this part, right? This is kind of tough, like, I don't, I don't, maybe an AI won't be able to actually run, actually, no, it should be able to run this, uh, sort of pnpm install. And then, you know, this is often like the issue is that someone works on this locally, installs it locally, but then doesn't actually run the command to put the package in. So it fails the CI build. Uh, that's something that AI could just be like, oh, clearly they have an update to package.json, but there's no, imagine this file didn't exist, right? The AI would run pnpm i and commit those changes, which would be really cool.
Quick question. Yeah. As we were talking about running a build for testing for green, currently, at least what I noticed when you refresh the page where there's something running or you have the instance of the test generation before you commit and everything, it just resets itself. It's not a problem now. The most you lose is just one, the tokens that you were, you know, working on the test. But I think if we introduce more complex orders of build, it would be a nice thing to have like a URL. Like you can, you know, you can even, I don't know, leave the site or something and then come back and it's fine. And then the URL gets— I'm— the URL gets deleted from the database. But that's like for the future. I just say definitely if we do more complex stuff than just, you know, API calls.
Yeah. Yeah, I agree. Yeah, I agree. I agree completely. Yeah. Like ideally people, you can kind of like spin things up, go to bed, you know, close the tab, come back the next day and it's done, right? Stuff like that, I think is certainly where things are going with AI. So. As things get more complex and there's multiple commits, uh, because, you know, maybe this codebase is really large, you can only write like one test file at a time, or you have to go section by section. Like, all of these things, people will figure out like the right ways to solve these kinds of problems with limited context window. Um, and maybe not, maybe that actually like we get to infinite context window, but it takes 7 hours to write the tests. Then totally you should basically have like queueing or some form of asynchronicity, I think would be really cool. Awesome. So yeah, that was a good chat. Anyone else have any questions or comments or anything like that? I'll stop sharing my screen.
Further design system, uh, do you want it to be similar to the Gumroad brand guidelines that you released recently, or, uh, no, not necessarily.
Uh, no, go crazy. Um, yeah, if you want, if you wanna, if you have suggestions, uh, I'm super open, uh, to the, to, uh I actually want them to be a little different. Uh, everything Gumroad, Flexile, Iffy Helper is like all slightly different, which maybe I'll regret and everything should have a similar design, but I like the idea of, of basically, you know, for the specific problem, you know, each, each customer is kind of different. Uh, Shortest, you know, is going to have like a more technical user than, than Gumroad, for example. So, um, you know, Iffy has like a more technical user, but not really as much as, as Shortlist. The only, the only brand I've done is I even forked, uh, this next SaaS starter kit Lee Robb did. So even the colors and everything are just from that. So there's like almost no, uh, I haven't spent too much time on the brand. The only thing I did for the brand is I took, I actually asked AI update the logo using only HTML and Tailwind. To mimic the brand of Shortest, and all they did was it took the S and it like made it like a little short. So if you go to shortest.com, you can see like the S is slightly, it's like 75% of the height it would normally be. So that's like the only, the only thing I've done. But yeah, I think design is a little trickier than engineering. It's less, you know, it's more opinionated, subjective. So there may be some back and forth. I might have some more annoying opinions about what a good design is, but I'm definitely open to that. So yeah, if you want to contribute in that way, like, would be awesome. Definitely. Awesome. Thanks. Sweet. Well, we're at the hour, so I'll wrap this up, but it was really great to chat with everyone. We'll see how the next couple weeks go, and we'll probably do this. My guess is we'll do this in in like 2 weeks or a month or something like that. But if crazy stuff happens this week, maybe it's worth doing sooner. So just follow the repo, the GitHub discussions I'll use to kind of remind people when we do the next one. And catch you all, catch you all online.
Yeah, thank you.
See ya.
Yeah, that's great.
Great to meet you. See ya.
