Shortest OSS, QA via AI - Contributor Call

And is this being recorded? We have my friend AntiWorker here, he'll take notes for me. How's everyone doing? How's your holiday New Year's season? Where are you all going?

Quite adventurous.

You what, sorry?

I said it's been great. It was quite, quite adventurous.

Oh, adventurous. Oh, exciting. Nice. Awesome. Well, we'll see if anyone else joins the call. Raz is joining. Nice. And let's see if anyone else will come. Oh, my camera just dies. Um, cool. So yeah, I'll just share my screen and we can go through Shortest and kind of the, the happenings since, uh, since we started working on it and what's coming up next. And yeah, I think it should be a good place for people to go who want to contribute and kind of know like why, why it exists and why we're excited about it and what's coming up next and how best to help out and all that good stuff. So let me share my screen. I guess I'm opening up one more PR to— as an example. Awesome. Oh, people should see my screen. And I will go through— this is kind of our internal Slack on the left. This is kind of the Shortest roadmap. It's pretty simple, but it's kind of a high-level overview of our priorities. Uh, so I'll just go through it. So Shortest started out as a Kind of basically what happened was Anthropic launched this thing called Computer Use. And many people have had this idea, I think. It's not that original of an idea that it's annoying to write tests, right? Like it's, it feels kind of like you may build a feature in like 10, 15 minutes, especially with like AI and Cursor. Like building the feature itself is, can be quite fast. And then you spend 80% of your time building, uh, like just writing the tests, you know. And generally, like, because tests are brittle and hard to write, uh, this idea of like test-driven development doesn't really exist in like— maybe if you're building an API or something like that, but generally most people, you build a feature and then you write the test at the end. And the tests are mostly like regression testing to make sure that like someone else's change, uh, changes in the future don't break your feature in the way that you want your feature to work. So that's how tests have worked for a long time. And often you'll see it in PR that's like 800 lines, like the product feature is like 200 lines of code and then the tests are like 600 lines of code, right? Or if you're working on something like SQLite, it's, you know, might be like a 1 to 100 ratio. And so when this came out, Claude 3.5 Sonnet and computer use, it kind of made me think about, oh, maybe there's like a better way to write tests, a way in which you can, just like you would tell a QA person what to test for in English, you would be able to tell an AI what to test for. And instead of having to have like, you know, fill in, you know, find input with this, you know, name or class or ID, fill it in with this value, etc., etc., for the whole form, click the button, go to the— wait for the next page to load, you know, like all the— all that stuff, you should be able to just say like you know, test that buying a product works, right? Like, why do you have to specify, like, the credit card number or the— like, just go to gumroad.com, pick a product, use the Stripe test card, and just try it, you know? And there's a lot of pushback against this idea in the development community. Generally, you know, people are like, oh, it's slow, or it's like probabilistic. AI is like probabilistic. It's not going to work. Tests should be, you know, secure and like, you know, fast and deterministic. I think these are all like problems that will be solved in the future, if not are already solved. And if you actually play with Shortest, you'll, you'll see that it's actually not really that much of a concern. You can still write tests that are deterministic, right? You can still use Vitest or something like that to write tests that test that, you know, an API call returns a certain value. But inevitably what you end up doing is mocking like most of the code so that you're testing some stuff and it's still important to make sure the JSON schema doesn't change or something like that. But at the end of the day, you're not really testing a lot versus the way to do it with, with Shortest, in my opinion, looks a lot better, right? Which is something like this where you're actually testing create an API key from the developer page. Do not close the— even this, this is like a bug that is kind of we need to fix, but really just like create an API from the developer page and then fetch that API key and use it. And we just added bash tool testing so we can have it even do like just use that API key to make a curl request that looks like this and make sure that the value returns. And in my opinion, it's just like a lot better way of writing tests, and over time it'll become more and more popular as a way of doing it. And so, yeah, so that's shortest— that's kind of like the origin story. You know, people should be able to write tests that not just engineers can read, but even designers and product managers should be able to read. These are all— and then if you need to do stuff like this, you can still have this kind of deterministic quality to it, right? Which is like, once you've logged in, you may want to test that the user exists in the database, right? Um, but to me, this is a lot— like, if this were truly just as fast as, as like, you know, doing XPath selectors in Playwright, like, to me, this is so much better. Developers will inherently have, I think, this like distaste for it because it's kind of like if, you know, like the first time you see HTML and JavaScript, you know, and React or whatever, everyone has this— Guillermo from Vercel calls it like the biting of the lemon moment. Like you bite, it just gives you this sour taste. And then you acquire the taste over time. You realize like why lemon, you know, tastes good. And so I think that's kind of what will happen here. And so there's a lot of cool stuff we've added recently. Amin added like, for example, chaining tests. So you can now chain tests and reuse them so you don't have to specify— like, let's say you have a bunch of things that happen within an app but different user roles. You could just test login as lawyer and then test all the, all the app actions. So I think eventually it'll be way better than writing tests the traditional way because you'll be able to do things that tests can't really even do. Tests are pretty limited in what they can actually test for. And so if you truly want end-to-end testing, like this is what we added recently with API testing, you can do something like this where you can have your string of what you want to test for and then the actual curl kind of like model, or you can just do it in English and it'll work as well. And AI will do all the hard work of actually figuring out, you know, the sort of what, what to do at each step. And then we've also added caching. So now we have caching, which we should probably talk a little bit about somewhere in this. Readme. Ideally you don't have to think about it, but caching is a way of basically saving screenshots and browser actions such that you don't have to wait for like potentially the call to come back from Claude of like what to do next. You can just see what the, you know, what was done last time, which is like click this button which is in this sort of XY coordinate. And then the last thing I'll say before going through all the issues is we seem to have some traction. Which is pretty cool. We launched it like the week of Christmas, and since then we have, yeah, about almost 4,000 stars, almost 200 forks. We're shipping updates to it. There's a lot of contributors, there's a lot of activity, so I'm pretty excited about it. We were number 1 on GitHub for I think 2 days maybe. Now we're probably not anymore, but I think eventually we will be again, and we're going to include it So this PR that I was showing off, we're going to include it in all of our repos that all of our projects that we're open sourcing over the next quarter, Iffy, iffy.com, helper.ai, flexile.com, and Gumroad, which is the big daddy. All of these will have shortest, you know, included in their, in their repo. So they'll drive more and more traction to shortest. And we can also— it's a good way of actually testing. Testing that Shortest works right in production environments and has the functionality we may need for, you know, for real production use. And eventually, once we have some sort of form of paid product that will live on shortest.com— I'm not sure exactly what it will look like— then we can use these 3,000 people as basically lead gen. And so we'll just go through and email the Hyundai Motor Group and be like, hey, what do you guys need? You know, do you guys need to make sure that people can buy a car on your website? Or like, what's the deal? And there's tons of— yeah, this is basically, in my opinion, like better lead gen than almost anything else because these are all developers. Developers are the most fun people to sell to. They have the most power in these organizations. They make the real decisions. ChatGPT, Dwarf, ByteDance, TikTok, Microsoft I saw the other day. And so I think we'll— I think Shortest has potential, I guess, is like the short of it. I think there's a lot of potential to actually make this a pretty interesting library that people use. And the early traction hopefully is a sign of that. But even if we didn't have early traction, we'd keep working on it because I do believe Eventually, as, as AI gets better and better, like this idea that you would write tests that, you know, like look like— like, let's just have Claude generate one. And it's of course going to use an artifact because Claude loves to write code. And I mean, now with AI, at least this is a bit faster, but it's still like, I mean, think about how much complexity that is. Let's say you change the onboarding flow and you have to like redo all of these tests, uh, right? Versus the equivalent of just being like And the cool thing is you can always specify your constraints. So the constraints that actually matter to you, like if you really want to make sure that they're, you know, picking a certain team size, right, you can always just say that with team size of 10, right? And boom, you have solved that problem. And a PM can go read this, the CEO of the company can go read this and be like, yes, I agree that that's the right approach. And I think that there's also going to be this this evolution of building products, which effectively, if you're writing a prompt to generate the product feature, I basically expect, expect this kind of parallelization where you, you have a prompt to build a product that will spit out a set of tests. You'll confirm that the tests test what you want it to test, their specs for the product specs. And then effectively, once you've kind of confirmed that, then the, the AI agent will actually go write, build the feature, and basically it'll know that the feature is built properly when the tests go from failing to passing. So you'll actually have this like sort of prompt-driven development, which is like test-driven development, um, I think actually becomes possible if you're able to write tests like this, uh, otherwise you'll never really get there because you're gonna— if the test fails, you're not sure if your code is broken or if, you know, you mistyped like the label, right? Like what if this label was like password confirmation or just confirm? All of a sudden the test will fail, right? But an AI knows like, oh yeah, I need to enter. You don't even have to say anything about confirm password. It knows that, you know, if there's a password field and a confirm password field, you're going to have to enter the same password again. So yeah, so we're going to, we're going to kind of ship it in all these places. We've shipped a bunch of, a bunch of things which I guess you could see in the releases section. So yeah, chaining tests. I just added this before function so you can now have before and after bash tools. Cloudy shipped. Krabus, I mean, has been shipping a few things. Oh yeah, like ChainTest, that's a big one. Types are automatically generated now. Yeah, so things are, things are, things are, are getting better quickly. And I think this is a good opportunity because as people come back from the holidays, you know, ideally they're telling their engineering buddies like at the company like, hey, we should use this. This is actually like way better than doing it the old way. We added Mailsaur, so you can now do like magic link login, which solves a whole set of issues. Like all these people are like, oh, you need to like have a, you know, complex database setup. It's like, well, you can just like, it's not that hard if you have a temporary mailbox on Mailsaur and you just like generate an email. You put that email in your GitHub Actions and your GitHub repo secrets or whatever, and then boom, you can like now test login. Once you're logged in, you can test the entire app. Pretty quickly. And yeah, it's pretty, it's pretty cool. The other thing, one last thing that I think is really underrated about this approach is, is the chaos of an AI, like the fact that it is a little bit non-deterministic, is good because that's what QA is. Like QA is testing things that the engineer would not have expected to test. That's why you have QA and not just purely regression testing, right? They're like, oh, I entered a negative number Right. So you could, you can do a lot of sort of things like that as well. You could, you could say, you know, shortest test weird stuff. Well, and it'll just do stuff, you know, you know, make sure the UI isn't janky, make— look around, make sure the UI isn't janky. And as the AI gets smarter and smarter and smarter, you know, you can define like what janky is, like what, you know, and like buttons shouldn't touch or like things should be vertically aligned properly. All these things that designers have to tell engineers to fix, you know, after the fact, they could just sort of codify in the test suite itself. So yeah, so anyway, I'll go through these and then I'll open it up for discussion and questions and see what people think. So yeah, so, so glad he opened up this. I mean, think about the activity too. Like, we've shipped a lot, but the oldest issue is only a week old. It's pretty cool. Yeah, so this is pretty, pretty simple. Basically like having some sort of watch function. There is an attempt at it, so I think he'll look into it again. But basically, you know, make sure that like if you're working on shortest, it's kind of annoying to run build all the time, so just like automate that with like the, you know, update the dev function such that it's automated. This is a pretty big one, I think, which is just generally finding ways to speed up test, test, execution. Probably the, the most annoying thing about AI and computer use and Claude is it's not instant. Like, you know, if you're running Vitest, it's like instant. Um, eventually it'll be probably a local model that you could even run. Like, I think a lot of this stuff, you know, this like web stuff will like eventually these things will get faster and, and, and, you know, more efficient and be able to be run locally and open source and all this kind of stuff. But yeah, Muhammad suggested some ideas on how to make this better. I had this— I— yeah, this is one I actually pinged. I have a call with— this is a nice thing also about having a popular quote-unquote repo is I can get in contact with the Anthropic team and ask them for priority stuff. So I asked them if they could support something like this. Right now every action, it's kind of like atomic. It would be cool if you could batch actions like filling out a form, for example. What if you could add in one callback, get like a full form filled out, kind of like autofill? That would be— that would make things a lot faster. Yeah, adding screenshots automatically instead of waiting for the AI, we should probably just kind of, you know, side on the, on the side of taking a lot more screenshots and seeing if we can kind of use the cache to get through those things faster. So yeah, oh yeah, running tests in parallel. Currently Shortest doesn't do this, but we probably should support something like this. The projects that currently use Shortest are probably so simple right now that they don't need like deep parallelization, but if we wanted to replace the hundreds of end-to-end tests that are in Gumroad, or the thousands maybe that exist in Gumroad, we would need some form of, you know, parallelization. The cool thing is because it's, it's, uh, it should be possible to parallelize them much more than what we, how we currently do it, because you're just loading up the website. So you could have like 100 browsers running at the same time. Uh, so yeah, so there, there are a bunch of ideas, uh, So yeah, I think both running parallel tests and waiting for Claude to get around to this will help quite a lot with making these things faster. But it's not, it's not too slow. It is actually, especially with the caching that was shipped. It's not too slow. Another thing is cost. You know, computer usage is not free. And so luckily Gumroad, we're kind of, I don't know. Willing to make this investment. Engineers are more expensive than tokens generally, but it would be cool, especially for like indie hackery type projects, to, to add something like this to the test execution log. So, you know, kind of underneath this is— I think I asked O1Pro to like give me a good suggestion of where to put it, and it basically did this. I think I added this little squiggly. But yeah, basically you should— it would be cool to list how many tokens were used, and maybe this is even a way that we can start to optimize performance because it kind of correlates. Like, the cheaper it is, the faster it is generally, which is great. And then once we have a local model, this can be zero, right? Like, once you're running it, eventually there'll be a way to ping a model through GitHub Actions that's not Anthropic. And the shortest API should make it easy to kind of select through them. Currently, I think Computer Use is the only one. Claude is the only one. Someone did ask here if we can support OpenAI, but I think Claude is the only one. So apparently Gladys thinks it is possible basically by just doing what Computer Use does manually, which is you know, taking the screenshot and having a prompt that says like, what would you do next? Uh, but I, I, I think in, in Computer Use's announcement, they say the way they were able to do it is to specifically train the model on doing that kind of thing. Uh, and so I think on a general client, uh, like OpenAI stuff currently, it won't work that well. The predictability, reliability will probably be too low. Is my guess. But if anyone wants to try that, feel free. Debugging. This is probably, this is probably pretty important. I think once you have— I think this is generally true with software. Like, once you have like basic functionality, it's not really about adding a ton more features. It's tempting. It's more just about speed and reliability. I remember I had dinner one time with one of the Instagram founders, Kevin, and I asked him like, what was the most important thing that you guys did on Instagram? Feature-wise. And he's like, honestly, features like don't matter. Like once you've launched, like just speed. It was like speed, reliability, and bug fixes. We're like, that's it. Just make your app insanely fast, insanely reliable, no bugs. And as long as you have product-market fit, like that's mostly what, what people want. Um, so yeah, some, something like this where, yeah, this is a small change. You should probably change debug AI to just debug. And then just have like a logger service that you don't have to, you don't have to have like if, you know, it just log, log, log, and then in the, in that service itself, it can decide whether to console.log or whatnot. But I think this will become more and more of a thing as more people use it. They'll have some issue where it'll work in one repo but not another. Like this, this person had an issue with this repo. I'm not sure why. But, you know, it has some traction, so, you know, if we got it to work in this issue in this repo, all of a sudden, you know, all of these forks would then have shortest in it, right? So it's a really good way of like spreading, spreading shortest around. That's another one. The DOM, making sure that the DOM is settled. Right now we use this like setTimeout with, I think, with 1,000 seconds. It would be cool to— I kind of changed my mind on this. I think we probably will need to rely on Playwright for a while because there's a lot of stuff they do wrapping around these browsers. But yeah, there's some room probably to improve on making sure the DOM is settled without having to wait 1000 seconds. Probably makes— will make things faster as well. This is a new issue, probably to do with Windows. If anyone has Windows No, I think everyone who worked on this is using Mac or Linux, so it's kind of hard to debug, but there's probably some issue with, with Windows and we'd have to work around that. And then the last and maybe most important potentially feature-wise is to get it working on mobile, so people who are building apps like right now Shortest is great if you're building a web app. Especially like deployed on Vercel, because you can just specify the preview URL and you're, you're done. Uh, you can just browser use— the computer use will just run on that URL. But as more people, you know, we want— the future is mobile. Uh, maybe the future is Neuralink or something, who knows? But for now it's mobile, and getting iOS and Android, uh, support working would be, would be really cool. And then just to, yeah, reiterate the bounty thing, we're building this product called Flexile, and Flexile helps us track— it does equity management, shareholder management, team management. It's kind of like Carta plus Deal plus Rippling all in one, and it'll allow us— we're building a feature that will allow us to do this. But basically, the one idea we're playing with is rewarding open source contributors with equity and dividends and profit sharing as a way of like not solving, quote unquote, open source. I don't think it's broken, but encouraging and incentivizing usage and just what I think is good. I mean, it just feels nice too. So basically, you'll— these— once we have the feature built, hopefully sometime in Q1, I'll kind of go through all these issues. And right now it's just plain text. I'm a fan of natural language, if you couldn't tell. And then I'll go— just go through and basically you'll have an option of If you want it all in cash, fine. If you want half of it in, in, uh, in, in Antiwork, which is the parent kind of company of Gumroad and all of our other— so you'll get Gumroad, uh, dividends through, you know, gumroad.com and all of our other products. And the plan is hopefully over time, like, as we have more projects that are all hopefully profitable and making money, you get like a chunk of that, of those profits via dividends distributions. Forever, uh, which I think is a— yeah, could be a cool way of— especially as AI and agentic stuff happens over the next coming years, might be an interesting, interesting business model that gets developed. There was actually a company that inspired this called Coffee and Power, or no, it was, uh, what was this guy's name? Philip Rossdale. Love Machine. Yeah, Love Machine. He had this company called the Love Machine where he tried to do this like 15 years ago, and you're gonna have to buy the article, I guess. But basically it was this similar idea, like basically a company that was like staffed by freelancers around the world, uh, and he never was really able to figure out. It was a bit too weird at the time, but I like the idea of having bounties, people assisted by AI can ship these features, then you kind of have this like marketplace of projects that people can contribute to. And if it works for Shortlist, we'll do the same thing on all of our repos. And we'll still have a team because it's important to be able to ship fast. You can't rely on random people around the world to ship something quickly. You still need product management, you still need design. There's still a lot of things you need. Architecture, actually, like fleshing out the roadmap, but a lot of small tasks, tech debt, bug fixes, things like that, I think may make sense to put in this, you know, tech debt, like, like rewrites, technical rewrites for, like, migrating from one stack to another or something like that. Like, for Gumroad, I want to move to Tailwind. There's probably like 200, you know, tasks as part of that that we could have bounties for each one or something. OK, that was a lot. Any questions? Any ideas? Any feedback? Any Anything? Yay, Rodrigo, what's up?

Hello, good morning everyone. I'm kind of new to the project. I've been checking on the documentation. It was suggested by me or one of my colleagues, and I'm a little bit concerned regarding the scope. The project doesn't say what is the scope of the testing, and I'm feeling that we are throwing different levels of testing in the same code, or the output is to be different levels of testing in the same code. And there's no clear boundary on what the project is capable or what we want it to be capable of doing. I mean, for example, unit testing, API testing, end-to-end testing, it can, like, it's shooting everything. Everything at the same time. And there are advantages to having boundaries in those levels of tests. So that's one of the things that I'm worried about. And the other thing is that— and we can discuss this later in a different call— if we want the big players to use this, it doesn't matter if you have— if the AI is capable of writing 100,000 tests in 1 minute and executing them during the day if you don't know what those tests are testing. What is that those tests are testing, right? So the traceability of the test, what feature is it covering, it's important to those guys because they want to know which of the 100,000 tests is relevant for the actual feature that is being shipped. So I didn't— I don't see again anything about traceability on the, on the project, and I think it's an important feature for it to have. I don't know what are the thoughts on that.

Yeah, yeah, I mean, I do think that of those 3 things you mentioned, unit tests, API tests, and end-to-end tests, the focus for shortest should be on mostly on end-to-end tests. However, end-to-end tests inherently may include API testing, for example, because if you have an API, like a core part of the end-to-end test is does the API work? You know, like does the API key that was generated with the UI actually let me do the thing that I want to do? That's in my opinion included in the sort of end-to-end test. I don't think unit tests make probably a ton of sense to cover with shortest. So maybe we can sort of add more, like, you know, clarify that this is more end-to-end testing specific of a framework. And you can still do like unit testing or like you can still always have things like this, right, where you have like an after block. So you can get pretty close to doing that. But generally, I think a lot of the value is in QA, is in making sure that the site doesn't break, that the core functionality doesn't break. In terms of traceability, there is probably stuff we can add to make it better. I do think that it's not— it kind of works like we, you know, it operates similar to how we run things at Gumroad where there's 16,000 tests that run in Gumroad. To merge your PR into production, all 16,000 have to pass. And, you know, if you break something, the test, you know, the test will fail. But you want to run all tests. I think the minute you get into a situation which you're like trying to be smart about which tests to run, it very quickly defeats the point of the whole testing suite because like the whole testing suite's point is to have like incredible certainty that you're not breaking anything, right, that you're not thinking about, like you're not aware of, because the code bases are millions and tens of millions of lines of code, and no developer can really make sure that their change is not going to break something somewhere else. Uh, and I think that's a lot of like the kind of important part of what, what a test does, especially as AI is writing more and more code. You'll want to make sure the tests will work. I do like the idea of these natural language tests because maybe there's a way of being somewhat smart about it at least locally if you're developing, right, where you want to just run the— you might be able to just like basically take all the strings and the strings are pretty self-evident like what they're testing for and you could say, hey, I built— this is the— this is like the code diff of my PR. Run, you know, go through this array of strings and tell me like which strings are relevant to testing this feature, right? And then on the CI build, you probably want the CI to run every test, right? All shortest tests should run. But I do think there's probably like— the cool thing with computer use is there is a lot of cool stuff you could do in terms of traceability. Like, for example, you could track all the URLs that are being changed and you could sort of like almost give a sitemap of all the tests via the routes that they're available on. Because that's one thing, you know, that an engineer has to do is they see like, let's say 40 different tests fail and it's not— you have to kind of look at them and figure out like, okay, based on what's failing, I think the issue is this model, you know, that is probably being referenced in these 3 different places. The problem is probably at the model layer, at this model specifically. But what would be really cool is if you could like like on the, you know, you could give a very clear, like, statement of, like, what's actually failing. Like, it's this specific settings. Like, the settings page is not loading. Just like a human, right? Like a human when they report a bug, they don't say, hey, 40 tests fail, right? Like they say, like, the settings page isn't loading and this is what's in the console, you know? And I think there's a lot of room to to improve the— I had a tweet the other day that was about this idea. I was talking to, uh, someone, François, from the Arc AGI Prize thing, and he was saying like, why, why is, uh, like TensorFlow and, and now what— well, now what do people use in like the Python community? Everyone uses like the same thing. I forget what it's called for, for machine learning. Um, but he, uh, he basically said it's about debugging. Like, it's not about writing code, it's about like how fast when there's an issue can you debug the issue. And the winning frameworks, the winning languages, are often those that prioritize the speed of solving. Because you get blocked— if AI can make you go like 1,000 miles an hour, basically you're spending all your time still stuck in traffic. It's just like when you're not stuck in traffic, you can move really fast, but you're still spending most of the time not moving, right? Uh, and actually, as AI gets smarter, you're going to spend more of your time not moving because you're getting to the next stuck point so much faster. Uh, and so I think, yeah, like it should be a core, core tenet of Shortest and like almost like a value that like we should prioritize in debugging and in a way that no one else can even compete with in a way, right? Because like we're able to debug, like do things in the— with like our eyeballs in a way that like most testing frameworks like Playwright can't really. They don't have access to that kind of stuff. So kind of an AI-first, I guess, approach.

Thank you. To summarize then, what I'm hearing is that the project is more for regression testing, right? It's focused on regression testing at the highest level of testing, which is UI or end-to-end?

Yeah, exactly. I would, I would say so, yeah.

Because— Got it. Now a quick question. So what is it using as a base for generating the testing, the UI, the code? Because that's one of my questions as well. And one of the advantages of having different levels of testing is that if the UI is not ready, you can still write API tests, right? If it's using the code or if it's using the UI, you have to wait until the application is deployed in order to generate the tests. So that's one of the things that I'm— the operational aspects of the implementation of this project is my concern because tests are usually done at the end or left at the end.

Yeah, I think that's a good point, uh, and that clarifies it a little bit too, I think, for me. Like, I think we should focus on visual regression testing, um, and you're right, like, at larger companies especially, you may have like the backend done first and then the frontend done, and then you can only write the end-to-end tests like once that entire thing is, is done. I personally think that's not ideal anyway. Like, ideally the org can work a little bit more simultaneously. But obviously big orgs, remote orgs, you have to plan, you have to, you know, yes, whatever. But I do think, yeah, like you'll still likely need unit tests, API tests, those sorts of things. And separately from that, you know, like I think of Shortlist mostly like a Playwright wrapper. You know, like you may not need to engage with Like if— like our competitor, quote unquote, right? Like you won't probably replace Vitest or Jest with Shortest. But you may say, hey, these 300-line Playwright files that we have— and by the way, none of us have like touched this stuff. None of this has changed in a long time. Or we can like replace this with these Shortest tests that are just like login, buy a product, you know, like the core kind of functionality of the product. I think for Gumroad, for example, we have 16,000 tests including unit tests and about 600 core, what we consider like kind of core integration tests, like core end-to-end. So those 600, you know, you could move to shortest and, but maybe even more of the other ones that are end-to-end but not considered core. Maybe there's like 2,000 of those. I think we could move to shortest.

One more thing.

Sorry. Yeah, no, please. It's great.

So currently, as parallelization is not part of the features, and as there is no way to selectively choose the test that you want to run because it's going to be a kind of like whole regression thing, I think parallelization should be a priority because sometimes if you don't have the time to run all the suite. You need to just run those small tests that will test the feature that is covered in the change. So if you cannot select the tests that you want to run or the level of testing that you want to run, you should at least be able to parallelize the running in order to speed the testing. I don't know if I made—

yeah, that's a good idea. There's, there's— let me show you one thing that you might like. So one idea I had for shortest before this one— this one is definitely better, I think— is this script that I wrote that I actually based off of Razvan's script, so he gets most of the credit. Okay, I don't know what the hell is going on. Why? But basically what this would do is It will use AI and it would basically— how do I— can I just— can I do like a word wrap or whatever? OK, here, this is better. Basically, it would go through the code, it would go through the specs. And kind of like what you're saying, it would look at the diff and it would get— and it would— the file system of all the spec files. And and basically determine to a certain degree of confidence level what tests to run. Then I just use GPT-4.0 and went through and did that. If you see the code, the way it works is right at the bottom. Yeah. This is what it does. It basically goes through and it does like 80% confidence, interval, 95, 99, and then 99.9, and then 100% would be just running the build, so you don't really need something like this. And then what it does is it prioritizes in that way, right? So, it basically— like let's say you're making a bunch of changes to like the login form. Like 80% confidence interval, if you just run one test, which is like the login integration end-to-end test file, you know, and it fails, you've done something wrong, right? That's a pretty good indication of that. And so, it would run that test first. It could be done in like 15 seconds. And then only once that passes, then it runs like more and more and more until— and I actually love this idea. Like I think there's something like— yeah, smoke testing. Yeah, exactly. I think most people, you either run— you either know what tests you're gonna run or you run the entire suite. And there isn't like this nice middle ground. And when you're building something locally, you, you kind of want something like this, I think. And especially as computers get faster and faster, I think more people will want to have like their entire CI run on their local machine. And so I think something like this may make sense to include in Shortest itself. This is actually called shortest.sh because that's— I thought it was like funny to be like, what's the shortest way of running a test suite? But But I think, I think this QA tool is a much better idea. It's much more like useful to people. This is kind of like a gimmick a little bit. Like people are not going to merge something with 8% confidence probably. But I like the idea of like if we build parallelization into shortest, then we don't have to do what everyone else does, which is just like parallelize blindly or by you know, a test length to try to like have all the, all the builds finished at the same time. That might be a good start, but actually using AI to like be smart about it and say based on this specific, uh, code diff, we should run, you know, 80%. And you can actually run these in parallel in a sense, right? So like this may finish in 15 seconds. If this fails, you can just cancel these builds, right? Um, or you could keep running them, but you'll have the failure like the engineer can now start iterating or the AI agent, let's say, can be like, oh, this failed, I can go fix that. Instead of having to wait for 15 minutes or 10 minutes for all of these to finish, they can start iterating and pushing code. Something like this could be pretty cool to integrate.

You can spin off 10,000 containers and run the test in parallel there. Concatenating the results is going to be a pain. So that's what parallelization works in that sense.

Yeah, that's true. Yeah, you won't be able to combine the, the test into a single kind of like end result. And then locally, right, locally too, I think the idea that you can just run them locally and very quickly like get, get a good result that your changes are, are valid. Or maybe you don't want to, maybe you want a failure. You're like looking, you're like, oh, this test shouldn't be passing, and then you need to refine your shortest kind of English to make sure that you— like, for example, you may want a specific text on the login button, you know, like one test I love writing is very something you can't really do anywhere else is this one. Boom. Like, how many times has you— have you merged a PR that functionally works but there's typos and now you have to— someone, you know, like, it happens all the time, uh, and you can have a shortest page, uh, like a just like a test file that just tests that like no typos are going to make it into main, uh, And you could— the cool thing with this is this is not competitive with anything else. Like, you can still have Playwright, you can still have Antenna, and you can still do all the other things. This is like— and maybe this is the direction and shortest sense of going in. I'm not sure. We'll see based on the cool thing with being open source. Like, we'll see what people use it for. But I love that some stuff like this that I think you could, you know, if something keeps annoying you, like, oh, people keep messing this up, well, prime test, you know, and the build should fail. Like, a typo is like a— is a build failure, in my opinion. Shouldn't count.

So I didn't know you could do that. And it's also interesting because with AI, you gain the advantage of localization as well. You can try this in different languages immediately.

Yeah, 100%. You could say, yeah, visit all these pages in, you know, as a Spanish user or as a German user, and you can chain them so you don't have to rewrite the entire thing. You can do like log in as someone from Spain and then boom, all the tests run for that instance. So, yeah, there's a lot of cool stuff. And as the sort of computer use CI stuff gets better, I think there will be even more interesting— like once we— now we have this, like we should be the first— like anytime Anthropic launches like a new tool for computer use, like scrolling or sliding or dragging or whatever, like we can be the first to try to get it integrated and could be— yeah, like one thing that, you know, like another one is dark mode, like, or responsiveness, right? Like make sure responsive, you know, mobile layout isn't janky, you know, stuff like this. Like like stuff like that, like boom, like designers no longer have to wake up in a cold sweat being like, oh, I wonder what the engineers did to my beautiful design. So, the test suite for designers maybe is a good tagline. I don't know. Awesome. Varun.

Oh yeah, so like I saw a comment in one of the issues which says that on a future thought we can just delete the web app and focus on the package. So like, is it in the plan or we have something planned for the web app too?

Yeah, so I have one idea. I'm not super worried or concerned about it, to be honest, because I think the package is the— is mostly the priority, but as I mentioned, but I do think there's some interesting ideas. One idea would— one idea that's like maybe like more obvious is to just run a managed service, right? Like CircleCI or Buildkite where, you know, Shortest effectively can set up containers and run, you know, parallelized testing and even like Imagine you could just like all you have to do is put in your domain and then you can have like a UI to like have your tests, your shortest tests. And then any time, you know, like there's a cron job or any time there's a deploy, you know, it just runs the tests and boom, like you have— like let's say you're— I don't know what's a good example. The problem is I don't know how you— you kind of need like a— like you need those tests to run before you deploy. So, I don't know exactly how it would work. It's kind of like a Pingdom alert. Maybe it's fine. But maybe like— maybe if you're running like the New York Times, someone from the New York Times like starred the repo as I was thinking about it. Like how could they use it? And they could do something like that. Make sure there are no typos anywhere on the site. And you just kind of every time there's a deploy or every 5 minutes or whatever, you could— so, there's that kind of service that could exist. Um, yeah, but I do think I, I want to think more broad, and maybe that's worth trying and building and designing. It would be fun to work on for sure. Um, but I think also like with AI, there's this idea that like costs will go to zero and monthly SaaS fees won't really be a thing anymore. So I was thinking about that as well, and I think maybe there's like an onboarding service. So you pl— you kind of like connect— imagine you connect your GitHub repo, like what are we doing here, right? We're like onboarding Iffy We're onboarding Helper, we're onboarding Flexcel, we're onboarding Gumroad. Uh, what if there was like a managed service where you just connect your GitHub repo and you kind of maybe specify, like, maybe there's some onboarding form of like kind of what kind of test you're looking to write, what features, functionalities critical, or maybe we just, you know, go to your website and like use AI to, to kind of suggest stuff. And then we just give you kind of a quote and we just say, hey, for like $600, we'll basically like write all your tests, shortest tests for you, which are pretty simple, right? It's basically, you know, updating your— it's just going through the README, right? Like updating your .gitignore, updating, you know, writing the shortest tests, putting them in the repo, creating the GitHub Action or like whatever, you know, workflow you have set up. With your CI, and then just charging a one-off fee and just being like, here it is, like it's green, it passed, like the tests work. And after that point, you have the tests, like you have— they're running locally, they're running in your CI. We don't need to be involved, and your engineers kind of can now basically use those tests to write new ones. The hard part is wrapping your head around this new project, how it works, is it good, is it bad, you know, how do you do before, after, you know, all these API testing within the end-to-end tests, all that stuff requires mail login, blah, blah, blah. It's like the sort of cost of— it's like a learning curve, right, that once you're on the other side, it's really trivial to write new tests, but there's like this pump that exists. And so I think that that could be a cool way to also just like test this business model of like that people like to opine on Twitter, like, oh, the future SaaS is dead, you know, um, or whatever. And, uh, well, if it is going to die at some point, then maybe Shortlist can be a good example of it, right? Like just all these QA tools exist and they're great. There's nothing— and, you know, there's a lot of value in paying, you know, a QA team. And like, I'm not saying that like that will go to zero or anything like that, but But I like the idea of offering a very different kind of service. So I think, yeah, those are kind of like two options. One is like a kind of a web UI around, you know, setting up tests that run regularly upon deploy. And the one is like a one-off onboarding that adds Fortis. To your repo with initial test suite. It's cost. Yeah, we could do both too. You know, it could start with this one and then there could be something like this. I generally think the web UI to me is not that compelling in terms of writing tests. And the reason is I just think Cursor is awesome. And like, if you're a, you know, if you're a developer, if you're an engineer, you're always going to prefer, I think, like having access to your terminal and having access to your keyboard and like all the things that you can do locally are just so, uh, it's hard to beat that, you know. Even I started like writing blog posts in Cursor because I can just do like appliqué, you know, like it's so much nicer to, to be able to do that instead of like Google Docs. I can't like have AI fix stuff, uh, not yet anyway. So my, yeah, my, my sense is that this one is probably more, more interesting and also easier to build, like it would be an easier shit. Um, and I like, yeah, I like the idea of just like charging a one-off kind of fee. And you know, you got like 100 people a month that, you know, do, you know, it's like $500 or something, and that's like pretty nice small, small business. Uh, the nice thing is we have Gumroad, so we're not like Shortest. The goal of Shortest is not, probably not to be like a multi-billion dollar company. But mostly to like help, help, you know, contribute, learn, participate in this AI movement. I think these other, other projects we have mostly help for Flexile and mostly help run Flexile actually are probably like what I'm most excited about. Flexile I think is the highest potential. Basically we want to, we want to like basically have every, every single company self-host their own cap table software. And right now they— if this is sort of like the end of SaaS kind of model where currently you pay Carta or Broadridge or all these HR platforms, like all this money, uh, because there isn't like a really world-class like open source suite to do this. Um, but if we can build that, then— and we will basically try to convince like Disney, like Disney should just run, uh, Flexile locally, like on-prem in their own, in their own stack. Uh, their lawyers should have access to it, their engineers should, should be able to ship to it, and they could— like, to me, that's way better than what they currently do, which is they pay like these companies like Broadridge, for example. Uh, this is like one of the market leaders in this industry. And now I know I'm getting off track from Shortest, but, uh, people can tune out. Uh, but there's this old school company, and, uh, they're so old school they put their ticker symbol in on their home page, which is kind of hilarious to me. Uh, but it's a pretty value— it's like $27 billion company or something like that. Um, they've done well. Uh, but they basically like sell Carta-like services to like the enterprise. Um, and I kind of want to build like an open source version of— I mean, you can see it's not the most elegant piece of software on planet Earth, Uh, but it, yeah, it allows you to do, you know, stuff like your IR, your, you know, shareholder management type stuff. Uh, this thing, I guess. Yeah, like this kind of stuff. You can text your share, your cap table, you know. Uh, stuff like what Carta does, what Flexile does, but it's not open source, so. Anyway, uh, cool. Anything else anyone wants to— Akash?

So I was actually thinking about the plugin thing, like, so all the problems that we're currently facing, like a parallel testing, or like all the, like, housekeeping you're doing with some supporting Windows or like TypeScript issues and all these things. I think it's taking like time away from like the core proposition of Shortcuts. So, oh, can I just share my screen a little bit?

Yeah, try it. It should work. Yeah. Feel free. This is, you're talking about the Playwright, kind of Playwright plugin?

Yep. So right now I just asked Claude, that, uh, how do you run test parallelly, right? So it has a lot of options, like probably a lot of these things are like optimized for production, and like it has a lot of, uh, like APIs specifically to support a lot of use cases that might come. So in that plugin, if you just use a workers:3, put a parallel: true, and if you run this, it just As you can see, it is running this one and this one like badly. So I think a lot of these things are like— so whether Shortest want to be another framework that does all these things, or if you just build on top of this, we could go into something like this where like we have Shortest plugin for Playwright and Cypress, Vitest, Appium. So I don't know much about Appium, but when I searched about mobile testing, I think this one came up. So all these existing things we provide, like, uh, plugins. And even if we— for our current thing, maybe we can wrap this into like a function like shortest that, uh, you know, works with Playwright. And, uh, and also like about traceability. So Playwright, I— when I searched, uh, Playwright Yeah, they had this, uh, thing, I don't know how to run that, but if we, uh, yeah, if we run this, you get a tracing like UI and stuff. Maybe we can extend that, but, uh, if you build it as a plugin, probably we get a lot of benefits that we don't have to, uh, like fight and build. So maybe saving some bandwidth and like focusing on like if we have like core package that just does, uh, the AI stuff properly, even like, uh, I mean, it could be a lot more easier to get adoption. And we also like building it. Yeah, testing framework is kind of like a very ambitious goal. Might be a lot of pitfalls and stuff. So yeah, yeah, that's, that's what I was Awesome.

Yeah, I mean, I agree with you. Uh, I think we are— I think Shortest is an ambitious project and will remain one. I think, like, I would love to have more stars in GitHub than Playwright, and we're 5% of the way there. Uh, but yeah, I do think the way to get there is to basically work with what people already are doing, right? And at the end of the day, Playwright is a Microsoft project, right? Like it's hard to, hard to compete with, with that level of scale and focus and resources and usage. And so yeah, I think, I think, I think we should pursue kind of what we're working on. I don't think anything is too audacious yet. Well, in parallel, I think it makes sense to, to kind of support, yeah, have a shortest plugin for Playwright that effectively you could have a Playwright test And it would be good for onboarding Shortest as well. If you already have like 300 Playwright tests, you're probably not going to be willing to delete them all and use Shortest. But maybe you could— there's maybe parts of your app that are quite annoying to write in Playwright tests, and you could convert those parts of those tests to shortest.ai, and you could just have that be sort of a way to, to, to kind of onboard a lot more people into Shortest. And then the, the goal would ideally, you know, for a new project, you could kind of just go straight to Shortest or, or so. Um, and so I do think it would be nice to support some of this stuff that, you know, like parallelization and prioritization, things like this. Um, especially because AI, you can kind of just like steal code, right? You can just be like, hey, just like take this feature and replicate it. But, uh, but yeah, I agree that we should have the player— the PlayerApp plugin is probably like a pretty good thing to do. Um, yeah, so yeah, if you want I can add that to the, uh, to the just the GitHub issues and flesh it out a little bit and then put a bounty on.

I, I was playing around with that, I was working on that, so yeah.

But yeah, I think, yeah, I think it's great. I think, I definitely think that even for like Gumroad or Flexile, like it's, we do have enough Playwright stuff that it may be a good way to test Shortus in that and without having to like write the whole test from scratch. Ideally you wouldn't have to specify the base URL and all that stuff. Like ideally it would just be able to work like with one line, you know, just like import and then boom, like just— yeah. But I think that makes a lot of sense. And I'm glad you called it out because I forgot to talk about it as part of— yeah, part of the sync. Yeah, I'll go through the discussions and there may be a— I'll just share my screen again, but I'll just share this window. But yeah, there's a few of these things in here. And let's see. Yeah, like shortest init, for example, is a good one. The plugin thing. Yeah, by the way, if anyone on this call wants to like upvote any of these and as a kind of a proxy for what maybe would be good to put into the, to the issues, then yeah, parallelizing stuff. Most, yeah, most stuff is already in. I think it's mostly the plugin parallelization. Maybe this init command is kind of a small one. But yeah, I guess we started working on Shortest even before computer use, so I don't know exactly what our approach was back then. I think it was— I think before it was just about building a tool like a web app that would allow you to write tests for a PR, but it didn't really— didn't really work.

Uh, yeah, you had that, uh, social media idea with Shortest that you shared a Figma design on Twitter or something.

Oh yeah, I also, yes, had like a, an idea for this which was, um, yeah, Shortest, uh, I bought the domain because I just love the, I just like the, the fact that it's short test and it has test in it. Um, and I was thinking for a couple years about what to build and one idea I had was, yeah, like a basically a Twitter competitor Because the original Twitter, you know, you could only have 140 characters. And that— I always loved that about it. It's— now it's— I mean, it's fine what you can do now. But mostly I don't like the fact that you can put images and videos. I think it just ends up dominating the feed. And if you just want text, like you have to go to Hacker News now or something. So, yeah, maybe we can get shorttext.com or something and then build build that. Um, let's see if that's available. Shorttext.com. Oh, I, I'm gonna see if I can buy this before I publish this video. All right, I'll catch you guys all in the future. See you on GitHub. And we'll do— I don't know if we'll do this weekly, it's probably too much, but we'll do it when it feels like we need to do one again. Um, we'll do one. And just ping me on, on Twitter or DM me on Twitter or or whatnot, and we can keep chatting. Thank you. Thanks everybody for tuning in. See you guys later.

Thank you. Bye guys.

Thank you. Bye.