Kevin Surace explains GENI's architecture for software QA: multiple private transformer models for images, text, and page locations, plus RAG and vector databases. Generic LLMs like ChatGPT produce unusable test scripts, so GENI needed to be faster and more accurate than both recorders and untuned models.
Four
About one million lines of code
10 to 100 times faster
Talk a little bit about we live.
In a world of AI right now. We're not that familiar, let's say, with the notion of digital twins and what you're building and creating. Talk a little bit about some of the rudimentary stuff for folks listening in on. Is this one LLM multiple LLMs is, you know, do I build GPTs? Is that what my digital twitch? I think there's a lot of AI vernacular. We're now kind of used to prompting. How does that vernacular map across into genie? Am I going to see some of that vernacular or is this.
Oh yeah, so, so, so ignoring the term LLM for a minute. The LLM is built on a transformer model, right? And transformers can be used for visuals, for audio, for language, for a variety of things, right? You can, you can train up a transformer and have it be really good at recognizing soda cans or whatever you want, right? And so we right now have four transformers in the inside the product. It's running on a private server. These are all private models. These are tuned very specific things and they are very different models. And they are again tuned to do. To recognize certain things that are important. So you've got images, you've got text, you've got locations on page, you've got, you've got the English language that's in the test case. You've got all these things that have to happen. And so we're using that set of models. We are of course using rag, for those who know what that is. We're using embedding, we are using a vector database. So all the things that you would think of around trans, the transformer world we are using. And one of those, you know, represents itself as an LLM, right? So these all work together with about a million lines of code. So the interesting thing is that people have said, can't I just go to ChatGPT and say write my test script from this test case? You can. And what you end up with is an absolute unrunnable, non runnable test script. The syntax may be correct, but all of the locators, it doesn't know what to do with. And it either puts junk locators in or, or wrong information or nothing there and says, pick out a loc. And so by the time you manually fix the hundred steps in that test script, you could have just gone to a recorder and recorded. In fact, the recorder was much faster than editing the, the garbage that came out of, you know, chatgpt or, or anything else. And part of the reason is they're not really tuned to do this. They have seen it and they can spit out something, but it. It has an equal chance of being right, as wrong as junk. And again, a recorder's already already fast. So let me give you an example. If I am writing code, I don't have a recorder, right? I'm literally writing code to create a website or to create something, or to create a game or to create an enterprise application. So the bar is here, I'm writing code, and now I have a copilot, and it makes me 50% more productive than this level. So now I'm not here. I'm up here. Pretty cool, right? Okay, here we are. In qa, I could write code, but you already have a what a recorder that's already up here. So I can't just come with a, you know, a co pilot and go from here to here. I'm still slower than a recorder by X. So we in Jeanne had to beat the recorder. So you can't just go to an LLM and say, spit this stuff out and get any help from it. You know, it helps you a little bit, but you're still way slower than a recorder. We had to be 10 to 100 times faster than the recorder. So we have had great tools in software QA that are already 10 times faster than writing code, and now we had to be 10 to 100 times faster than that. Right, if that. If that. If that makes sense.
Absolutely. That makes perfect sense.
