AI Testing Made Trustworthy using FizzBee

In a land of testers far and widely journey, seeking answers, seeking skills, seeking a better way. O the test Illumination Testing podcast guiding testers with automation awesomeness. From ancient realms to modern days, they lead the way. Oh, the test is on a major testing podcast.

Hey, do you want to know how to design reliable scalable distributed systems using model based testing? Well, if that's you, you're in for a treat because we have the creator of Fisbee in the house, JP who is an autonomous. If you don't know, FSB is an autonomous testing platform that validates software behavior and helps teams ship code with confidence. Before FSB, JP spent nearly 20 years building large scale distributed systems and AI applications at Google and other startups. He really knows this stuff, knows AI, knows reliability, model based testing. So this is something I think you need to learn more about. You don't want to miss it. Check it out. Hey jp, welcome to the guild.

Hi Joe. Yep, it's nice talking to you and test guild audience.

Awesome, great to have you. So I'm just curious to know, you know, Google's probably a really good gig. What drove you to do a startup?

No, actually after Google I was at a couple of other places as well. I wanted to explore and work in a smaller company. I always wanted to start my own company as well. So just moved from bigger company to slightly smaller to even smaller and then decided maybe I should jump in. That's about it. When I was working in my previous job I noticed that was need for a better way to design a software or better way to prove our design is good. That's where I got into formal methods and then back then the formal methods tools were too complicated that very few people were using. And two years ago with the chat, GPT and Copilot started and pretty much now like a 1020 coding agents also there. And now I feel like we need. It's not. Coding is no not much of a bottleneck anymore. It is actually what instructions we give to the agent and the quality of that. The instructions that we give to the agent becomes more important and being able to validate the output of the agents is crucial as well. So that's where I thought okay, this would be a valuable tool that I can build and that's where the a office be started.

Nice. So do you see this as a problem everyone's facing now? You know, using Claude or Cursor or anything, you know when they're prompting they're able to generate like I said, a bunch of code. But how do you know it's doing what you want it to do or if it's good code. Is this the like the main pain point that you're aiming for?

Yes. Again based on the feed or talking to people that keeps coming up on one side you will see like okay, I built an entire app with wipe coding, but they're not production apps. Yeah, it will work for most cases, but the moment you see talk to people in industry they would be like okay, it generates a lot of code, it is good for first two to three months and after that is when things start to get messy because every edit is now becoming more complex and you don't know whether the test is wrong or the code is wrong. And with the many self healing ones, even for testing, it's easier for some trivial ones where you can say that okay, yeah, fix the test whenever my code changes. But the thing is that in many times we don't even know whether the code is correct or the test is correct when a test fails. So self healing testing solutions don't really solve that problem as well. So and whether it is for coding or even for other cases, I do see people keep saying like okay, one fact checking hallucinations are real with LLMs and even in coding that is again, it keeps coming back. Often.

The tools are now getting much better at generating the code.

But then we are still not, we cannot be fully confident. It's not like a compiler where, okay, if it compiles, you never have to look at the output of the compiled code, right? Whether it is a higher level ones like TypeScript to JavaScript or low level ones like a C compiler, you don't look at the generated output. Whereas with LLM generated code you're supposed to review to ensure it is correct. But the speed at which the LLMs can generate the code, we cannot keep up reviewing it. And we humans have a very short attention span. It's also a boring work. People like to write code, not to review code. So it becomes even harder and it gets worse for tests. No one likes to review tests as well. Even in the olden days when people were writing tests manually writing we can force them to write, but reviewing it, just skim through the test, right? So yeah, that's always been a problem. So I believe with the high quality tests generated now, we can be very confident that the output generated by the LLMs are valid or the code generated is correct. And if it is correct, then we will be able to ship faster.

Love it, love it. You know I first learned about FIZB on my New show, I came across your LinkedIn profile and you announcing it and I think you said something in there, that development is incremental, but testing is cumulative. I'm just curious to know, can you explain that? Does that make sense? Like why is testing cumulative? Are you saying that like the more and more code that's being generated you just have to create more and more tests to validate it?

That's right. Basically every time you add a new feature, the each feature is incremental. Right? You're just adding one feature, but you're testing not just that particular feature, that particular code. You need to test how that works or interacts with all the previous features and the rest of the system. So it's not testing only that part in isolation. So the whole suit of automated testing is for that purpose anyway where there is no regression introduced kind of a thing. But yeah, that's where the incrementality comes. And that is also why as a startup when we don't have many features, we can just keep launching. As a company gets bigger, we are, we have lots of features and because each customer has a different set of high priority features for them and you have a whole bunch of things, things get messy. What happens when each of them interact with each other? So that's why I say testing is cumulative.

Absolutely. So if we're able to generate code for software and automation test is basically a framework that's software. Why can't we just rely on like GitHub, Copilot or something to generate the test for us? Why is that not enough?

That's a good point. But like I said basically so far that's one solution that most people are using anyway. We'll start with the test. Either it is for because it's just another code. The issue is still the test themselves are example based tests. Like instead of you telling try with this example, LLMs are going to create whole bunch of examples for you to test. The code itself would be written for those examples. The difficult part is again the reviewing. We still cannot be confident that it tested every scenario or any estimate on how good the quality of the test. And with a bunch of examples the issue comes like whenever there is a small feature change, even a small API change now like a hundred different tests would be updated. And when I get a code review with hundreds of tests updated and now do I have the patience or even the interest to review through all of them is non trivial. So again there are some people trying out like things like how two different LLMs want to create code One to test kind of a thing, but it's still, it's the same similar problem. Whereas the solution that would work for these takeoff problems are like property based tests. As an example, in a property based, you're just defining the property, you're not listing down the examples. If you have used hypothesis or similar quick check and similar tools in other languages, you just say that, okay, for example, I've seen a square root problem, okay, if your square root of any integer would be or any number, real valued number or positive real value number, the result if you multiply it should be the same. So now you can generate any number of real valued numbers randomly and then say that okay, it's correct. So which means the amount of test written is trivial, trivially small and you don't need multiple examples. It's a similar concept again. Another type of such tests are fuzzing, where you don't list down an example, it's just randomly generated. Now you can be confident that your system won't crash on a random set of bytes, right? So those are the type of examples we are looking for.

Those are the type of tests I believe would be more valuable going forward and going back to property best tests. Property based tests are good for stateless systems and typically suitable for unit tests and small individual apieces. But if your system itself is stateful, like a naive example is a key value store, okay, you're inserting, deleting and update and all can happen.

Continuously. But we can also test with multiple ordering of these events without having to list them. All we need to say is okay, these are the actions that can happen. You can put, you can list, you can delete and just a crude API. If these are the APIs, the test can generate like maybe thousands of sequences randomly generated with randomly generated out and validate whether the state transitions are valid. So that's the thing that FSB is trying to do for this. In addition to having your code, we need a way to know whether the state transitions are valid or not. That is where the model comes. We define like some trivial models to indicate how the system is expected to behave. And that's usually very concise. And a mapping to indicate this state in a model map is mapped to this particular API in your implementation.

All right, so for older people like me, when they hear model based testing, it was very much a manual approach and it was based on a lot of assumptions, I guess. First, for people that don't know what is model based testing and like how does your tool handle model based testing? From a more modern approach than maybe when I started.

I've been looking at it only in the recent times on the latest tools. So the thing is that the concept of model based testing is obviously not new. It has been there for a long time. And basically we need to create a simplified model or in memory model, for example a key value store. Maybe we may be able to have a trivialized model using a hashmap or something like that. And then we compare whether the behavior matches or not between this. Like if you are put on the real system and put on this trivial model works correctly or not. Or if it is more state complex system like an email application. Now real email application would be much more complicated, but in a trivial model you put everything into a small list of list and the in memory data structures. When you say send, it just puts it into an outbox which is again going to be another list or something like that and change the status to sent and so on. And basically you can just put them into an in memory structure. So that is a high level concept of model based testing. You have a rough model and we just validate whether your real implementation is similar to your model or how well it maps to your actual model. The issue is representing that model itself used to be hard. So different people have been trying out different approaches on how to define the model. Many tools used the same language itself to represent the model. For example, I think a hypothesis of Python also in Python also supports model based testing. They call it rules based testing, which is roughly the an extension of property based testing for stateful applications, which is quite similar to model based testing. The model should be trivial and we should be able to quickly extract it from a design document itself. And the model should be independently validatable. That's what I was looking at. Because if I'm going to design some system and the design itself is going to be more and more valuable, then the design itself should become testable. Am I designing my product correctly? That's where the formal methods come in. But again formal methods are another that has been there for like 20, 30 years. Very few people use and everyone says that we should use but too complicated kind of a thing. That's because that you also used to be too mathematical. So that's why coming from both these, I felt having a Python like language to express your model would be more accessible to most engineers in the industry. And with the LLMs coming in we would be able to generate that model also. But the model being something that unlike traditional formal methods, which are too complicated for people to review. This is both concise, so we it is within our attention span and then it can be easy to understand and readable. And unlike other systems where pure plain text which cannot be validated, we can actually validate by using the formal methods techniques. So once you validate that, basically you can even set various assertions including the safety properties and the liveness properties. Like some good things should eventually happen when you send them. Even if some replicas, crashes or anything, the system should eventually become consistent. Those kind of eventual are called temporal properties or liveness properties that also can be validated at the design phase itself. These are very hard to test it in the implementation, so we can actually validate those in the design phase itself. So pretty much FSB started off there as a formal methods and then we wanted a way. I want, I was looking for a way to test whether our implementation matches the design.

Awesome. All right, so say you have someone vibe coding, not necessarily a hardcore coder. What's the next step then? What do they have to give Fizbee then in order to understand what they vibe coded when they may not even know what they coded? I don't know if that makes sense. Does it plug in to like cursor or something? You just say hey fisb, like an MCP server. Create a model based on whatever I created.

Yeah, actually right now I'm working on the MCP and agents for generating the FISB spec. But I have a very basic, you know, copilot has instructions. So I can give that instructions to copilot instructions. And it is able to generate a valid FISB spec, but it is not fully trained use. We still have to review it and whether the model is correct or not. Thing is that here in Fisbee you can actually also it would automatically generate interactive visualizations like how in a whiteboard if you are explaining a design. So when you say this is a like a key value store implementation, if you put a design, it would actually generate an interactive visualizer where you can actually click through the button to say okay, put means this is how the data would be stored. Now click another button and say okay, maybe if there it is partitioned or sharded by key, this is how it would be stored kind of a thing. You can actually visualize them, it automatically generates them. So that is another form of a review where you can not just review the spec, but now you can also review from the visualization to understand whether the matches are expected behavior. So that's the one part we still have to be a little Bit more manual work is needed and rest of the things are pretty trivially automated because basically once you have the model and if you have the code, it can extract the API from the code and say that okay, this is an action and this roughly maps to this particular action or API in your call. So that is so trivial that yeah, that is done automatically. That said, I'm actually looking for another way where you can have that specific and give that to the copilot to say that okay, this is my design implemented. Now the design itself is much more precise and validated compared to a free form text. So since because of this giving a precise instruction to means that it can actually refer without any ambiguity, it can generate a code more correctly. But it would still be wrong. But since we also have the tests, it can go on a loop repeatedly until the test pass. So that way your implementation at the first pass itself it will be correct. So that is the thing that I'm going after.

Very cool. So you're testing kind of non deterministic code nowadays with AI, how do you make your test deterministic? How do you make your test repeatable or stable? Even when the code is constantly changing, you don't know what it's going to change. Is it because it knows the model and it says you made this change? I know the API, it affects these components based on the model I created ahead of time.

Yeah, exactly. So as long as the behavior did not change from your design or the model that you have specified, it can heal itself if the interface doesn't change. Obviously I don't need to change anything because every sequence is going to be tested from the model. Okay, whatever is valid in the model, it's going to be tested eventually in your code. But if your behavior itself changes, then this would be a problem. Either your code might be is more correct but your model is not up to date or vice versa. Now is it a regression or is it a new feature that you added which was not modeled? We cannot make a decision. We can say that hey, this is there's a mismatch. Now it's up to the developer to decide whether the to update the model or update fix the code.

So how many more bugs does this catch? Do you have any case studies from your own experience how many bugs this catches as opposed to not using something like fsb?

Actually there's an pretty new Right now I don't have any full on case study. The autonomous testing was launched only two weeks ago. Right. So it's pretty new. But for the design Verification it was independently used before as well as part of the formal methods. As a formal methods tool, without even testing for the code, we have found many concurrency issues or when what happens when. When basically two requests go in concurrently and there is a specific sequence of thread interleavings which can cause an issue. So people have tried out like people from Confluent and Shopify and a few other companies also have tried this out before and that technique is not new even with other tools like TLA plus like Amazon is big proponent of formal methods for their cloud infrastructure in MongoDB and again confluent and pretty much every cloud infrastructure company have tried out formal methods to verify their design. It's just that mapping from the design to code, whether code is correct or not used to be done ad hoc. Many, very few people do it or people do it as a research project and never made it into a product because it's used to be too complicated.

Yeah, so one of the things that did catch my eye as well is when you announced this. Like you said, it's brand spanking new. You mentioned it's able to find concurrency issues. How's it able to do it? I know a lot of times people need to use like kind of like intrusive kind of tracing and maybe custom hooks in the code. How does FSB get around that to find the currency issues?

Okay, so this is again by comparing with the model. So the thing is we need to run them run multiple actions concurrently. That's the first one. And again so traditionally in most testing we don't do concurrency. We don't check for concurrency because it introduces flakiness. A naive example is that if you do a put and a clear happen concurrently, we don't know what to even set as an expectation. Because of that, many times people don't even write those tests. So here we know that based on the timestamps of when these two different threads happen, how they are overlapping, I can check whether it could have happened, one came first, which one came first and versus the other. If it is not overlapping, we know that the one that completed first should have happened first. But if it is overlapping, I have to check for both the cases. It's called linearizability checking. But most real databases are not linearizable anyway. And still with fsb, with some.

Unique novel techniques we can test for non linearizable system also like eventually consistent systems using Fizz B. So but the high level idea is trying out all combinations based on the timestamps to see it. If it matches the design. So again, effectively it is not checking whether the your system is linearizable but it is checking whether the your system matches the design. So having the design is crucial here.

Very cool. All right, so if someone's listening to this, they're like all right, this sounds pretty cool. How could someone get started with fsb? Is it open source, is it paid? Like someone wants to try it out, what do they need to do?

The fsb, the formal methods tool tooling is completely open source that they can try it out right away. Just brew install fisb it has a binaries works both in Mac and Linux. I haven't tried it out on Windows, I never have access to it. It should work and the tutorials are pretty clean. Most people should get start would be able to start on formal modeling within a couple of hours of our training. I also have like the FSB custom instructions that you can put it in copilot that would make it like super easy to build the model. I'm just coming up with an agent that could deep in the long run I want to accept a design document and create a model out of it. Right now that part is not done. So so for the testing part again, right now it's again free, but it is not open source. So again brew install FSB MBT and you can just install it and try it out locally. Right now I don't have any paid service or long run. Yes I wanted to instead of developers running it on their machine, I just wanted to be on hosted service so that they can put it in their.

GitHub actions or whatever and every time they merge it should automatically trigger and test. So that's a long term but right now it's manual. The users have to install it locally and not test it.

So do you see this as a mandatory technique going forward as we go in more and more AI generated code? I would think this would be more and more mandatory this type of approach. I assume you'd think the same thing because you created this. But is that. Are there any other methods or things you see as needed as we go into 2026 with the AI type code?

Yeah, like a noted I did. I do believe this will be way more crucial. And this all comes under the broad umbrella of autonomous testing. Again, different people say when it comes to again autonomous testing it means different things. Like sometimes it's about execution, sometimes it's about test generation. Where it is the they're automating the mechanics or the test cases are still example based. In my case it's like even that is automated. So you never have to even review whether the test cases are complete or not. Those type of things I believe is going to be way more valuable. Think about it like let's say you have a website and you give these are the actions that can happen on that website. Tool should be able to say okay, they randomly click through various buttons and fill in with random data and say whether it matches your expectation. So that is going to be more valuable than any other tool which says that okay, where even a free form text if you say okay, go click this button and then this button and see if this happened. That is not really autonomous in my opinion because they are automating the mechanics like how to click a button or what code to write to click that button. But I don't even want to give the sequence of steps. I just need to say that these are all the actions that are possible and this can happen. So that I believe is going to be more robust and valuable solution in the long run. On one other technique, again I think I mentioned before, property based tests are going to be crucial as well. And fuzz testing and those kind of things where you don't have to list down exact test cases to test that would be I definitely believe would be more valuable.

So if someone's job is a software tester right now, just like a coder, are their jobs in jeopardy or are they just going to morph? They're going to morph where they need to do more property based buzz based testing. Is it going to be more high level just making sure the AI is doing what you know it's doing from your experience. So like where does a human come into play here in this world?

Very rarely automation. I've taken jobs even though people have been saying every company that says that like a is going to take up your job, they are the ones hiring more engineers. Other companies are not hiring. It's those okay, so I don't see it to be going away. It is just me going make them more productive in doing things which humans are good at. Because finally we are going to sell to humans, okay? So they understand what is the expected behavior and they can do the model properly. Hey, this is how it should happen. So instead of doing the mechanic of individual things. And if you look at again since you we talked about automation, you know like even 30, 40 years ago I remember people talking about like how a basic language would means that anyone can write code. So we don't need after engineers. But then no, we just the software that we build became more complex, that we need even more specialized skills. So I believe that's going to happen. And I don't know if I should give you example from manufacturing jobs are still there. It's like it's huge. So automation rarely takes up a job.

Love it. All right jp, before we go, is there one piece of actual advice you can give to someone to help them with their AI testing efforts and then what's the best way to find or contact you?

To contact me, LinkedIn is one easy option. We have a discord channel, you can email me directly jpisbee IO and from advice perspective.

Right now I would be biased in saying anything. So I believe testing is going to be way more valuable and instead of just creating more more and more example based tests which you don't even review, look for better techniques. Like I believe property based testing and model based testing are going to be more valuable. Again, I don't want to say that anything bad about property based testing because.

I believe model based testing is an extension of property based testing. So I always say that, okay, they both would be extremely valuable irrespective of what tool you use. Whether you use FSB or other model based testing tools, I believe this is going to be extremely useful.

Awesome. You can find links to all those awesomeness for FYSB and everything else down below. Thanks again for your automation awesomeness. The links to MD Value we covered in this episode and head on over to testguild.com a565 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation and so software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges. We're excited to collaborate. Visit test guild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and liars the bards began their song A tune of knowledge A melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.