Podcast

In a land of testers far and widely journey seeking answers, seeking skills, seeking a better way o the test Illinovation Testing Podcast Guiding testers with our nation awesomeness from ancient realms to modern days, they lead the way. Oh, the testicle of nature. Testing Podcast.

What is proactive observability and testing? Don't know. Well you're in for a treat because joining us today we have Anam here, a co founder of Reveal, a proactive observability platform. He also is an ex Uber AI employee so he knows a lot of stuff. I actually will caught my attention as he released the paper. Uber released a paper on an awesome project that he worked on that I think it's going to give you a lot of insight on what is proactive observability and testing and how it impacts your end to end testing. You don't want to miss it. Check it out. 16% of the world's population, that's 1.3 billion people, live with disabilities and have 6.9 trillion in disposable income. If your websites aren't accessible, you're missing out on this massive market. Beyond loss revenue, non compliance with ADA or EAA put your business at legal risk. But when you build with accessibility in mind, you enhance your UX drive sales and expand your customer user base. The Challenge Manual audits are slow and free. Tools often miss critical issues and lack scalability and developers struggle with prioritization. What do you do? No worries. BrowserStack solves this their all in one platform automates Ada and EAA compliance powered by their Spectra rule engine, which detects 40% more critical issues than legacy tools. Workflow analyzers scan multiple web pages in one go and entire user journeys without having to rerun scans for every page state. You can also validate accessibility using the screen readers such as nvda, voiceover and talkback on real devices with zero setup. The best part? Even if your team lacks expertise in accessibility testing, features like assisted tests let you audit websites and web apps by answering simple step by step prompts. Plug and Play. Automated accessibility checks into your workflow. No setup, no maintenance overhead. Expand your market, save time and build truly inclusive experiences. Learn more and try for yourself using the special link down below. Support the show Check it out. Yeah. Welcome to the Guild.

Thank you for having me, Joe.

Awesome to have you. I guess before we get into it, like I said, you sent me a link to something called Dragon Crawl, so I think it had something to do with challenges you had at Uber's end to end testing that you're working on and the process led you to developing this thing called Dragon Crawl. So what is Dragon Crawl?

Yeah, we can go over a quick overview. Dragon Crawl was my intern project back in 2022. Problem Uber was having was that they run in 200 different cities. Each city has slightly different UI, so testing it becomes impossible with like normal Playwright scripts or like Appium scripts, because every time there's a slight change, you also have to change like the code. So Dragon Crawl was using LLMs to kind of semantically understand the app and the test. So you'd have like a test like core Trip Flow, which is ordering a ride and picking up a ride. And it was testing that using LLMs so the UI changes wouldn't break the flow.

Did you do this as an intern? Did I hear that correctly?

Yeah, it was originally my intern project and then the team grew a bit and then I joined back full time as a machine learning engineer.

All right, so how did you come up with this idea then? Were you just messing around or did someone suggest it to you?

Yeah, it was actually. So like the original project was just to use like normal testing, but I was somewhat interested in LLMs at the time. It was a new field. So I kind of suggested it to my manager saying, hey, this LLM thing is kind of interesting, can we do this? And then my manager, who was really nice, he's like willing to take the risk of using LLMs. And then we ended up doing that, ended up being pretty successful, found 11p0 bugs. Basically just means people can't order a ride. It counts like 25 million saved. So it ended up being a pretty big project within Uber.

Thanks. How did you get there though? Like how did you achieve such an awesome result?

Yeah, I think it was a lot of help from like my manager and my mentor. One of my mentors, Juan Marcano, was like so helpful in it. And I think another thing was just like really bounding the scope. The original project was just for ordering a ride. That was the highest impact thing at Uber and then just building out like really quick proof of concepts to get it into production really quickly. I think that's what helped us make it really fast.

And so what was the initial issue? It was the tests were flaky and using LLMs you were then able to make the more reliable. Is that how it worked?

Yeah. So the problem that they were having was yeah, they were really flaky because like there would be pop ups or like they would have like a B tests where like super slight like variations in the ui. But they would break the test because like the selector ID would be different or like there would be a pop up in New York for some like regulation. And the tests were like super flaky, like 50% flakiness and like they just were getting no signal. And then like engineers, like they see no signal on a test, they just want to push to production. Like these tests are always red. Like I'm not trusting them. And so from there, like Dragon crawl, it had 99% reliability. So you got a lot more signal from the test. Like when a test failed, you know it failed. Like it was a true failure.

And you mentioned those tests were Clarite tests, Were they Selenium tests?

It was a combination. Most teams like it was pretty fragmented. At Uber, most teams were using Appium. There were some people using like Playwright would be for web. There was also like manual testing as well, just because like they were so flaky that you had to manually test.

Nice. So you talked about some LLMs and how they enhance the testing reliability. How did you find out, like, how did you tap in then to like observability part of it?

Yeah, so the observability part came after we had the idea actually talking to, I think you know him, Jason Arbind, he was telling us how like all these companies are using so much observability, they're spending so much money on observability. So we had the idea of connecting tests to traces. So like you run an end to end test and then you connect that to the traces it generates. So if there is a bug, you can see exactly what microservice the bug happened in. And that's what we're doing at Reveal. We're connecting the test to the traces to like have the visibility into where the bug happened. Because at Uber, even if the Dragon Crawl found a bug, you're supposed to still spending so much time finding exactly where the error is. Now you can kind of see where like what microservice it hit the failure at all.

Right, So I guess let's dive into that a little bit, what that looks like. So you created Dragon Crawl at Uber and then you founded Reveal. How is Reveal built upon the principles of Dragon Crawl? So it sounds like you then like how did you build it? How do you get the observability? How do you get the tie in now with your platform?

Yeah. So reveal uses LLMs to run end to end tests like written in natural language. Similar to how, like a similar principle to how it's done. In Dragon Crawl, but we connect it to the traces. So imagine you're Amazon, you're running a checkout flow. You want to test that end to end, like the end to end experience of how a human would go through it. And then it's connecting to the traces that are omitted. If a company is using over telemetry or something along the lines of that, and you can see exactly where the error would happen from that, you can. The triaging time becomes a lot faster.

For the folks that don't know what telemetry is. What does that mean? Does it mean like it actually shows you where the issue is in the function in a SQL statement. Like, what does that mean?

Yeah, telemetry is kind of like your, like datadog, your splunk. Generally code is like instrumented. So like let's say you have a microservice, you'll instrument that code and let's say you connect the front end. Like there's a front end microservice and there's a backend microservice. Your telemetry will allow you to see like the what is called span that's going from the front end to the back end. Basically just see every server that's hit.

Nice. And do you run these tests in production or is it in staging? Do you see developers using it early on to find issues quicker when they run that test on check in?

Yeah, there's a large variance. I think smaller startups gain more value from it in production just because it's easier to do. It's. You don't need like most startups don't have staging pipeline for our enterprise customers. They definitely get more value from running it in staging or shift it further left. Because the problem is if you get a bug in production, like once you're testing in production, you have that bug in production. If there is a bug and that just costs a lot, you lose a lot of trust. At uber. Like a P0 bug costs $2 million per every hour that is in production. So the costs are really high if a bug does get into production. So there's a large variance. Startups running it more to the right in production and then enterprises running it more to the left in their staging platforms or even local host.

All right, so if I have an existing test suite and I wanted to get this telemetry data using your platform, what do I have to do?

Yeah, so the steps is to like, if you're a large enterprise, you probably will have like open telemetry instrumented. If you're a Startup, it's instrumenting with OpenTelemetry, things like Sentry use OpenTelemetry. So if you're using something like Sentry, you actually already have OpenTelemetry set up and you can see the traces and like we can connect to the traces. Yeah, so that those are like the steps just connecting some kind of opentelemetry, something like Sentry, something like Datadog, to your code base and then we can capture those traces.

Nice. All right, so speaking of Jason, I actually had him on a webinar yesterday and he was talking about Agenic AI. It's like the new word I've been hearing the past few months or so and I think on your site you mentioned something about Agenic flows. Maybe I'm wrong. If that's true, what is Agenic Flows and how does it help make your UI more resistant to changes? Your test more resistant to UI changes?

Yeah, I think like what some people will do for testing with LLMs is they'll use the LLM to generate the test case and then you'll have like a playwright test generated delineation with Agentic is Agentic wouldn't generate it on the like beforehand. It would kind of test like a human would. So you observe the state. Like, let's say you're on a homepage, you get the state, you get the dom, parse it into like a LLM form and then you're sending that state alongside the task you want to do to an LLM and you're kind of doing it in a loop. So it's kind of like how we would test as humans, where it's like, we see this state and we have some tasks we want to do, let's say like login with X email. And then it's getting that state, doing the task and then running it through a loop rather than generating the test before and like running through it.

Nice. So how does this work? So is this just the observability piece? So you bring your own test to the platform and then the platform handles the gluing of the test to the telemetry data?

That's part of it. Our thought on this is like, there's a lot of observability platforms, There's a lot of testing platforms previously. Like a lot of the companies we speak with, they want to put an investment in testing, but it's just so flaky. Like if you're getting 50% reliability on your tests, you're just not going to run them, you're not going to have any trust in them. That's why a lot of these companies are using observability platforms, because they give you data and give you signal. Problem with observability platforms is that they're in production. Bugs cost a lot of money for users. Users lose trust in the product. So we're kind of unifying it and running these resilient tests, resilient agentic tests using LLMs and then connecting those to get the visibility of tracing and visibility of observability to know where the bug happened. So it's, it's a bit of both. That's why we call it like proactive rather than reactive observability, kind of finding the bugs before they happen in production.

Nice. So can you give an example of that? Of finding a bug in production?

Yeah. So let's say you're Uber. You release like a, a B test for Australia where there's now a pop up for ordering a ride and that breaks like you can't order a ride anymore. An example that Reveal would catch would be it would run that test in Australia for ordering a flow like kind of emulating how a humor would go and order a ride and it would see like this pop up caused me not to order a ride. And because it has the open telemetry traces set up, it would be able to see exactly like the a B test that caused this error. And now you as a developer, you catch the error before it goes to production and you save like the $2 million you would've lost in revenue if it had gone to production and just running these over and over again just to have confidence to be able to ship to production.

Right. So do they get alerted or the build just denies the check in?

Yeah, the typical workflow would be that you would push this like a B test or push this like merge request, pull request for a feature. It would run the test and then you would get alerted that yes, this build failed because this test failed and you should look back into it where the error is.

How long has this platform been around? Because one of the first times I heard of it, I don't know, how new is it? Do you have any existing customers and what are they saying about it?

Yeah, we've been working for around five months. We've been working with some startups as our early customers as well as some enterprises that we're starting to work with now.

Nice. So were you in Y Combinator as well?

Yeah, we're in the fall badge for yc. It just ends. We're just like wrapping up the Fall batch.

How does that work? Did you get a lot of any feedback from that? Does that actually help you? Like I'm just curious myself, like Y Combinator. What and what's the benefit of it?

Yeah, I think it's super helpful like especially the group partners we have, Andrew Micklis, the co founder of Pagerduty and Dalton Caldwell, being able to bounce ideas back with them is like super, super helpful. Especially like Andrew, he worked on a very similar, I guess similar area of product where he's selling to like large enterprises. Getting feedback on like how to do sales, how to develop all that stuff is like it's really invaluable to be honest.

Very nice. So you know there's a big, big push for AI, genetic, a genic AI. Do you see this as being a commodity soon and like where do you see the future of this? Do you think this is like a must have or do you think this is only certain companies would benefit from it?

I think right now only certain companies benefit from it that have fast dev cycles and have like high impact of failures. But as I guess the problem a lot of the companies we talk to are facing is they're using like cursor or like a cogen tool for all their generation and they don't really have anything validating that it's working. They're trying to ship a lot faster now and bugs are just getting into production. So it's. I think LLMs as a tool will be commoditized in the sense that there's only a couple major players but any like vertical integration, like vertical application. I don't see it being commoditized and I do see it being a must have as engineers want to shit faster using co generation tools and the speed of development becomes a lot faster.

So like what's your vision for Reveal then? Do you have like a master plan like you're going to take over this segment or this use case?

Yeah, I think the master plan is just really unifying testing and observability. A lot of these companies try to run tests, they fail because there's not high reliability, there's flakiness. Like the app is undeterministic because of some like ML, like personalization that they use and then they also use observability tools. Observability tools are good, but the bug is already in production. Bugs in production costs a lot. You have to triage it spending a lot of time. So why we call it proactive observability is it's really just observability platform that catches bugs before they hit production. And just like I guess the master vision is to combine the two companies only need to have one we test across Android, iOS and web. So write one test in Android and it'll work across iOS and web as well. I think companies are tired of having a bunch of fragmented services. A bunch of the companies we talk to have three observability platforms. They're tired of paying so much for them. And yeah, I think we can really unify it and give customers a better experience.

Nice. So if people have a lot of manual tests or a lot of manual processes, does this help at all if they want to transition to more of like a automated deployment type Transition?

Yeah, a hundred percent. A lot of the startups we work with, they were manually testing, like completely manual testing before they onboard it to reveal and now they just run it on like a per diff basis. Every time they commit they run the test and it's automated, their manual testing.

So I'm just always curious, you developed this with something in mind. You worked at Uber, but now it's in the wild. Do you see people using it in a way you didn't think about that made you think, huh? Maybe this is a use case that we may need to expand on or add more features to.

Yeah, I think some of the unexpected use cases were stuff like compliance. These financial, a lot of these financial companies have like strict compliance regarding their end to end experience. Example would be like a company like Intuit would need some compliance pop up to show up and that's like mandatory or else they get sued. End to end testing that on every diff is, is a use case that was unexpected that they couldn't do before because like they use the LLMs for kind of like the semantic reasoning behind it of the compliance. Kind of like how a human would see a compliance pop up. Another thing is like startups really want to test in production rather than staging. They just want us know that the end product works. They want to ship really fast and they just don't care about staging. Like they just want to see that the end product works.

How hard does the sell for this? It seemed like it'd be a no brainer but like do you find any type of resistance, like is it hard? Is it a hard push to get people to understand what this platform is doing and how it can help them?

I do think it's a harder sell in smaller companies for larger enterprises. This is like we find it to be like a hair on fire problem, like bugs are getting into production testing is really hard. They're paying a bunch of manual people to do it. For startups it's like less of a problem just because like if there's a bug, it's not the end of the world. They can just roll back and revert. In regards to like knowing what observability is, testing is I guess the connecting to tests, connecting your tests to tracing is a bit harder of a sell just because it's a new thing that not many other companies are doing. But yeah, for enterprises it's not too hard of a sell.

All right, so what does it work with? Because I know enterprises have a bunch of different technology. We mentioned Playwright, Appium, Selenium. Does it matter if it's mobile native? Like how does it work? What does it work with?

I guess yeah, it works across the board. We have created our own custom Playwright drivers and co own custom Android and iOS drivers for the device automation. So we work with like it's very easy to onboard it because it's essentially just taking the state of the app or the web and then doing an action on that. It's agnostic to any platform.

Nice. Okay, before we go, is there one piece of actionable advice you can give to someone to help them with their AI tracing kind of testing efforts, observability and what's the best way to find or contact you to learn more about Reveal?

Yeah, the best way to contact is at reveal AI or on LinkedIn and Amhera. And in terms of actual advice for AI tracing and observability, I would say less is more. We find that people try to test everything, testing just the most important parts of your apps. But making it bulletproof is way more valuable than testing everything and testing like the tail end just because that's where most of the value is going to be.

Thanks again for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com a539 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation and in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit test guild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.