Shift-Right in Performance Testing with Jonathan Wright

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Calantonio. Here it is.

Here it is. Here it is.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance Podcast. Today we'll be talking with Jonathan Wright all about the shift right paradigm in the performance testing lifecycle. We'll also probably touch on a bunch of new developments in performance testing and site reliability. Jonathan is the CTO of Digital Assured and President of Vivint Worldwide. He's also known as a global strategic thought leader specializing in emerging technologies, innovation and intelligent automation with nearly 20 years of experience. Also a little known fact, he is an automation cyborg. So get ready to discover the latest in all things performance testing related.

Face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work. Even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to Test Guild.

And.

Click on the link to learn more.

Hey Jonathan, welcome to the Guild.

Thanks for having me, Joe. It's great to be on the Guild and I'm looking forward to talking today around the shifting right paradigm with the cognitive engineering.

Awesome. Before we get into it, can you just tell us a little bit more about yourself?

Absolutely. Even though you did the great introduction there, I've been sat in the performance landscapes in the performance engineering world since the early 90s. I started off with Astral load Test and then moved up to the Loadrunner tools then. Now I'm just an advocate for all types of performance engineering tools and modern approaches.

Very cool. So you didn't mention cognitive learning I believe. Let's just start up right there. How does that relate to performance testing in the moderns times?

Absolutely. So cognitive engineering is kind of using heuristics from things like performance engineering, site reliability engineering and chaos engineering. So it's taking those kind of disciplines and applying them into the development of cognitive platforms. So as far as that's concerned, it's platforms with artificial intelligence, machine learning and deep learning.

Very cool. So most people that are familiar with automation testing are probably a lot of tools are coming out for with AI, so called AI machine learning type functionality. It's a new trend. So you see this also happening in Performance testing as well.

Absolutely. But I think there's kind of this, I call it the cognitive engineering trifecta. So this kind of what you're seeing in the industry is this intelligent operation tools. So AIOps, you may have, may have heard of AIOps and this is this, these tools that are able to a bit like application performance management tools, but they've started to introduce.

Insight driven from they're building their own algorithms and the idea there is to identify pinpoint failure analysis and diagnosis engines to identify potentially when something's going to break before it breaks.

So obviously usually what I've been seeing happen with the automation technology is all this is vendor based where people have been used to go vendor and now they went open source and now it seems all this newer functionality is coming in a vendor type solution. Do you see this the same type of thing with performance testing, then that vendors will start introducing this, then over time more open source will be able to integrate into their solutions as well.

I think it's a difficult one based on two things. If you look at the different levels or layers of performance you've got things like the, the infrastructure at the bottom end, then you've got your applications that you're building, your middleware and this before you get up to those business transactions. So as far as the automation of using a user interface, that's kind of the very last point when it comes to the performance. So partly this shift, right, this move towards focusing on operations and I guess this was the move towards DevOps we're looking at an ops dev approach. So coming back from operations towards development and looking at how we observe using monitoring tools and how we act with automation tools to generate or synthetically create realistic loads and volumetric profiles that can be measured. So you can then start proving a hypothesis in that right hand side. So I'm expecting the system to do this and this is when the ability to say for instance, deal with systemic failure. So if a payment gateway goes down, predicting that and then being able to ensure that doesn't happen before the site decides to cascade fall down, those are those new tools that are coming through in the operational side. So unfortunately as we know in IT operations this has been managed by a large vendor scale mainly because they were providing the infrastructure as well. So you find that people like Microsoft, Google, AWS, your micro focus and IBMs of the world have played in the IT operations for such a long time and I know this is sometimes referred to as kind of heritage or the ecosystem of heritage products that have been deployed already but that's the focus really for performance. We have to learn from the right hand side, we have to learn from what's actually happening in the real world and model that based on real user journeys. And I think this is where we're starting to see a number of different vendors that are coming from the software development lifecycle. So let's call it like the automation landscape. Those guys who've been creating automation performance tools, marrying up with the APM tools of the application performance management tools. And yeah, hopefully we're going to start getting some open standards around, sharing data between those and that will lead to hopefully some open source platforms as well. But there is ones already that are out there, things like Graftana. I've got the dashboard in front of me at the moment which I'm looking at, which is taking in all those logs. There is those open source platforms already. It's just depending on the level of.

How far you want to go down the rabbit hole to be able to track transactions down to a database level, to an individual node on a Cube instance or a Docker container. It all depends on what level you want to go down to.

Before we dive down a little bit deeper, I guess we should explain what is shift, right?

So partly what I talked about on my TED talk was around cognitive learning and the focus of this was looking at the learning aspect. So if you think of, if you break it down into kind of four, three quadrants, the first quadrant being the thinking aspect. So this is really where you're, you're modeling something, you're saying, well actually we expect the behavior of a system to work in a particular way. And then what happens is we are very used to the software development lifecycle which is the creating aspect. So we create the software that maps to the requirements or the model of what the business are wanting to do. And then the last segment is the, is the learn aspect. And this is where we're learning from the operational tools that the behaviors that we expected and we predicted are correct. Now this is usually quite a linear process. So kind of design, build, test and then a product comes out and it gets deployed into production. My argument was really this is actually a continuous evolution. So evolution over revolution. It's not one big big bang. You're learning from what's happening in the real world and you're feeding that back between the learn and think. And these are where these tools are starting to come from is the feedback loop from the right hand side to validate what's going on on the left hand side in the software development life cycle. So that's really the big, the big one is around DevOps was around blurring the lines between development and operations and kind of ops dev is blurring the lines between the operational tool stacks and what actually gets built and why it's getting built before it actually gets built.

Cool. So it's like it used to be production, that would be you release the software, that you're done with it. Now it sounds like first level is monitoring and then once you do the monitoring, you somehow using cognitive learning to determine how it actually is behaving in the wild, taking that feedback and feeding it back into the system from the beginning, which then informs you of what you should be developing from the start. Is that how it works?

Absolutely. I'll give you the example for the stuff that we've been doing for Brexit because it's something everyone's quite familiar with. We've been doing performance engineering to prove that we're going to be able to deal with the increases of load on the UK government infrastructure. So obviously, as you'd expect, that infrastructure's on quite old, it's been running there for many years and it does everything from your taxes to your patents to your, you know, to your any, any kind of internal government request. So partly what we're doing there is we're saying, okay, we know what the behaviors are of the current state in a pre Brexit world, but nobody really knows what a post Brexit world is really going to look like. So partly what we had to do is design the end system to be ready for a no deal situation where there was no deal and we had a hard exit or the one where we did have a deal and we had an 18 month period to make and change all the rules that are set by this, by the European Union. So partly what we're doing there is we're saying we're preparing the system ready for two potentially different outcomes. And then we're using the existing profile of how people interact with the system and do their daily taxes and then we're going to change the flows so that we can see what would happen in a pre Brexit landscape of both a deal and no deal situation. So I think the easy way for those guys that are familiar with chaos engineering, so Netflix and a few organizations use chaos engineering. The idea with chaos engineering is you build a hypothesis based on a current state of behaviors. So that's what today looks like in the UK government. So taking all those flows, taking all those patterns of user behavior. And then what we do is we introduce changes to prove that the system has to be able to deal with things in a different way. So this could be, you know, lots of new tax codes and a huge amount of additional people coming in to check what their new tax code is after the Brexit happen. So what we're doing is we're running those experiments, so we're automating those experiments so we can continuously validate them. So that's kind of the idea with Chaos Engineering is you can introduce variables or changes.

In an environment that is safe. So this is what we've done with a lot of what we class as this kind of.

Cognitive learning aspect is we model real user behavior from the right hand side in the application performance management tools, we take it over to left and we run them as functional scripts, as non functional scripts to create the blend of user behavior in an environment that you've made those changes to. Either they could be software changes, they could be infrastructure changes. Whatever it may be, you, you're proving the hypothesis that you've created, but you're learning from the intelligent operation side of things.

So Jonathan, how do you prioritize what to focus in on? It seems like there's a lot of data when you're in production. There's a lot of different techniques you can do to try to make your site more reliable, more performant. Any tips on how you would prioritize, like what activities someone would do to get started as they're shifting, Right.

So really with shifting, shifting, right, you're observing. So you have to work out what, what activities or behaviors you're observing in the business transaction landscape. What you're looking at is user behavior and that could be on a front end, for instance. That's great because that covers your UI layer. But then when it comes down to what is your actual application doing? You might have ERP systems or CRM systems that are doing a whole stack of workload. What you need to be able to do is you need to be able to do workload modeling. You're modeling the workload coming from different systems. You're modeling the business transactions that are coming through so people using the system, those digital interactions. Then you potentially have a whole stack of underlining, logging and historical data and metrics which you can also look at. So what we call, what's been known in the industry is Performance Application Lifecycle. So it PAL for short was the ability to take and quickly create very simple.

Kind of scenarios based on what behavior is happening in production right now. So this shift right approach actually can be really easy and really effective because all you need to do is literally turn on your monitoring for a 24 hour period, look at the behaviors, what are happening, and then you can pull that left and model that quite quickly and easily. I think for somebody who's brand new to this and they want to be able to do something incredibly quickly, you could look at just creating something like using Fiddler or a proxy to intercept all the traffic that's going into a particular system. So you know, if you've got a tool like Burp Suite, which is you can use port swagger to sit in front of any endpoint. So that could be an endpoint for a ui. So it could be an application server, it could be an endpoint for a ERP CRM system. What you do is you capture all of the traffic for, for a full day and then what you can do is you can literally take that and rerun that in a different. Against a different environment. Yes. As you'd expect, things like sessions will change. And this is where I think when you talked about the vendor, that.

SmartBear, where they use AI algorithms to actually go through there and do all the correlations and get rid of all the session IDs and make them dynamic and make it so you can use that capture, which you've done on the right hand side, quickly to the left.

That's the power really of performance engineering, is you can prove a high point hypothesis incredibly quickly. The question really is around how do you do the modeling to prove that the user behavior is how you're expecting it. And that's really where the real science behind it comes in and it is where the engineering effort really steps in, is to model the workload, model the volumetrics, understand the user journeys, and then create a realistic load profile that can be run against the system to prove that that system will work in those under those particular predefined conditions.

We've been doing performance testing 20 years probably, and that used to be the biggest issue back in the day for me is doing a performance testing in our staging environments. And then when you go to performance, it's totally different. So you keep talking about modeling, and I hear people not complaining, but having issues with modeling as well, is that their modeling actually never turns out to be exactly like it is in production. So like you give an example how you can make sure your modeling is. Is it using this cognitive learning to make sure that you have more accurate models that you're creating or.

Yes, so. So the example Which I'll get, I can quickly give you as a bit of a flow. So all these tools are kind of interchangeable. So. But if you had a application performance management tool on the right hand side, the customer I've just spoken to this morning are looking at implementing application dynamics. But there's Dynatrace, there's all these different other platforms that you can get from an apm. You've got Splunk.

You'Ve got the Microsoft Tools, whatever APM tool you've got. What that's looking at is it's looking at every single, it's monitoring every single transaction that goes through the system. So what you can do is you can take those transactions, you can export them out. I use a vendor code called Pro Bit, which takes out all the sessions out of any type of APM tool and creates a model. I call it a semantic map. But that model is just a bit like a standard business process.

Model which you would be able to use. You could open up in Visio or you could create in PowerPoint that shows the map of what the activities were and the data associated with each one of those activities. Of course, as things change, as things get released into production, you could just refresh that from live digital sessions and then get a new model of behavior, of what's happening so you don't have to keep on generating the model. I give you the easy example. When I did this, when I was based out in Silicon Valley and I was doing a presentation to Diego from Forrester around this, what we had is we had an APM tool on the right hand side and we just launched an application, a brand new application through the pipeline and it was sat on aws and we literally said to him, okay, go off and use this application in any way you want. And so he said, okay, fine. He was doing that and we were carrying on demonstrating some of these cognitive testing capabilities that we're building. And then we came back and said, have you gone through the system? He said, yeah, I've been through it. I played around with lots of different settings. We said, okay, perfect. We opened up the APM tool and we could obviously see his behavior. You could see his request and responses for each activity against each system. We exported those. In that case, it's just a JSON payload, but we exported those and used a tool to model those on the left hand side. Quickly we created a simple model which showed the journey he'd gone through the application and also the data he'd used, the data he'd signed up, the Data he changed and logged back in through the system and made an order for. And then we said, okay, we'll now run that. And obviously that's when your Selenium comes in or your Source Labs, your ability to run it into, into a cloud infrastructure with lots of different data, lots of different browsers or mobile devices. But we re ran that scenario and he went well, that's amazing. How can you literally take any blend of applications on any scenarios that have been run in the real world and use that as a model without any kind of overhead? And we say, well this is just, these are just very simple flows through the system. This is a great way to do model based performance. We've talked about model based testing before. It's the same approach. But model based performance really looks at how you scale this up as far as data engineering is required for large amounts of test data to be able to generate different types of quality of data through the system with lots of different permutations. Then there's obviously lots of test data management tools out there that allow you to synthetically generate large amounts of data as data is one of those big challenges when it comes to performance is how do you create thousands of different flows through an application or data sets that are needed for that performance Load test Because I've just been running one this morning, which I did half a million.

Items, I posted half a million items into, into the system and that was going through a quite a modern, what we call AI as a service platform. So this was a whole stack of GraphQL coming from the front end. It was going into the back end for Kafka for doing streaming. So I obviously you're looking at when the producer produces the data onto the message and when the consumer consumes it and does something with it, whether that's right into the, into the knowledge graph. So Neo4j platform, or it's writing the information into a MongoDB or it's sending the data off in some REST API. All of these different activities will need to have a load profile associated with them and a data set so that it can actually generate the data throughout that entire ecosystem. So it's incredibly complex and it's really kind of looking under the hood. But a model becomes the first step because you can model anything. You can model the data sets, you can model the UI activities, you can model messages or database or API calls. So it really opens up the opportunity. And model based testing is a great way to start these kind of activities.

Who's generating these scripts then? Is it Taken the input from your production and then generating the scripts for you as well. Because back in the day you had to create your script. So is it generating the scripts you'll need and also the data needed automatically or how much interaction is there between the tester and the actual output of what is needed to create that performance test based on information that you gathered in production?

It all depends on where you want to come. So the example with Brexit for instance, that was real user journeys. We literally had a week's worth of user journeys and behaviors through system. And we modeled that that entire week's worth of user journey. So it created all the scripts automatically. It created all the data sets automatically. Of course, from a GDPR perspective, we had to go through there and ensure that there wasn't any sensitive data and it was masked or if we couldn't mask it, then we had to synthetically generate similar ones.

Similar data sets to be able to go through the same similar scenarios that were going. So obfuscate the data if we couldn't find and replace it. That was fully production. All captured through your standard APM stack. Which is the power of the operational side is that we're very used to the development life cycle, the software development lifecycle. But the real value of DevOps was the operational tools was using the tools in the operational space to provide us with the insight that we know so that we can, you know, we've got a lot more information and we've got a lot more data. And of course you should have those APM tools not only running in production, but against your.

Pre product environment, your test or your dev monitor is code. So having monitoring throughout the lifecycle means that say for instance, you've got an entire regression to be done and that's in this case manually. You can still capture all those manual tests and reproduce the automated scripts out of it. But if you've got automation, which is the example for Brexit, so they'd already invested a huge amount in BDD and doing spec flow in net and they generated loads of Selenium scripts out the back of it. We ran through all of the regression packages and then we converted those same scripts into non functional scripts. And as soon as the functional script updated, the non functional script also got updated. We leveraged traditional automation to create non functional, but you can also use the manumation approach and manual effort to create both automation and performance assets as well as real users. On the backend system you've got pretty much, you've got four or five different Options of how you actually generate the scripts and baseline those scripts against a production environment or a set of test scripts.

Also with automation, for automation to be successful, developers need to create their test to be automatable. So is it the same thing with Shift, right, with performance, do developers need to do anything in order to make sure that when it does get to production, it has hooks in it to make it so it is able to gather this information you need once it is in production?

No, it's not really the same kind of thing because of course with performance you're capturing the request response pairs, you're literally getting the raw data. Of course, if there's data that's maybe encrypted or there's data that you cannot have access to, then you may potentially have a challenge around how you create those scripts. But I'll give you a bit of an example with the AI platform that I've been doing some performance against this today.

I use a gated methodology which we've designed internally, which is a goal orientated approach to creating performance scripts. This is mainly around the moving.

The moving target which is building an AI platform. So when they start off and you start off with your kind of requirements, you've got to think about and this is one of the kind of questions and it's a bit of a behavior change which we've had to bring into the organization, which is you want those operational people in your definition phase. So they're going through maybe Sprint planning or something and they're saying, well, we want the end state to look like this. You know, we're going to have to optimize that the MongoDB improve the caching. You know, we've got an expected to have a higher transactional throughput. So we're going to have to look at cube and how we scale that and you know, how many nodes we've got, you know, all those kind of questions that they may be adding to the, that particular story. And then we come in with this gated AI approach where we look at creating a goal which we're trying to achieve. So for instance, we're trying to prove something so that if it's the, the patent system, we're proving that it's able to correctly categorize the, the designs or the trademarks or if it's in the, for this AI platform where it's using computer vision, so it's looking at pictures and images of fashion, for instance, and it's, there's obviously going to be misclarification. So there's going to be an error rate which we're expecting. So that's goal is the overall view of what we're trying to achieve. Accuracy is the level of accuracy we're expecting. So typically, as you'd expect with using something like Tesla Flow or you know, any of the other computer vision platforms, there's only a certain level of classifications that are going to be successful and that's typically for enterprise kind of grade training algorithms. You might be talking about 80%, 85%. So therefore we're expecting 15% of those transactions to actually fail. So they're going to not, they, they'll not get classified and then they'll go over to use some kind of augmented intelligence. So using Mechanical Turk or something or someone human eyeballing a picture of a shoe or something and creating the data set manually. So that means that the actual loads only going to potentially get through to say 70%, 80%. But we've also got a time aspect for the T of the gated is the time frame. So if we've got to push through, like I said this morning, half a million records, you know, that needs to be done in a certain period of time so that, you know, just dropping it onto for instance, Kafka and then you know that just the consuming period may take, you know, minutes, hours, days, depending on the, the amount of data that you've created, the data set that you're running. So you know, gated, the D aspect is your data. So what you use for your training sets the size of your synthetic data set for performance. So we use a data generator to generate all permutations of the data that we need for our performance test. And that's really where the E aspect of it of gated is, the enrichment. So this is where we have to use different types of heuristics to go off and enrich some of that data. So you know, I give the example with the, the image for the fashion aspect. You know, the temptation when you're training a algorithm might be to say, well actually I'll take, you know, 100, you know, I'll take a thousand or a million pictures off Google off trainers. And that's great if you're wanting to create a generic trainer algorithm, but that's not very good for then recognizing anything else but trainers. And if it's looking at it has to identify shoes, trousers, you know, he has to be more generic. But also where do you get the date and the metadata to support that training exercise? So sometimes when it comes to, let's just Say Nike as a, if Nike was the customer we would create those meta models and those semantic models from historical data that they have. So I'm guessing, you know, Knight will probably have a big ERP system which will have a whole stack of information about sizes, colors, you know, manufacturing SKUs, weights, materials. We can extract all that information and create an AI ready data lake to mine all that data and all that useful information and all the photos that they may have on their, their website, they might have historically on previous years of website. And we train the algorithms specifically around.

Their data set and you can use things like spiders to go through and extract all that information. And once we've got that model we can create a more realistic view of what a pass or a fail looks like. And this is the whole point is that you know what you would get in the traditional automation landscape of an assertion. So or you know, check whether there's a value comes up. Each set you run will have different outcomes. So it's no longer about a simple pass or a fail or a check that the value equals something. It's more around, you know, confidence of the data sets that you're doing and the outcomes that are coming through and whether or not the accuracy is to the level you're wanting or it's within some kind of tolerance level. And that's really what performance is around, is around understanding tolerances and what's good enough but also what that has as far as the impact to the end customer and whether or not that's going to potentially mean if they can't publish every single trainer on Nike onto Amazon and they can only do it once a month, you know, what does that mean as far as the business objective? And the business objective might be, you know, we want to be able to do some three or four seasons of shoe promotions per year and if you can't do that based on the platform then that's an overall fail. But as you can see these, these individual tests are now going up a level above to a business context. And you know, all of these systems have to interact and be performant to prove out an overall business hypothesis.

So Jonathan, some people listening may think this sounds very sci fi. How many people are doing this now currently in production? Where do you see the growth curve? Where are we in this approach in the shift right type of performance environment you've been speaking about?

Well, I've been kind of talking about shift right for probably like the last 10 years now. And it's unfortunate that maybe that we're slightly behind the Times, you know, a lot of the modern practices are still very much on that kind of creation, creating new scripts and, you know, starting from non functional requirements. So, you know, understanding that they're expecting three seconds or something for a system to load. And this is a bit of a shame really, because performance engineering is an art and that is kind of understanding what are the future volumetrics, what happens if you take on 10 customers, what happens if you take on 20 customers, what happens if they have a thousand items versus a million items and maybe the business grows or maybe it's later in the lifecycle. So, you know, I had a customer contact me yesterday who had just deployed an application into production and they said it's, you know, it's incredibly unreliable, it's unresponsive, it crashes. We, you know, we've turned on all the tracking and you know, the response time for every Page is sub 3 seconds, but everything just freezes. And I said, well, you know, it's not about the request response time, it's not proving that the UI and the domain object model is loading up in, you know, in an acceptable amount of time. It's that all the back end systems and the workload that's going through in the workload that, you know, it's under network virtualization or service virtualization, understanding all the different endpoints, that that's actually the reason why it's, it's, you know, and this is why things like site reliability and chaos engineering are really helping drive the industry to start thinking about these questions in a slightly different direction. Instead of thinking of performance testing or load testing, which was kind of what we saw back in the Mercury Interactive days, which was more, you know, let's call it, which is what Astra load test was, which was in the day. It was just the QTP product stuck into a click and script. So a, just a container. So like a headless browser proxy, really, back in the day the idea was this UI very much kind of focused on user activities. Whereas now in the landscape of microservices, just because your rest APIs has been sent a message doesn't mean that it's ended up on Kafka or it's got all the way back to your graphdb, you have to look at every single part and every single transaction of that journey, that pure path all the way through the systems and all the different systems it touches. And that is incredibly complex. And the UK government, when we implemented the APM tool, it looked like looking up at the sky at night. They're all stars where you were all interconnected somehow between each other and mapping, that was an extremely complex exercise. There's no real way of saying, well, there's a tool or a way to fix this. I think the approach is close.

Understanding your APM tools.

Understanding the apm, understanding the infrastructure, the IT operations tools, they're the tools that we should be learning from the DevOps perspective. Once we're in those operational spaces where we're looking at graftana and getting data around queues and what, what's happening with the actual system and transactions and looking at the consoles and really the logs and what has been generated out of the back of the logs. And then once you've got that whole landscape, you can start plugging in from the APM tools, the test modeling tools, to model those out, because they're going to continuously change. And this was the problem, kind of what we saw probably 10, 15 years ago was people were creating performance test suites and then they were quickly getting out of date like they would be doing in test automation. It was very easy to get out of date, which is why if you can refresh them from a production source or from a APM stuck in your test or your development pipeline, then you could quickly regenerate.

Those performance scripts without having to manually craft them by hand, which, you know, it's great to craft them by hand as far as you can do assertions and you can, you know, you can add in additional logging information you can take, you know, you can understand better where the data is coming from and how it's passed between each one of the calls, for instance, or through, you know, database queries. But I think that can get down to too granular level. You've got to start off with an end state or a steady state. And I think that is where the APM tools and.

A production environment really helps. But a lot of these new platforms like the AI as a service don't exist in a production state, which is why you then need to be able to model the data as well.

As the tests. So you can't, you can't get data from a system that doesn't already exist. And equally you can't train these systems without huge amounts of test data. So it becomes a test data management challenge. And that again is one of the big things we've been focusing on over the last 10 years is around model based data as well. So how you model complex data which has lots of different states in it. And I think this is the thing, it's not just tests, it's data and performance. Three separate models.

All right, Jonathan, before we go, is there one piece of actionable advice you can give someone to help them on their journey as they shift right with the performance testing effort and let us know the best way to find or contact you.

Sure. So I think I might have mentioned earlier my big recommendation would be to look at the operational tools and spend a bit more time investigating the advantages so attending events that are from the IT operations side. So I was one in Helsinki last week and it was incredibly useful to talk to the operational guys to understand where you can leverage structured and unstructured data that you can use for your tests and then, you know, understanding if you look wanting to look at the cognitive side of things. Like I'm speaking at a deep learning summit this week with around AI Ethics and you know, there's a lot around Google Brain and DeepMind and the Microsoft Research guys, they all have fantastic artifacts which you can download around cognitive. So you know, if you want to find out more, go and have a look at some of those white papers. I'm doing quite a lot of work with the European Commission, which is the AI alliance, which you can download papers that people are submitting around AI and what they're doing within their industry. So there's plenty of sources of material that you can go off and look at.

The best way to contact me would be through LinkedIn. So LinkedIn.com and then it's the one word automation will come to my page or you can contact me via any of the other mediums that I can I'll provide with the link below.

Thank you Jonathan for your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com p like performance P03 and while you're there, make sure to click on a try it Today link under the exclusive sponsor section to learn all about SmartBear's awesome performance testing solution. Load Ninja and if you like this episode, why not rate it and review it in itunes? Ratings and reviews go a long way to helping the show get more people to listen to it and it really helps me get feedback to see what what direction I should take future episodes. So that's it for this episode of the Test Guild Performance Podcast. I'm Joe and my mission is help you succeed with creating end to end automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the guild and continue your testing guide journey. This has been a Joe Calantonio production.