Performance Testing SAP with Helen Bally

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Helen Bailey all about sub second response times in Warehouse Management System in an SAP application and performance testing in general. I don't have a lot of topics on this. I'm really excited to talk about Helen. I'm really excited to talk with Helen about this. Helen is the Managing Director at IT Core Enterprise Testing Services Line. She leads the IT Core Enterprise Testing practice and has helped numerous Swiss and international companies with a test strategy, test automation and performance testing. She has a love of complex technical projects where she could drive results by bringing together technology, people and processes. Like I mentioned, if you do anything with performance outside of just simple web browsing performance, you don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks, and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and loaduipro, an API load testing tool to help teams get faster full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey Helen, welcome to the Guild.

Hi Joe. Thank you so much for having me. It's a pleasure to be here.

Great to have you. I'm really excited about this topic. But before we dive into it, is there anything I missed in your bio that you want the Guild to know more about yourself?

No, that was great. Thank you very much. I guess I don't know if I did. We mentioned that I worked many years in the States. I worked in London, but I've also worked in the States so it's always nice to talk to people from the States and moved over to Switzerland in 2003 where I've been working for IDCOR since then.

Very cool. So I don't speak with many folks that do any type of performance testing around SAP. So for folks that don't know, I guess can you just give a little high level what is SAP and what maybe you were dealing with there? Your recent project.

Okay, so I'll give the context of the project first and then I'll talk about SAP and why SAP is a little different. So I've worked in performance testing in a sort of leading a competency center and there I've been kind of responsible for heading up management of a lot of performance tests. So be it on E commerce systems or web systems or SAP systems. And I've got a lot of experience in SAP, so I'm able to kind of contrast what's different about SAP versus another kind of system. Perfect. So the particular challenge that we had this was as you know, SAP is an enormous system and even if you're working on a digitalization project or an E commerce project, probably somewhere in your company there's an SAP system or two tucked away somewhere, maybe doing finance, maybe doing purchasing. And in many large companies SAP is really the backbone of everything that's HR finance, purchasing, logistics and it handles all the sort of enterprise resource transactions. And when you enter into the SAP world you enter into a whole different world of language where people talk in acronyms and they understand German words and it's very much sort of a closed community. And SAP itself has got a lot of different products and they talk about ECC as really being like the central component. And that's really the sort of what I call the mothership where you're going to have maybe one big instance or maybe a big, you know, if you're a global company, maybe four and this is where you're going to have manage all your customer orders, your sales, your logistics and everything. But then there are other products and satellite system from SAP as well. And one of these is EWM warehouse management. And this is a system that SAP was very much pushing people to go towards ew. And with EWM it's a warehouse management system and so it can be installed as, as I said was as a satellite system. So you might decide to set up regional EWM system or you might decide to set one up a warehouse so you to have a look. And so it's got a smaller footprint than the mothership, the central component. But obviously that means that you're going to have to communicate between SAP, the mothership system and with your satellite systems. And this communication is done through as ever with SAP it's done through some proprietary interfaces using like IDOC and R. So you could imagine if you've got a warehouse management system, what are you concerned with? Warehouse management you're concerned with put away processes. So you've got stuff being delivered to the warehouse and you want to get it into the warehouse and you're dealing with picking processes where you're saying, trying to get the stuff out of the warehouse and onto a ship, onto a. Be it a truck or a rail car and get it out. So these are really the principal processes we're talking about. Anyway. So in my particular context, I had a customer and I'd done a bit of work for him before, and he came to me and he said, you know, we've got this global roller that we want to do of this new warehouse management system called SAP ewm. And because we're a very large company and we've got a lot of warehouses, then we're looking at, you know, implementing this multiple times. So it's a company that's hundreds and hundreds of warehouses. So we really, really want to get it right. And for us, performance is really important. So for us, we want that all the key operations that an operator does, and they're doing most of their operations with an RF device, we want those to be under one second. And I was really excited by this because for anyone who's worked in performance testing, I found very often your customer doesn't really set metrics around what performance they want. Very often they tell you, I just want good performance. But they're very. It's unusual that someone actually comes out with a bold statement with an actual.

Number and a real number. Right?

And a real number and a measurable number. So this was very exciting. And I love the idea of like sub second as a target. And so. And they'd already started, you know, doing some things. But he wanted to turn it into a proper project because he could see that it required some kind of organization.

I'm just like trying to wrap my brain around this. When you cut these requirements, like, we didn't even know where to start when people stopped bringing in RF devices and you have these satellite regional places, like, I mean, I'm just used to bringing up a browser, putting on a load, measuring the response, like, where did you know where to start? And how to even process, like a game plan to handle this.

First of all, he was clear about the target, he was clear about the scope of the transactions, which was the. What they called the RF transactions. Fortunately for us, we weren't dealing with actual devices and all of this. They said, we have an RF emulator and we will take the response time really from the RF emulator. And he says, for the moment, we're not going to worry about latency at the warehouse. We're really just going to take from a base system in the head office and we're going to measure it there. So they've given some thought to this already. So that was the first part of looking at the rf. And in fact, some people have been working on it for a while and they'd already come up with a number of different challenges that were making this difficult. So the first one was to test the actual communicator process. So the RF device that, that was easy because we were allowed to use an RF emulator. They'd already given some thought about the scope of the transactions that they wanted. So they said, we're really concerned with the operator clicks on really looking at the putaway process and looking at the picking process. So these are the transactions that the operators do all the time, and it's the ones that they needed to be the fastest. And if you think that, you know, on one delivery you're going to have maybe 50 to 70 pallets on a truck that you're going to have to unload. And the thing about the warehouse management system that makes it perhaps different from other systems is that we're not dealing with a huge number of parallel users. Like, I tested many of the systems you talk about, you're worried about, you know, what happens under load when you've got a large number of customers coming or a large number of people going to your site to do something. But in this case, we really weren't talking about a huge number of parallel users, so we weren't really worried about the sizing of the system. They were actually pretty confident on the sizing of the system because often when we do performance tests, that's really the first thing we're looking at is how is the system tuned and how is memory, how are resources allocated? But in this case, what we were really looking at was, well, what happens when one user on the system by themselves clicks something? And when they click, there's a huge amount of intelligence built into the system. So if I'm doing a putaway, so I've clicked, you know, the, the barcode to read the barcode, the system says, well, what sort of product is it? Is it a dry product, is it a frozen product? So where it calculates whereabouts in the warehouse, it's going to have to go. And so it understands the structure of the warehouse and where the different areas are. And then on top of that, it's going to think about, okay, what kind of batches are there already? Is there warehouse full? Where is there a free shelf? So with every click you do, there's a huge amount of intelligence for the system to really come up with either what's the optimal way for you to put this in the optimal place for you to put this pallet into the warehouse. And so our challenge really was finding out not where there was a system resource issue in the system, which is often one of the things we have to do in performance testing, but it was really looking at the actual coding itself and how we could optimize the coding. So we were straight into a performance engineering project, which for me that was very interesting as well because, you know, typically you just start off with, throw a bit of extra CPU at the problem and hope it will be okay.

Yeah. What I love about it, like you said, when you think load performance test, people automatically think of load test. But this almost seems like single user type performance under real world conditions, which must have been fun to try to emulate.

I guess you're exactly right. So the real world conditions was something that we got a lot of advice on. So there were some really smart people on the customer side and including some architects from SAP as well. They knew the system really well and they gave us a lot of direction on what was going to be important. Important. So very early on we decided we'd need our own dedicated system. And to anybody working in the normal, should we say digital world, that sounds like that's not a big issue, you could spin up a new server. But in the SAP world that's really quite unusual because the nature of the systems, they tend to be quite expensive, they tend to be quite large, they tend to be highly integrated with other systems, they're integrated in the transport paths. And just spinning up a system for someone just to use for themselves is not so common. So we felt enormously privileged to have that. That meant that we could really protect our data and we could really protect our resources. Because you can imagine when you're measuring response time of a second, you know, you really want to. Don't want anybody running anything else on the system that might make you doubt, you know, if you did it. 1.2 seconds last week and now you've measured 1.1 second, was that 20% a gain in programming efficiency or was it just that you point two of a second out the week before? So we really wanted to make sure that it was repeatable. So an isolated system was really important for us. The other thing for the real world conditions that was really important for us was having a warehouse that, that looks like a real warehouse. So as I said before, the programming logic that's done when you do a click is quite complex. And if you've got a very simple warehouse structure, then that logic is not going to be difficult. But if you've got a test warehouse that's got, you know, lots of shelving, lots of storage areas, storage types, and also lots of inventory then, and also with different aged inventory, then the system is going to have to work harder to calculate the right answer for you. And so having these real world conditions was going to be important to be able to get the right sort of response time and make sure we were doing something realistic.

Yeah. So along those same lines, then, I guess the performance test data, how do you make that realistic? Because how do you populate an SAP system must have all these different components that need to be populated with certain amount of data. Like, did you seed it with transactions or like, how did that work?

So we were able to take an existing. When we took our system clone, we cloned it from a warehouse that had already been set up. So the structure of the warehouse was good. The next thing we had to do was fill it up with stuff. So we got the inventory levels of a real warehouse in production and calculated how many batches we needed to have. And so we did a fairly simple batch program just to fill it up to, I think it was about 70% full, our warehouse. But then the thing that, for us that was really smart was and again, the SAP architects involved in the project, they helped a lot here. One of the challenges we have was reproducing these deliveries going into the system. Because the deliveries were quite complex. They had maybe 50 pallets on and it took a huge amount of time to set it up from SAP and we wanted it to be totally and perfectly reproducible each time. So what they did is they. We figured a smart way to hook into the program on the SAP central component side that would send the delivery over to the warehouse management system. And we basically hacked into that program and it sent a little clone of our delivery each time. So in the normal world you would just call that setting up virtualization. So we basically were able to hack the integration between the two systems and then we could just send our little delivery or our big delivery as many times as we wanted, using this utility program that reproduced the function call over to the EWM system. Again, I think in a normal web system you'd be like, okay, big deal. But again, SAP is there's a lot of proprietary stuff going on and that was quite a smart thing that saved us a huge amount of time and made it very reliable as well.

Very Nice. So I guess another proprietary thing or something people may not think about. I think you mentioned this in your presentation at the pat. Sometimes when we do a performance test, we focus in on a normal day's transaction workload. But I think you mentioned in the presentation that sometimes people ran a report in SAP and that kind of crashed the performance. Like, how did you catch that? Is that. Like, how did you know that things like that would affect. Affect performance in an SAP type system?

Absolutely, because we were dealing with tiny, well, not tiny times, but, you know, the difference between 2 seconds and 1 second. So we really wanted to make sure that nothing was interfering with our measurements. So first off was to get our isolated system so that we were the literally only people on the system, so that, you know, if there was somebody else running a report, we knew that wasn't interfering because there was nobody else on the system. The other smart thing that we did was my performance engineer came up with this, is that we'd have these deliveries with say 50 pallets on, and, you know, each click you'd go away, put away each of the pallets, you click, click, click, one by one. So what we would do is there's a feature in SAP where you can take a trace of a transaction and it's used by anybody that wants to debug what's happening. And when you take this trace, basically the system captures every function called and all the data that's called sending to all the different components. And it also says how long it takes. So it gives you this perfect breakdown of exactly what's happened when you made that click and that transaction. So we wanted to be able to capture these traces of what was happening in the system. But at the same time, we were worried the tracing itself would interfere with the performance. So what we did is we do every n timesing something like every five times. So we would do four palettes, we would capture the response time of those four pallets, and then on the fifth palette, we would stop capturing response time and we would put the trace functionality on. And this was all handled by the script and it would capture the trace. We'd give it a little naming convention saying what test run it was and what iteration it was. And then we would start recapturing the times again. So all these transactions would run overnight and we'd come in in the morning and we'd be able to see. So we downloaded, we had a nice big dashboard and we were able to analyze and say, okay, this time is interesting. We want to drill down and understand you Know this is coming in as three seconds and we want it to be one second. Let's go and look at the trace transactions. And we set up a meeting with the developers and together we'd be able to say, okay, listen, this thing here is taking three seconds. And we'd look at the trace and they'd be able to see exactly why it was taking three seconds, which was fantastic data for them actually to be able to actually see in real world conditions how long their transactions would take, but more importantly, why.

This may seem like a pop quiz, but I think you mentioned seven smart things as well in your presentation. We go through them really quick. The first one was virtualization.

We've got this tight integration between SAP where you create, you have a customer order, you create a delivery, you set up all your data if you like, and then you've got to send that delivery over to your warehouse now management system and this is sent through. There's a couple of proprietary protocols in SAP. One of them is RSC, the other one's IDOCs. So depending if you're sending delivery or shipment, you're sending these. So what we were able to do was to sort of hack into this interface and we were basically able to stub it so that we could remove ECC effectively and we could resend our delivery as many times as we want. And that saved a huge amount of time and made sure that we had a really repeat, repeatable process. So this was the first thing that helped us with that. The second thing that we did, which was, I think, I mean in the performance testing community, was clear to anyone, was scripted everything. And I say it's clear, but because when we came in on the project they were still like, okay, do we need to script? Can we stopwatch it? We're only doing with a single user, but we wanted everything to be, you know, very scientific and very evidence based. And the nice thing about a script is it does it the same way each time. We were able to capture the response times and then extract them afterwards. We did have some fun choosing which tool we'd script in because we've got the, the SAP protocol. You don't have the same range of performance testing tools that you do for say web. So you've got a limited number of tools. We had a small debate about whether we would use like a functional testing tool which works on pulling up the screen. But eventually we went with a performance testing tool because it was going to be faster.

What tool did you end up using?

Yeah, so we used Loadrunner we could have used as you've got limited choices for SAP GUI, you've got the IBM Rational, I believe there's SAP GUI, you've got LoadRunner, that does SAP GUI. Now you've got NeoLoad, that's got a SAP GUI protocol. So you've got quite a limited choice when it comes to actually doing performance testing with SAP gui. And again we could have used a functional tool. But strangely also in performance testing the SAP GUI protocol is actually extremely stable. I've never seen anything like it in the web world. But you can almost write the script yourself if you know the language and you can pull open a script that's six months old and just run it and it'll, and as long as the data is fine in the system, it'll work. And which is exactly. That's the right response. Like with a web, with a web script you would never expect that. You're like whoa, hang on a second, I just got to correlate the pajazers out of this. But SAP gui, you can almost write it yourself. It's a really, really nice protocol. And so that way. And I think the last reason was my performance test engineer just, it was a complete whiz with Load runner and so he's like no, I'm writing. And I think you're always better off when you're writing with a tool, using the tool that you know really, really well. It's like, you know, I'm just like you, you do it as you, as you want to do it. So setting up script was a great way to get everything automated. And again that's like a no brainer in the performance testing community. The third thing was the isolated environment which as I said before was a huge privilege also in the SAP world. The fourth thing we did was to make sure that the stock was really representative. And as I said, we sort of had this kind of ideal warehouse that would be 70% full. But one of the things we did as well is we had utility programs that would go and restock in different areas because as we would run our test scripts then over time, you know, we'd pull out too much stock or we'd put in too much stock. And so we wanted to make sure that we kind of had a balanced amount that it wasn't, you know, overstuffing it or under filling it. So we would also control that on a weekly basis. The fifth thing I think I said also was the, the SAP trace capture. And again that's a, it's a really nice feature in SAP. But if you do want to drill down, because this was a performance engineering system, it was not about, like, finding whether the bottlenecks and resources were. Were resources were. It was really all about how can we optimize the programming. And you can really look down at every call and you can get down to the millisecond level. And you might say, okay, 10 milliseconds. What's 10 milliseconds here or there? But if you've got 10 milliseconds, it's called in a loop, and that loop's called, you know, 50 times, and you're doing that loop another 50 times as well. Those 10 milliseconds could add up. And it was really useful to see, you know, on a real delivery or a lifelike delivery, what the response times actually were using this tool was and why. And I think the other challenge that comes sort of stepping away from my seven things but a small bracket there is that there's one kind of performance test project for me which is like, find the problem. So you're. You're called on to, you know, you send a bunch of users to the system and then you run and you find a bottleneck and then you sort of, you know, rub your hands and you leave. And they, they're left to. To figure out what they have to do. Whereas there's other ones where you really have to see through. And that's what I call more performance engineering project, where you really have to see through the recommendations and show that they work as well. And this was really good to be part of the second kind of project where we weren't just showing where there was a bottleneck. We were showing, okay, here's the issue. And now please, Mr. Mrs. Developer, can you go and resolve this? And then they would come back, you know, maybe after the next Sprint or so, and they'd come back with the new program, and then we'd rerun everything and we'd have to demonstrate and, you know, present to our clients, you know, where the improvements were. So that was really nice to be part of a sort of a performance and engineering project, end to end. So coming back to my seven points, the sixth one was we set up regular execution, and we also had a dashboard, and everybody loves a nice, colorful dashboard. And it's a really good way of talking to your customers who at the end of the day are paying for everything and want to see results. So we were able to extract all our data nicely from our performance testing tool and automate, getting that in and able to show the trends over time because in some time some of the cases we were reducing by 20%. So going maybe from 1.2 seconds down to 1 second, which. And finally the last thing that I did, I was really, really grateful for me and my customer, we've done a lot of things, a programming level and at a tool level and integration level and we wanted to capture all this into a kind of repeatable methodology. So he gave me some time at the end of the project to really put together kind of start to finish. If we ever they wanted to do this again, then this is how they would set everything up. And I was perhaps the happiest person that I'd done that because six months later he called me up and said, okay, we're putting in a new database, we're putting in a Hannah database, which is the new database from SAP at the time and we want to see what's happened to the transactions because when you put in a new database you'll find that some things have improved, but you may find that, you know, the coding isn't optimized to deal with a new database. And so we were able to crack this all open, bring out all the tools and run everything again to see what it.

Very cool. So I mean I haven't done hard core performance testing in many, many years. But one of the frustrating things for me was when we go live, it always was different in production than it was in the test environment. So quick retrospective, you had a real client, you went live, he came back to you six months later, so obviously you did something right. So how did it perform in production? Did it meet what were you told them it would do, like how it perform and any key takeaways of what you would have changed or did you modify your test methodology based on what you learned from what happened once they went live?

Oh, that's such a good question. I'm trying to think if I actually know the answer to that question. I don't think I ever got the return from. I know they had certainly there was the impact of, of latency and change things, but I don't think they had any issues. The main issue they had really I think was on the reporting side and because that was one of the things that we looked at when we went to look in for the HANA database is that they'd found that some of the reports or the queries that they had were not performing at all well. And this was kind of a focus when they switched databases and that's where we saw a huge difference. But I Never got, should we say, I never had any negative return of after they'd gone live in production.

Well, I guess you knew if they came back six months later, you must have done something right. So must have done something.

I hope so.

Awesome. Okay, Alan, before we go, is there one piece of actual advice or lessons learned you can give to someone to help them with their SAP performance testing efforts and what's the best way to find or contact you?

Oh, so the lessons learned is for me it's a really, really different experience actually project managing through the changes. So I've got a lot of experience, you know, with project management, got a lot of experience with performance testing. But this is really the first one where, you know, we identified. Here's all the programming changes you need to make and you need to really see these through, get them implemented because as I said, often we're dealing with resource changes in terms of SAP, I think it can be quite daunting if you've got a lot of experience as a performance tester working on web because these people tend to speak crazy language, they talk in transactions with four letters and it can be quite daunting. Probably the two best pieces of advice is One is it's all about the data and getting the data right in an SAP test is by far the hardest thing. The scripting bit is the simplest and easiest bit because the protocol is so nice. But spending time getting the data right and working with a good analyst to get a good spec and making friends with that analyst because when you do something wrong, your data is going to mess up and you're going to need their help to fix it. That's probably the best advice for working on SAP and trying not to be too put off by the vocabulary. Making friends is good.

And Helen, what's the best way to find a contactor? Because I think you have a lot of great experience in performance and people would love to probably contact you on advice. Probably.

I would love to be contacted. My email, I don't know if you can share my email, which is h.ballytocore.com which you can also find on our ITRKOOL website as well.

Thank you Helen for your performance testing awesomeness. For links to everything value we covered in this episode, head on over to testguild.com P37 and while you're there, make sure to click on the try them both today link under the exclusive sponsor section to learn all about SmartBear's awesome performance testing solutions. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast Podcast I'm Joe and my mission is to help you succeed with creating end to end full stack testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.