Testing SAP APIs: Strategy and Execution with Michal Krawczyk

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Michael all about his new book Testing SAP Strategy and Execution. Michael is an SAP mentor working at INT4. He's recognized as one of the lead leading SAP integration experts with More than 170 articles and blogs on the SAP community. In addition, he regularly speaks at SAP events and he knows a lot about the SAP Application Interface framework. He also is, as I mentioned, the author of the book testing SAP APIs strategy and execution. Really excited to have him on the show. I have hardly anything on the topic of saps. You really, really want to hear this? Stick around. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Lab. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Michael, welcome to the show.

Hi Joe, great to be here.

Awesome. Great to have you. So Michael, is anything I missed in your bio that you want the Guild to know more about?

Yeah, there's just one maybe small topic. So my company, the company that I'm a part of, INT4 has created also an automated testing tool for the SAP APIs.Int4IfTt and this is the book is also about the topic of SAP APIs. But we also discuss the typical use cases of the API testing.

So Michael, why a book on SAP API testing? Do you see this as a topic you see people don't know much about.

Or people strugg I believe so the topic of API testing is relatively very very fresh at SAP. There's not a lot of API testing done in the SAP projects where we can actually do a lot of the testing. But this brings up to the question why do we have things like this? And it's mostly because people are just not aware that the API testing can be done and actually what the API testing is.

So Mike, I know a lot of companies have like SAP UI automation frameworks that when you download they have like blocks of code that already exists that you can use to create SAP tests. So why would someone use an API over say a UI based approach?

That's a great question. So with relatively a large number of projects currently happening in the SAP world, the UI testing, the UI testing does not work. So let me give you an example. When we're Talking about the S4 HANA conversion projects, which are the projects where you migrate from the old version of the SAP ERP system to the new version of the SAP ERP system, basically the UIs are getting completely changed. So in the past we used to have the very old sub GUI screens and now they're being replaced with the UI 5 screens. So it's very, very difficult to do any kind of UI testing where if we think about the API testing, nothing gets changed over there because the systems, all over the legacy systems, all of the EDI partners, all of the B2B partners, they still expect the same messages from the system and they still want to send exactly the same messages to this system. So API actually is the only thing that doesn't get changed. So that's why it's very easy to retest it. And on the contrary, the UI testing is very, very difficult to perform on those kind of projects.

So when you say API testing, I think most people automatically are going to think nowadays REST APIs does that's APIs that are baked into the product itself, that have been around for a while and if so like what protocol is it and how do you interact with them?

Yes. So there are two topics that are really, really crucial to describe right now. So the first one is that we're not talking about API testing only, we're mostly talking about SAP API testing. And the main difference is exactly as you, most of the non SAP APIs are usually synchronous messages based on a specific, specific protocol. Just like for example rest, the one you've mentioned. Within the SAP world, it's completely different because most of the SAP APIs are first of all asynchronous and they use plenty of other protocols, like for example JMS, IDOCs or RFC, many other. But what this causes, just the distinction between the synchronous asynchronous messaging is one very, very crucial idea, that the asynchronous mess are usually persistent. So what we can do in order to test those messages, in order to test those APIs, we just have to fetch them and rerun them through the same set of transactions or through the same set of interfaces. So this is one of the biggest differences between the non SAP and SAP API testing. We can just reuse existing messages, we don't have to create anything. So it's just to let you know, it's completely scriptless. We can just go to the production, FETCH let's say 20,000 messages. And we can just easily rerun those messages on the development or on the test landscape. You can do it within minutes. There's no need to script anything to code anything. You just grab the messages and run them through on a different landscape. And this is something that you can never do with the UI testing. Or even if you would have to script something for the synchronous testing, this would be much different because synchronous test cases, synchronous testing always very, very rarely gets persistent. So we would never have those messages that we can reuse. So this is the basic difference between the non SAP API test and the API and the SAP API testing.

Okay, so how does this work? Do you have like it knows processes or API calls that have been made in the system in production, and then you're able to capture that and replay it in a development environment.

So here we also need to go one step back. I discussed those topics with a lot of testing gurus. One of my testing gurus is actually Michael Bolton from the Develop sense. And when I speak to him, he's always telling me, hey, why don't you go one step back? Why don't you go one step back? What do you actually understand by the API testing? And this is also a bit different in the sense of the SAP API testing, because most people and most testers actually, when they think about API testing, they actually think only about API functional testing, where you just have a URL, an endpoint, which you just need to invoke, you send a message over there and maybe you get a response or maybe not, if this is asynchronous type of test. But when we're talking about the SAP API testing, this is just probably one part of the API testing which I can describe later on. Another very, very common approach is actually to do the transport layer testing. Most of the SAP systems, most of the SAP RP systems, they actually communicate with the backend system, with other external systems through some sort of a middleware. It can be an SAP middleware like for example, sub process Orchestration or maybe SAP cpi or this can be also a non SAP integration platform like Mulesoft or Delbumi or anything else, Web methods, any other. And one of the very, very crucial things is actually to make sure that the changes on the middleware are also tested. And we don't to link those two with each other. Right? We can test them separately, we can do a transport layer testing first and we can do an API functional testing. So we have at Least those two types of completely different types of API testing. And now dependently which one we want to go with, we do different things. And let me maybe explain the API transport layer testing. So this is the type of testing in the SAP world. We've got tons of projects which involve middleware migrations, let's say from the old version to the new version, or maybe from one middleware to another, maybe from the on prem middleware to the cloud middleware, or there was sometimes the way back or maybe between the different cloud middlewares, like between SAP CPI and maybe Mulesoft or Delbumi. So we've got a lot of middleware related projects and they actually do not touch any of the backend system functionality. So what we can do over there, we can just take the messages from the old environment and once the new middleware platform is enabled, we can just rerun tons of messages through the new platform. This way we can check the whole new landscape, whether we had to develop those interfaces manually right now, or maybe we just migrated. So we're just doing some, a little bit of changes. It doesn't matter what we can do. We can just grab tons of messages from the production system, from the production middleware, the old system, and we can just rerun them through the new system. So this is the type of the API testing, the transport layer testing is something very, very new and very specific for the SAP API test. On the other hand, again, as you said, we've got the API functional testing and with the functional testing again, what we can do is we can take the existing messages because they are persistent, usually on the backend system, for example, as IDOCs or proxy messages. So these are the different type of asynchronous messaging in the SAP world. And again, we can just rerun them and check whether the business documents that got created using the reference documents are exactly the same as the business objects created using the current run. So again, basically we're just fetching the existing messages and comparing the outputs from the reference document with the current run.

So Michael, this might seem like a weird question. When do you test SAP because it is a vendor kind of framework? Is it only when you do like an upgrade or. I know there are multiple modules with SAP, so when someone adds a new module, is that when you would do testing?

Again, there's multiple use cases where you can do the API testing. So one of them, for example, a very, very typical one right now for the transport layer testing is where you do the upgrade of your SAP middleware. So you just upgr with the new service pack, maybe you're upgrading to the new version. And this involves a lot of changes, right. There's tons of new functionalities that come with the new system, so you need to be able to retest that. And currently, so just to let you know the numbers, currently we have, we're coming to the end of support for one of the version of the SAP middleware sub process orchestration and we will have around, as far as I remember it was around 70 to 75% of the customers out of the 12,000 customer base, they are still running on the version that need to be migrated to the new version version. So we're talking about really, really big numbers here. Right? And this is the transport layer testing. Also the customers are going from the on prem middleware to the cloud middleware. Again, it's a different product but it needs to be tested. Right? You cannot just say, okay, let's just replace the middleware and you just do the new development. But who will be testing that later on? I mean there's no one to test this, especially in 2020 right now. Right. So we also have a lot of projects like this. But actually one of the biggest type of projects right now is the S4 HANA conversion project, which is the migration from old ERP system of SAP to the latest version of the erp. And over there this is a very huge project. But mostly this is still a technical project. Even though there's a lot of new functionalities that are coming in with the new version, most of the customers, they only want to make sure that the old functionality is working and only then they want to start working on the new functions. So again, very, very beautiful case for the API testing. And as I mentioned, because the UIs are getting changed so much from the old non web technology right now to the UI technology to the Fiori technology. This is the only thing that you can actually test automatically for those kind of projects.

So Michael, I would assume that an SAP implementation is different from every company and it could be very complex. How does one know that they even testing the right thing? I was speaking to someone that was doing like a performance test against SAP and then he realized sometimes people run a weekly report or a monthly report and that killed the performance at that time. They never had that as part of their workflow. So how does someone know that when they're using these API replays that they're actually creating a realistic kind of work workflow?

So that's one of the again one of the beautiful things of the asynchronous APIs. So the only thing you need to do is you need to take the messages, the existing messages from the production system. Of course, it's a subset, right? So let me give you an example. Imagine that daily you have 1,000 interfaces, so 1,000 APIs, and each of them is sending 1,000 messages per day, right? So what you can do is you can say, okay, for my test, what I want to do is I want to test, I want to take 100 of those, of the messages. For each of those 1,000 messages, I want to store them in my system and I want to rerun them just for the testing purposes. So that means that I'm actually testing 100% of my APIs with a very, very good subset of the test cases. So this is the way how we do it. You don't need to, of course, you can select certain messages, right? If you know that some of the messages are better than the other, you can select those. But the best way is actually to take as much as possible from the production system and just rerun them on the test, on the development landscape.

So by doing that, you ensure that all the different interfaces that are being touched, because like I said, I talked to someone else and they were talking about like ewm, all these weird components that I never heard about. Like, I just want to know, like when they replay those APIs, we know that then it's going to touch all the specific interfaces that you need to be aware of without really knowing them.

Yes. This is one of the things how you can make sure that at least the majority of them gets tested. Because let's say, majority of the projects of the SAP project, the customers are just doing the change impact analysis and they say, okay, yeah, if we need to do this sort of migration, then out of this 1000, we just want to test those five interfaces. So think about the numbers, right? Out of the 1,000 interfaces, we just want to test five of them. And we will test them manually. We will try to test them manually. And what do the project teams usually do? They would test it with maybe five or six transactions. But with API testing, this is not the only way. What we're trying to show with basically with our product and with the approach that we've described together with my colleagues in the, is that the API, SAP, API testing can work in a different way. We can test, we can take hundreds and thousands of messages. So just to let you know, in terms of, again, in terms of the volumes. So most of our customers, they're running anything more than 20,000 messages per day just for the continuous testing and they run it in less than one hour because that's the speed also of the API testing. So they can fetch 20,000 messages from the production system, rerun them on the def and test system, it takes less than an hour, so they can do it on a daily basis. So we talking about, and if we compare it to the ui, this is not a UI replacement, don't get me wrong, right? We're not. API testing is not replacing the UI testing, it's just complementing that. Basically the API testing is if we think about the testing pyramid and if we go with unit testing, integration testing and then the user acceptance testing, the UI testing is usually done only on the UAT testing, user acceptance testing, while the API testing can be done in a much lower environments. We can do it on the unit testing, we can do it on the integration testing because we just take the messages and rerun them. So it's very, very simple to create, to maintain and very, very quick to run.

Since you do have a lot of coverage then at the API level, how does someone know what needs to be automated at the UI level then? Do you have any like tips on what should be at a UI level and what everything else should be at API level?

To be honest, we don't do the UI testing, so we just provide the tool for the API testing because we think that the UI testing, especially in terms of the SAP customer projects is just too difficult. So the customers have to define, they have to do the change impact analysis, they have to find the correct use cases that they want to test because they will be only able to test very, very limited number of transactions. Right? But this is also something in line with the market trends, I believe because when I was reading a book from Wolfgang Platz, he's the co founder of Tricentis, in his latest book, just from last year, he's mentioning that the API testing is actually the only type of testing that can be fully automated. And you need to think about this, what he's saying, right? He created a company for the UI testing and now he says here's the. Yeah, still the UI testing will be bigger and bigger, but the API testing is the only type of testing that can be fully automated. And we're just working with this trend because in the SAP world it's very, very easy to do this.

So I know a lot of things folks struggle with in general is performance, but it sounds like because you're able to generate a lot of messages using API. Do you see customers also using this for performance testing as well?

Yes. So this is actually one of the side features of the API testing tools, right? Because even though if you don't design them as a performance testing tools, the API testing is really, really quick. So just to let you know, for example, with Int4Iftt, we did not design the tool to be a performance testing tool, but since of all of our customers, they have such a large number of test cases, just because it's so easy to create them and so quick to run, they're using the tool. I wouldn't say they're misusing, right? Or abusing, but they're using the tool for doing the performance testing as well. Just because it's so quick. Right, the APIs. So basically the API testing is as quick as the API itself or the transport layer itself. So they're amazingly fast. We don't need to emulate any kind of screens, we don't need to do any sort of that. We just send a bunch of messages to the URL, to the endpoint and we check the results on the other hand, or on the other end. So it's amazingly quick.

So I've actually been testing for over 20 years, but I've never touched SAP. Are these APIs new to SAP or like, I'm just curious to know, over time has something changed with SAP testing that now people should be embracing APIs or the API's always been there and people just didn't know about them until recently?

It's great that you're asking this question. So from the functional point of view, there's absolutely no change, not even in the last 20 years, but I would guess even in the last 40 years there's absolutely zero changes in the API. The only thing that got changed is actually that SAP, in the early 2000, they introduced their own middleware process, orchestration. Right now at the time it was called, as it had a different name, external infrastructure. From that time, the customers realized, hey, maybe there is another way how we can process the messages, how we can expose the APIs, so we can do it through the middleware. But if we're talking about the backend, so about the ERP APIs, nothing got changed. It's that just no one understands a lot about this possibility of testing that because people are still concentrating on the UI testing. Also that's one of the reasons why the testing in the UI project and the SAP projects is very, very difficult. People regard this as very, very complex. Complex because they never have any other possibility apart from doing the UI testing.

So I think another struggle, and this should be a mindset, is when people think SAP, they may think more of like, I'm not saying it's a legacy application, but it's been around a long time. So they probably assume like you can't do modern type of testing development against it. So is that a myth? Can you use like common practices like continuous testing, continuous delivery, test driven development if you're working on an SAP application?

That's another great question. Because in the SAP world, things like continuous testing and continuous deployment, they don't even exist. When we talk about doing continuous testing and continuous deployment using the SAP API testing, usually the companies that have SAP, they just do the big guys and they think that we're talking in a alien language or something like this. And when we go to the non SAP world, this is something completely obvious that you have to do continuous testing, you have to do a continuous deployment, otherwise there's no way you can work with the SAP world. It's completely different. The only thing you can do is actually the UI testing. If you're trying to do something else, then you have to teach people. So if you ask me what would be the reason for writing the book, that was one of the reasons. So what we're trying to do in the book, we're trying to answer a few basic questions. We're trying to answer the questions like, why should you do the API testing, What should you test in the API? Who can do this, when should you be doing this and how should you do this? Right? So we're answering those basic, those five basic questions. But at first we need to teach people in the SAP world about something that has existed in the non SAP worlds for many, many years already. But in the SAP world it doesn't exist because I don't know if this is because people regard SAP as a legacy application or as a very old application. There's tons of things that you can test very, very nicely in the SAP world. It's just that no one is aware of that, I would say, or no one wants to push for that. That can be a different reason.

Besides awareness, are there any, like, myths around SAP API testing that you have to debunk a lot when you speak with clients?

Yes, yes, definitely. People think that API testing is basically the same speed and the same difficulty that's as complex to create as the UI testing. And when we show them that, hey, let me show you how you can create the test cases for all of your interfaces in one Hour, Right. We just deployed the crawler, it goes to the production system, fetches many test cases for all of your interfaces or all of your APIs, and we download them and after one hour you're ready to run. And people don't believe us. People think that this is not possible. We need to see it. What you're showing on the videos, what you're showing on the OpenSAP training that you have is not possible. So we have to show them that this is possible. And even sometimes then they don't believe us. But when the customers are starting doing some, a bit of the API testing, then they finally start understanding, okay, yeah, it's very, very quick. So why don't we run it on a daily basis? And then they understand the concept of continuous testing and some of the customers want to go even further. Then they start thinking about, hey, we have continuous testing, what about the continuous deployment? And we say, yeah, this is beautiful that you're thinking about that. It's a standard again in the non SAP world. In the SAP world, no one is doing that. Let's try to do it. So one of the biggest myths is actually that the UI testing is the only type of testing that can be done in the SAP projects. This is the biggest myth. But of course this myth is also related to a very, another very, very specific feature of SAP testing world that most of the, I would say all of the SAP projects or all of the SAP customers, they think that if you're trying to get a testing tool, you can only get a single testing tool. And this testing needs to be able to do everything. So basically what they choose, what they always choose, is they choose the UI testing tool. And in the non SAP world, whenever we speak to our non SAP customers from the testing world, they always tell us, what do you mean, one tool? No, no, we have five, seven tools. We have one for the mobile, one for the ui, one for the API, one for the performance, one for the loads, one in order to schedule everything to work together. So we have multiple tools, commercial, free, everything. In the SAP world, everyone is talking. Yeah, if we need to get a tool, it needs to be one tool that does everything. So of course it usually does just the UI part.

Yeah, I guess. Speaking of tools, then what tools do you see most customers should be using or don't know about, maybe that are in the market that could really help them?

I would say that there's tons of even non commercial tools that many SAP projects are not using. Like for example, when we're talking about the very Very simplified API API testing tools. These are things like Postman, soapui. Not a lot of projects are still using that and especially the testing teams are using that. So we're not talking about the developers. Right, the developers can do those things, but ultimately the testing is in the SAP world, is done by the testing teams or is coordinated by the testing teams. So they are the owner of the test landscapes and they never use those tools. So I'm not even talking about, you know, using very, very complicated enterprise testing tools. I mean not complicated, but enterprise testing tools like our tool int4iFTT. But it's even those simple, really simple and sometimes free tools like Postman. The testing teams in the SAP SAP world never use them, they never integrate them with the UI testing tools because they're always under this strange umbrella that no, we can always have to use a single testing tool. There's nothing that can break this chain.

Okay Michael, before we go, is there one piece of actionable advice you can give to someone to help them with their SAP automation testing efforts? What's the best way to find contact you and get a hold of your book Testing, SAP API Strategy and Expert Execution.

Yeah, so if I can help in this way, in any way, you can always of course reach me by my email or on LinkedIn. But one piece of advice that I would give is if you're doing the testing in the SAP world, just try to think outside of the UI testing. UI testing is not the only type of testing that you can do. There's tons of more. I'm not only talking about the API testing. There's performance testing, load testing, mobile testing, there's tons of different types of testing. And we can do this in the SAP world, right? We can still do this in the SAP, but don't get close with a single tool that you have at your company. There's also a lot of the free non commercial tools that you can use and they can be used also in the enterprise projects. So don't limit yourself to the tools that your company bought. Use everything that is possible.

Great. And Michael, is there a simple URL people can use to get to your book?

Yes. So you have to go to the SAP Press comm and then you have to type, you have to search for. If you search for my name you will find all of my books and then you can find the book about API testing. So that's the easiest way to get using the search on the SAP Press page.

Thank you Michael for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com forward/a311 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed creating end to end full stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to test guild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.