Drupal Performance Tuning with Brain Tully

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Brian Tooley, all about performance testing, Drupal applications, Lamp Stack, performance tuning, and much, much more. For the past two decades, Brian has developed web experiences and products with one goal in mind and that is to build high quality software that delights the end user and provides an optimal user experience. Brian has extensive experience developing and performance testing Drupal sites for companies like hp. He also has a passion about web performance optimization and user experience based on his work with large popular sites like consumersearch.com to help them become more performant. He has a lot of knowledge and actually was one of our most active and helpful members at this year's Perf Guild. Really excited to have him on the show today. You don't want to miss this episode. Check it out. This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks, and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja as a SAS UI Load testing tool and Load UI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey Brian, welcome to the Guild.

Hello there Joe. Thanks so much for having me.

Awesome to have you. Before we get into Brian, is there anything I missed in your intro that you want the Guild to know more about you?

I guess that I've been working at Acquia, which is one of the main Drupal companies that kind of fund Drupal and utilize Drupal for the past four years. And I head up their launch readiness practice.

Awesome. So what is the launch readiness practice?

That's a good question. Launch readiness. Basically what we try to do is before an application goes live, we kind of run it through the process of making sure that performance standpoint and a security standpoint, everything is good. That any assumptions that were made early on in the project still are valid. And certainly on the performance side, where I tend to focus on, you know, that's where we really put the application through the process of, you know, load testing, performance auditing, doing front End audits, et cetera, just to make sure that they're going to have a successful launch.

Great. So before we actually dive into the nitty gritty of that, for folks that don't know what is Drupal or lamp, either one.

Yeah, sure. Drupal is a content management system that's probably been around for over a decade now and it's kind of built primarily on PHP, MySQL basically open source, Open source content management system. It's kind of been evolving over the past 12 or so years. It was originally developed by college student Dries by who's now the one of the main folks at Acquia. He's still kind of like the cto. And yeah, we're all like very involved in the Drupal community. I think what attracted a lot of people to Drupal was just the open source mentality, the fact that it was using program languages like php, which are relatively straightforward and simple to kind of learn. And yeah, I think Drupal has been evolving, like I said, over the past 12 years or so.

Very cool. So is there anything different about performance testing, say a Drupal site than say maybe a typical web browser application?

You know, regardless of what framework or what system you're using, there's certainly going to be a list of things you're going to want to test. Sometimes it depends on other some tools better than others to test that application. Certain frameworks, like if you were testing a project or an application that's running using Django. Django is another content management system that's built with Python in mind. So you might leverage more testing systems within Python rather than say PHP or if you were building a site on Ruby on Rails. But I think the concepts are relatively simple is, you know, you want to make sure that you're testing as early as possible and as frequently as possible. And you want to make sure that what you're testing is valid, that you're testing like user scenarios that are going to be representative of what you know, real users are going to be doing and that you're doing it on a basically with a load, a traffic load that is going to represent what real people will be visiting your site with.

I know like with some solutions like SAP and Salesforce. I'm not saying they're exactly like Drupal, but they usually have like APIs that allow you to do things with them. Even WordPress has like rest APIs you can use to interact with it. So when you're doing performance tests, is there anything in Drupal that gives you maybe an API access that folks may not be aware of.

Yes. So certainly Drupal is very similar to WordPress in some ways in that both are PHP based, both have APIs, RESTful web APIs that you can perform things behind the scenes rather than standard page requests. And certainly with the newer versions of Drupal Drupal 8, which has been out for the past few years now, they've evolved significantly where now it's leveraging the. I don't know if you're familiar with the Symphony PHP framework, which is more object oriented php, so in some ways it looks more like Java than it does old, old school PHP procedural code. And with that there are now APIs. A JSON API which is like a JavaScript framework, just similar to the GraphQL, like a query language to get JSON responses so that if you wanted to, you could run headless Drupal or decoupled Drupal so that if you wanted to kind of put in place a framework like react or Gatsby JavaScript, you could have a front end that basically calls the Drupal API so that you're still using Drupal as a content management system because that's really where it shines and helping you develop kind of your content information architecture and editorial experience. But if you wanted to say, have a different, completely different front end like have React or Gatsby or Vue JS serve your site, you could do that.

So is that a common protocol you usually use when testing Drupal? Is it the API protocol? Is it HTTP protocol? I know at the Perf Guild you had some knowledge around JMETER and Selenium to know how much of that comes into play.

Yeah, absolutely. I think for my purposes and my experience with Drupal, I think JMeter is probably the tool that I lean on the most, mostly because I've been using it for the longest period of time and I had knowing how Drupal works and knowing what are some of the common Drupal gotchas, I find those are relatively easy to do in JMeter since Drupal is primarily just making HTTP requests and then depending on the application. Nowadays every application seems to be a snowflake. So you really have to kind of gauge how much of your are going to try to emulate the browser experience. So in addition to HTTP requests, are you also does a page load also make several API requests like, you know, API JSON requests and, and then, you know, make sure that you sort of emulate Those in your JMeter test as well so that you get as realistic a picture as possible.

So what are some of those common Drupal gotchas you just mentioned?

Well, I think any content management system that you're going to be using, there's going to be sort of an editorial or an authenticated user experience. So if you're going to try to emulate those that in Drupal or in your JMeter test, you want to try to build around that. So that could be simple as submitting forms or logging into Drupal. Whether it be the typical Drupal login or if you're integrating with like an external single sign on solution, there are certain gotchas that are there in place to prevent like cross site request forgery. So for instance, in Drupal anytime you submit a form, there's usually like a form token that, you know, form token, a form build id. These are kind of like dynamic variables that change based upon certain security parameters that are hashed within your settings. So that no one can do like a denial of service attack by, you know, constantly submitting forms, et cetera. It'll basically validate that those tokens are valid. So you have to know when you're developing a test how to extract those values from the page before you submit the form request so that you're submitting valid values. So knowing little things like that, knowing that for instance, if you're trying to develop a test that integrates with a single sign on solution, there's usually extra like JavaScript web tokens. There might be certain security nonces that you would need to figure out. And in those cases that's sort of one of the main areas where I'm trying to get performance testing earlier in the process, in the development process, because it's usually the developers that know that information the most. So they can certainly develop, for instance, Postman collections. Postman is an application you can use to kind of simulate HTTP requests, other types of requests and sort of automate them. It's especially good for building like a REST API. Like if you're using a Swagger API Restful endpoint, you can use Postman to emulate requests. So having like a dedicated or a completed Postman collection of like how the API is supposed to work, how you're supposed to get a token, any of that information is really helpful because that can help you understand the API and then understand how to develop that in any testing framework you're using, whether that be JMeter or Locust or anything else.

Interesting. So you're on the launch readiness team. So is that after everything's been developed and ready to go and you're putting it through its paces before you deliver it to a client? And if so, how do you get it so performant? Like, you didn't mention developers getting them involved. Are there any other tips you have to get them more involved so when it is delivered to your team, that it is more performant?

Yeah, absolutely. So, as you probably know, over the past few years it's been this momentum within the testing community to shift left. So the idea that you're not waiting to the last minute to kind of test your application and that can be, you know, behavioral tests, functional tests, unit tests, but also now, you know, performance tests. So the idea to shift left would be, you know, earlier in the process, in your development process, or as you're developing features, as you're deploying features, you know, why not build a relatively simple performance test, at least a performance measurement that you can then gauge after each sprint or after each release. In other words, is there, has there been any regression in performance? And you can use that, you know, especially as you get into the idea of performance budgets where you're kind of consulting and kind of, I guess, compromising with the business team in making sure that the features that they're requesting are not hindering performance. And if so, what can be done to modify that or ameliorate that so that it's not as bad gives you a little more flexibility in being able to say, well, we can do this, but maybe not do this. Maybe this feature isn't a good idea because it adds 500 milliseconds or it adds a complete second to each page.

Great advice. So, you know, you did mention authentication credentials. It is because it's a Drupal site. Is there any, like, attention issues in Drupal that a lot of people get tripped up on? Like, like a single point of entry that no matter how many users you have, they all have to hit and it'll cause performance issues.

Yeah, absolutely. I think in any framework, when you have authenticated users, you're, for the most part, you're bypassing cache. I mean, any performance recommendation you have, any performance focus should be. Cache should be your main, your primary reason of, like, how we're going to make the site perform well. So the more you can cache, the less the application needs to be bootstrapped. And by bootstrapped it needs to open up php.php then needs to call the database, et cetera. All of these expensive operations, the more you can Offload that to cache, the faster your application will be. But with authenticated users, for the most part, you're bypassing all cache, so you're kind of bootstrapping every time you're making a request. And so a lot of the focus has been certainly with Drupal 8, the latest version of Drupal is they released the concept of dynamic page cache, so that even for authenticated users, sure, the most parts of the request are going to be uncached, but some things can be cached. So for instance, identifying pieces of a page that are cacheable, that don't change whether you're an anonymous user or an authenticated user, and let's cache them as dynamic page cache so that when the page loads, it can use something similar to like Edge side includes, where you're loading the cache sections of the site first and then for those items that can't be cached, load placeholders and then load those items that need to be dynamic via like AJAX requests. So it gives an improvement in perceived performance.

Speaking about performance, doing this performance test, are there any key metrics usually use with Drupal or counters that you use that you found helpful specifically to Drupal beyond, you know, CPU and memory?

I think, yeah, I think with Drupal you want to, you know, obviously you're looking at average response time, you're looking at cpu, you're looking at memory. You're also looking at how many PHP procs are running. Has there been exhaustion? So in other words, the application has so many users that require uncached requests where the number of PHP processes that are running on the server are overwhelmed. Once that happens, the exhaustion happens, you start to see fatal errors. You can also have PHP out of memory errors. So depending on how much memory you have on the server where PHP is running, you can allocate a certain amount of PHP memory per processor. And then based on that, that kind of tells you how many PHP processes you can run. But if your site, if you have certain part of code that requires more memory than what you've allocated, you'll have other fatal errors, memory exhaustion errors. So then sometimes you might need to increase PHP memory for the application, but then because of that you're limiting the number of PHP processes that can run. So it gets to be, you know, it gets to be a little tricky and like, what's more important, giving the application more memory or having more PHP processes run. And sometimes you can get around that by conditionally allocating more memory for certain processes or certain routes, certain functions within Drupal. But yeah, I think those are probably the two main areas of where you'd of metrics you'd want to look at. And then of course, with Drupal, since you're using the database, and since you're using other solutions like memcache or Redis, which offset some of the caching from the database to those services, you then want to make sure that you're looking at those, that they're behaving as well, so that the databases, you know, the database CPU is not thrashing, that the disk IO is not thrashing. And in terms of memcache and Redis, you want to make sure that you're getting enough cache hits that it's sized correctly so that there's not frequent evictions. Because otherwise you're, you know, you're kind of bypassing the benefits of using memcache or Redis.

Do you have to ever worry about the version of php? I know on my site it says you need to upgrade to PHP 7.3 in order to get better performance with WooCommerce, I don't know, does Drupal have like you have to use a certain version of php, Is that something you even have to worry about?

Absolutely. I think with the latest versions of Drupal, they certainly recommend php7 at a minimum, because like you said of all the performance improvements. So I don't think going forward PHP5 will be supported. But even as you said, within PHP7, you know, there have been incremental increases from 7.1 to 7.2 to 7.3, 7.4, which I know was just released recently, has some additional features. And then now learning about PHP8, I think they're planning on some significant changes as well.

Now, because Drupal is a content management type of site, kind of like WordPress. What do you focus on when you're doing a launch readiness? Do you just focus on the end user or do you also focus on maybe an administrator that needs to go in and update content and things like that?

That's a great question. I think I would say it's probably 60, 40, where 60% you're focusing more on the end user. I think primarily since anonymous users are. Unless you're developing like an intranet site, which you know, we've certainly done, where everyone is logging in and certainly authenticated users and the administrators, editors becomes valid as well.

You know, you also are an expert or you have a passion for a customer or a user experience. And a lot of times that depends on like a front end so it's not like a traditional load or stress test, it's like a one user type of thing. Any best practices you use to help make sure that your front end is performant from a one user type perspective?

Yeah, absolutely. So, you know, in addition to using all of your kind of traditional backend performance testing tools metrics, I always recommend using some of the free open source front end auditing tools. Some of them are built directly in the browser like Chrome has, you know, Chrome Developer Tools and Google Lighthouse, which can be really informative and very simple to use. I also lean a lot on webpagetest.org which is a great service developed by Patrick Meanin, who I believe now works at Facebook. It's a free service where you can enter your page URL and it will run it through the process of, you know, finding out what are your front end metrics, what's the time to interactive, the average page response time. It'll give you some recommendations on, you know, are your images compressed enough? Are they saved in the correct format? You can also see the waterfall view. So you can see that are there certain requests, specifically external requests? I find a lot of clients are relying on external JavaScript libraries or to add features to their site or to add to measure their site. Some of those requests can be blocking requests, so they'll slow the page from loading until they finish loading. So knowing techniques about how to load some of those things asynchronously so that they don't block the page and other things on the site that can be lazy loaded, can they be sort of offloaded until after the initial page loads and then things can load in afterwards, specifically things that appear below the fold. So you know, depending on what laptop you're using or how big your screen is, when you load a page you're only going to see a certain amount of the page in your viewport. So the more you can offset loading the items that appear out of the viewport until they appear in the viewport, that can give you significant performance increase.

Nice. So I'm a big fan of checklists, so I know you're part of the launch readiness team. Do you have a checklist of things you always go through before you say, okay, yes, this is ready to go shipped to the client?

Absolutely. We actually leverage, we've automated some of that. Acquia has developed a tool called Drutney, which I believe stands for Drupal Scrutiny and it's an open source library that I would recommend anyone who's using Drupal to leverage and it has some nice plugins and it's basically a command line tool that allows you to run certain audits and audits are packaged together as profiles. So there are certain profiles you can run that, you know, basically run one command. You'd wait maybe a couple of minutes as it goes through all of the audits and it'll give you a nice report of like, hey, here are what we say are the best practices for, for this version of Drupal. And you're doing this, you're doing this, you're passing this test. But hey, we've noticed that you, let's say you're not, you don't have memcache enabled or let's say you're not aggregating your CSS files and your JavaScript files, which is a best practice. So make sure you do that. So I think we try to. There's certainly a list of a check, like you said, a checklist of best practices that we follow. And we've also, like I said, built this open source tool that helps us automate that because we just found we were doing that over and over, so why not automate it?

So we spoke about PHP being upgraded. Does Drupal come out with minor upgrades? And if so, do you have to worry about testing between versions?

Yeah, that's a great question. Yeah, Drupal comes out with, I think Drupal just comes out adopted with Drupal 8 and Drupal 9 now, you know, the semantic versioning. So that's really good. Semantic versioning has been around for like an enterprise software for quite some time. So I think Drupal is finally getting on board with that. And Drupal comes out with basically patch releases anytime there's like a significant security or bug that is fixed. Then there's sort of the minor releases. The minor releases usually means that there's additional features that are getting released. So I think right now the current version of Drupal might be 891. So. And Drupal 9 of course has also come out. So that'll be. If it's not already out, it's. It's coming out soon. And so Drupal 9 is going to be. Even though it's a major version change because of the work that's been happening in Drupal 8, a lot of the changes from Drupal 7 to Drupal 8 have been completed. So now the first version of Drupal 9 is aim to be as easy as possible so that you don't have to worry too much about things breaking. So I don't think any new features in Drupal 9 will be until Drupal 9.1. But certainly you want to if you're doing an upgrade, certainly from even a minor version of Drupal, from Drupal 8.8to Drupal 8.9, there's usually, I wouldn't say so much testing that's required, but certainly there's documentation in terms of making sure that you're running any update scripts that need to be run to migrate the change.

Okay Brian, before we go, is there one piece of actionable advice you can give to someone to help them with their performance testing efforts and what's the best way to find or contact you?

Sure. I would go back to what we spoke about earlier, which is this idea of shifting left and kind of testing. Testing early, test early and test often is sort of the philosophy I'm trying to follow myself and also promote it within Acquia so that we can catch these sorts of any performance issues, any security issues earlier in the process rather than, you know, waiting until the last minute. And that performance testing doesn't have to be super complex. You know, there's lots of tools like I mentioned WebpageTest, Google Lighthouse, even running simple JMeter tests using something as simple as the BlazeMeter Chrome extension where it basically records your user journeys in the browsers. And then you can just press like a tape recorder button and then press a play button and it'll kind of run the test. So you can start off simple and then just, you know, start drilling down and getting more complex as needed. But don't be afraid to performance test and test early test often.

Great advice Brian. The best way to contact you best.

Way to contact me is probably on Twitter Brian Tully B R I N T U L L Y and Then I'm on LinkedIn so and my email address is Brian Tullyquia.com thank you Brian.

For your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com P41. And while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two great tools to help you with performance testing Load Ninja and Load UI Pro. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.