Importance of Feature Flags in an Agile World with Swasti Sharma

Welcome to the Test Guild Automation Podcast, where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Well, I don't know if you noticed, but I've been really busy hosting my own conferences this year, as well as events for other companies like applitools. And the one technique that keeps coming up over and over again is Feature Flags. And it made me realize I've never had a podcast episode on Feature Flags, but I remember that I did have Swasti Shama as a speaker at the 2018 Testing Guild and her session was on the importance of Feature Flags in the Agile world. So today I'm going to take her session recording and create a mock interview to share with you all the testing awesomeness she shared around Feature Flags in case you did not attend that event. And as a side note, if you missed the 2018 Testing Guild or any other of the automation, performance or security testing conferences I've run in the past, even the ones I started in 2017, you could still purchase and access all the recordings for each event@guildconferences.com all you need to do is go to guildconferences.com click on our events, select the conference you're interested in, and then click on the Register now button and you get instant access to all the recordings of sessions that occurred for that particular event. But anyway, as I mentioned, I think Feature Flags is a really important concept as more and more companies are trying to also shift right and take advantage of activities you could do actually in production nowadays that helps the customer experience and helps you achieve higher quality software at speed. You want to miss this episode, check it out. The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial, visit test skill.comSourceLabs and click on the exclusive Sponsor section to try it for free for 14 days. Check it out.

Hey Swasti, welcome to the Guild. Before we get into it, can you just tell us a little bit more about yourself?

I started my journey in testing at Google. From there on I went on to Yahoo to lead their display advertisement platform. Then I got an amazing opportunity. For me it was a career changing opportunity to work at LinkedIn pre IPO and it was humongous amount of experiences, the changing processes and everything that I learned at LinkedIn really valuable and I was at LinkedIn for six years before I moved on to another company called 10X where I was leading their test engineering and app support art.

And now just recently I joined another company called SmugMug. Many of you may have heard about they acquired Flickr recently. I'm joining there as director of engineering. So pretty excited about that. As you can see that I've gone through so many changes in so many years in terms of companies and when it comes to companies, even one company itself follows different processes. I have seen so much process change just like you all continuously evolving processes in terms of releases and the tools they use to release a software out.

Swasti, you've worked at a lot of different companies. I'm just curious to know your opinion on how things may have changed and where we're going. So before you get into the session, maybe a little background around why you thought feature flags are so important.

The current trend in industry where companies are actually moving really fast into agile world and most of the companies are following the two week release cycle and how as a test engineer that's changing our lives because certainly in the waterfall model we used to have so much time to ourselves, right? We used to have ample amount of time to test the product file bugs retested to regression testing, do automation. You know, there was no hard pressed pressure on you to release a software with very high quality in two weeks time frame. It just seems unrealistic to me right now the amount of pressure all the test engineers are in. If they focus on testing the software out, they lose the time to automate the features. It's just a cache 22 situation. If they don't automate the.

Test cases then the regression suite becomes humongous. And humanly it's not possible to be testing stuff manually in so many browsers and so many different devices and things like that. So it's just becoming crazy for test engineers to cope up with the current trend of two week release cycles. Many times I've seen developers if you break down the two week release cycle into week one and week two and your sprint starts week one, many times I've seen developers giving the most crucial of the features also in the second week for you to test, if they're giving you a good crucial feature to test on the second week and you just don't have time because all of a sudden you didn't have any work, any relevant work in the first week and second week. You're just bombarded by the amount of tickets you're getting to validate and Then you are left with no time for automation.

Life seems crazy at that point of time. And I just wanted to give you some idea and some perspective of how you can benefit from the already existing tools and bring them into your process to improve your life a little bit as test engineers or as test engineering managers. So we will be talking a lot about how we can incorporate a B testing frameworks into our release cycles to make our lives easier and how to release the software with the utmost quality.

So for the folks that may not know what is a B testing or I think it's also called feature flags.

AB testing is basically testing piece of your software or feature to the audiences in a way that you can validate and you can test different variants of feature for different set of audiences.

So can you give like maybe a quick example of what that would look like?

For example, if on a UI there's a print button at the top and people want to actually see if the print button is even needed there in the ui, they can put a print button as an AB test and then release the feature out to 50, 50% of users. Other 50% don't see the button, they see and then we capture the metrics and the tracking to figure out which one is useful. Are people actually using this or not? And then make a judgment call on that. And I've seen lots of companies doing a huge amount of A B testing and that is another challenge for test engineering people.

So Swasti, I'm just curious to know how, have you seen, how is this usually achieved? Is it in house software, is it off the shelf software that's used to help achieve this type of functionality?

Majority of the companies using AB testing, I don't know if they have framework or not. Majority of the companies I have used a B testing in is they have their own either their own a B testing framework. Because earlier remember how AB testing used to work is they used to write a piece of code in an if else statement, right? And that's how they used to verify that, okay, this condition satisfied, do this, otherwise do that, right? So that's how a B testing used to happen way back. But as you're going more and more into a B testing, life is becoming difficult because the management of a B testing is becoming crazy. You cannot say it's just humanly, it's just impossible the amount of code you will have to write to just manage that. AB testing for multiple features, right? Luckily for us there are many companies who have come up, at least few I know of who have Come up with the a B testing framework or a platform I have personally used at LinkedIn. We used to have our own AB testing framework, internally built, a B testing framework. But at 10x I have used a tool called LaunchDarkly. It's a paid tool. This has been super helpful by the way, for me both at LinkedIn and at 10x.

So why do you like a tool like LaunchDarkly or what kind of features does it have you think most AB testing tools should have?

LaunchDarkly is a tool which manages how you can run a B testing. There's another tool in the market called Split IO. Split IO is from one of my peers at LinkedIn. So definitely something you guys want to check out. Definitely something that be super valuable. What both of these tools do essentially is you can have a set of feature or you can have one feature and then you can keep the feature behind a flag. They call it feature flag, right? You keep the feature behind a flag and say, okay, if my flag is turned on, please display this feature to the user. If my flag is turned off, please do not display the feature to the user. Right?

And suddenly I thought that, wow, we have this check, right? So what if I don't have enough time during the released cycle and there's a blocker bug coming at late night and I've just been handed over this code on my plate and do I am expected to spend my whole night testing this or do we have to fix forward? If there's a bug, why can't we just turn the flag off? Like what is so critical that it has to be released every two weeks? Right. So some of it is. A lot of it is more of a culture change for the companies. A lot of it is culture change for the companies.

So back in the old days where I come from, it was really frowned upon to do anything in production. So can you talk a little bit more about what this mindset type of shift needs to occur for companies and teams to accept this type of approach?

So they have to have a mindset that it's okay, it's okay to let the features feature code go out in production, but the feature being turned off if my QA hasn't qualified the feature for me, or if the development didn't get time to work on the feature.

Right now I see what happens is either we scramble to fix it forward if there is a bug in the critical feature that is planned to go out. Majority of the time we try to fix it forward, spend a lot of time and especially Test engineers have to spend a lot of nighttime because that's when they get the code in their hand. Right. They spend a lot of time scrambling to test that code, which they have no clue about at the last moment. Do you think they would be able to do a good job there? Certainly not, because they did not get time to think about all the different use cases to exhaustive testing there. Right. And that is very much the complexity of being a test engineer in Agile world. Now imagine a situation where a feature comes to you, you don't have time, you know it's behind a flag. At any point I can turn the flag off, I don't have to roll the code back because many times this, because of so many interdependencies, rolling the code back introduces more regressions than it fixes things. Right? All of us know that. And if your regression is not automated, you miss that part as well. So that's really not effective solution. So imagine situation, you got a feature and now you know it's behind a flag and you're testing it. You found a blocker bug and it's like eight in the night, you found a bug and you're like, oh my God, I don't have any developers here. So you make a point to tell and communicate it out to your product and your dev counterparts that, guys, I have tested this. This came in very late in the cycle. I don't think we can release it because I don't feel comfortable, I don't feel proud to ship this code out. So we'll have to turn the flag off. What are your thoughts on that? Right. It's always, I've seen many people.

They hesitate to push back a little bit, but it's your responsibility. You know, if. I totally agree, if the business needs something, you should work to make it happen, you know. But many times we just don't push back enough. Even when we see that the code is not of a good quality. Why are we letting. Letting it go out is something I never understood. So for me it's very important to push back a little bit and say, guys, this is the thing you have to set as a culture, right?

So are there any steps or approaches you would take if you didn't have this functionality in your current team and organization? How would you implement or how would you go about getting stuck started with feature flags or a B testing?

The first thing you should do is if there is no a B testing framework in place in your company, then you create a framework, you either build it, but I would not Recommend recommending building a solution. I would instead ask you to look, check out LaunchDarkly and Split IO. Both of these are super awesome feature flag management softwares. Implement it. You'll get so much recognition and so much value out of implementing these tools at your workplace. I have personally seen people actually benefiting a lot, you know. So if there's no, no feature flag management tool in your company, I would really recommend, especially for consumer companies and the company is doing a B testing, highly, highly recommend you to leverage these platforms. What they can do is they can put a feature behind a flag and then you can release it slowly to your audience based on a geography and IP address, or if someone registered at certain period of time, or if the member ID ends with certain number or a digit, right? And, or you can, you can really hone your target audiences with these tools and it's very easy. It's like an on and off switch to do that. But your development will have to integrate the code into it. It's still.

It'S still not very easy to do. It's still time consuming. It's still a little bit of work, right? Because managing the code now of AB testing framework is also an added work for a developer. So I've seen a lot of pushback from developers as well. But the process I introduced is once the AB testing framework is introduced in your company, what you can tell the development team, talk to the development team and say, and the product team and show them the benefits of it. For the product team, it's extremely beneficial because with the switch of a button, they can actually release as a percentage. If your company does not have an A B testing culture, then the first thing you should do is introduce that culture of AB testing. It'll be super beneficial to yourself and to everyone in the company, whether it's product or development for product, it'll be super beneficial because now they can release a feature and target the audience the way they want to. They can target the audience by gender, by IP addresses, by geography, by email addresses, by bunch of other things. Now they can roll out a feature slowly and steadily to the audiences to see how the audiences react or to do user research for the people who are already part of that experimental bucket. All of this without changing the code, right? These software tools like LaunchDarkly and Split IO, they basically give you the tool at the tip of your hand to just turn the buttons on and off and actually add targeting based on what you want to do.

Swansea as you know, a lot of times it's Hard to convince developers to embrace a certain, certain technology or a certain approach. So can you talk a little bit about maybe the benefits that developers would get out of using feature flags?

It'll be super beneficial for your development team because now they can. They don't have to roll back the code, right? The, the satisfaction they get out of releasing the code out to production is super huge. And that's why they are very bummed when they have to roll back the code. Firstly, because there's so much dependency of different services on each other that rolling the code back is never a neat process. You know, there's always some of the other regression issue or something or the other at the last moment, which you will again have to fix forward. So now you're left with either fixing forward the regression bug which came out of rolling back the code, or fixing forward the bug which got introduced with the new feature code.

It's not a good process. Instead, imagine a situation when the feature is implemented as a feature flag. It has a feature flag. All you can do is just turn it off and move forward. And later on when you get time, after your scheduled release is done, take your time, verify the fix and do a hot fix and turn the feature flag on. It's so much less pressure than actually doing it in the middle of the night when you're sleepy and when you're tired from the whole day. And it's just not a pretty process, you know, for anyone.

Secondly, it's. You can also sleep properly at night thinking that you're not pressing the changes to go in when you're super tired and you just don't know what to do.

So this seems like a great approach, but there's always pros and cons to anything. So can you tell us a little bit more about why or some things for folks should look out for that could be an issue by using an approach like a B testing or feature flags.

A few things in my experience which we need to take into account as test engineers. One, Please remember that this does not come for free. What I mean by that is there has to be still some amount of code which is maybe it's three, four lines of code, but there's some amount of code the developer has to put in.

To actually integrate with these tools, right? So it's never free because once your feature is live to 100% of the audiences, now the developer has to go back and clean up, which is called sunsetting of the feature flag, which many people I have seen do not do. And thus it just Complicates the code, right? So sunsetting. If you are introducing an A B testing platform process, then you should also have a sunsetting the A B testing feature process, which is create the ticket in the next Sprint or the Sprint after whenever. Once your code is live to 100% of audiences for some time and you're confident that we don't have to ever switch it back on, please clean up the code because that is just not good practice in general.

The other thing I've seen test engineers do is once they get used to this process, they completely forget that they have to test the feature flag aspect of it, of the feature. So what I mean by this is they get a feature, they just assume that it's behind a feature flag and they start testing it. At the end day, they find a bug, they go back to developer, they say we need to turn this off. Now when they turn this off, either something really broke or the feature flag itself was not implemented in a correct way or it was not implemented at all. So all of these things I think as learning, you should always make sure whenever you get a feature, the first thing you test is with the flag off and on scenarios work. If they work, then then continue with the testing of the feature in detail. But before that it's, it's just, it doesn't make sense. I think with some of these things in mind, you can totally, totally leverage these processes and actually have a peace of mind and release product which you're proud to share and not just under pressure because that's not exactly why we are hired for.

Thank you Swasti for your testing awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com alikeautomation a310 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Source Lab's awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.