Fake Data in Testing with Chiara Colombi

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Kiera all about fake data and testing, which I think is a really important topic that's often overlooked if you don't know. Kira is the product Marketing manager at Tonic AI. She's a bilingual wordsmith dedicated to the art of engineering with words. And Kira has over a decade of experience supporting corporate communication and internal companies, including a bunch of different companies around the world. She has actually translated for the Pope. Being an Italian American who's also a raised Catholic, I have to hear all about that. And she has a lot more overlap with translating for developers than you might think. Like I said, test data, fake data, really important topic. You don't want to miss this episode. Check it out.

The Test Guild Automation podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit test skill.comSourceLabs and click on the exclusive sponsors section to try it for free today. Check it out. Hey Kira, welcome to the guild.

Hi Joe. Thank you so much. It's awesome to be here.

Great for you to be here. I'm really curious to know, like I said in that bio, I know it's off topic, just curious to know how you end up translating for the Pope, which Pope it was and what that was all about.

Yeah, so it was for Pope Benedict, but my background is primarily in content creation and translation and for large stint in my career I was primarily translating from Italian to English. And one of the projects that I worked on was in depth guidebook to the art and architecture of one of the Papal Basilicas in Rome. So the end client is really the Pope because it's one of his basilicas. And yeah, I kind of say it's similar to translating for developers because there's a lot of, you know, highly technical language that you're using and you have to make sure it's accessible for anyone across the, you know, kind of spectrum of consumers of that information.

Absolutely. I'm just curious to know, you know, when it comes to test data, fake data, if you're using your application in different countries, does fake data come into play when, say you want to? Maybe you're doing all your tests against English, but then again you release it against Italian. And you may not have actually tested that data set. Could that cause an issue?

Yeah, I think it depends on the specific data types that you're working with. I mean, addresses is one that comes to mind and we do. You know, that is something that I've heard a lot about people saying, okay, well we need to make sure that the addresses are localized to France or to Japan or wherever you are in the world. And that's definitely something that you need to keep in mind when you are generating fake data is not only just the localization of it, you know, the translation of it, making sure that it's accurate to where your users are, but also the characters that you're using that can also come into play. So there's a just one of the many nuanced complexities of creating realistic fake data.

Absolutely. You know, I also already forgot, how did you end up at Tonic?

Yeah, it's kind of a fun story and like I mentioned, my background is primarily in content creation. I'm also a fiction writer. And the way I ended up in Tonic is actually I was working on a side project with my brother. We were creating a video game, a text based adventure video game which maybe some of your listeners have played or not. It means basically you're really interacting with text on the screen and there's often a very strong story component to it. So I was, you know, drafting the text, creating the story. And as we're working on this, my brother's tapped on the shoulder by a couple former co workers and they say, hey, we're, we're starting this, we've got this startup idea, data synthesis for developers, though not the data science use case as much. We want you to join. And my brother eventually was convinced, gladly, happily. And since I also had that content creation component to my work, he said, by the way, my sister does content creation if you need any early marketing support. And that started off, you know, just small and eventually ballooned into four years later. Hi, I'm your product marketing manager.

That's great. That's actually how I think I first came across Tonic is you're a content creator. I think you've written a few articles I saw like how to, I think mass data for testing. You wrote in one blog post optimizing modeling software testing strategies with better test data. So I thought we'd dive in a little to maybe some of those topics. When people think of testing or development, and I'm not sure they always think about fake data. So I guess before we really dive into it, what Would you say is fake data or synthetic data?

Yeah, I think in the context of software development, fake data, there's a couple different flavors that you have to keep in mind. Because fake data, the term synthetic data as well, they can both mean a lot of things to different people. So the most basic level, the simplest level, fake data you can think of as just dummy data that you might just plug in random, you know, characters in to just fill your fields. Very, very random. Or you might use some open source libraries to source a bunch of first names, last names, things like that. You might write custom scripts to create data that looks the way you want it to look. And then on the opposite end of the spectrum, you've got, where you're really thinking about data synthesis is generating data using machine learning, generative models or neural networks. And that's really more the data science approach to creating fake data to really achieve the highest degrees of realism that you can in your output. Another important distinction to make as you're thinking about fake data is you've got to think about the fake data. What are you generating that fake data from? You've got two options there. Basically you're generating it from scratch. You've got no data to begin with. And that approach is typically what's called rules based data generation in which you define some rules and then based on those rules, the data is generated. So for example, you might say, I need my users to be within the age range of between their 20s and their 60s, generate data that fits that age range, whatever the distribution is evenly distributed.

Sure.

But you're not looking at specific existing data or range or distribution. You're just saying here's the rule, make the data that looks like that. On the other end of the spectrum you do have generating data based on real data. So that's a very different approach. It's probably familiar to teams, QA and testing teams, who are maybe de identifying their production data in house to use in testing and development. So again, here you might use custom scripts to remove PII from your production data. But it gets really tricky, especially with today's data, given the scale of today's data, the complexity of today's data. But that's like a, in a nutshell, what we mean when we say fake data.

Absolutely. And I used to work in healthcare and one of the biggest issues was you couldn't use real data because it was patient data. So you had to find a way to kind of, you know, mask it and everything. So what are some other issues you've seen maybe with Using production data or building test data in house for testers that may cause issues or that people may not think about.

Well, it makes a lot of sense that you bring up healthcare that's health tech and health data in general, that's been more protected for a longer period of time because you've had HIPAA in place for a lot longer. But more recently, you've got the newer data privacy legislation that you've got. GDPR just celebrated its fourth birthday, so it's out of those toddler years, full and running. You've got ccpa, California Consumer Privacy act, which is now two years old. And these regulations are what's really hitting other industries outside of healthcare, outside of finance, that also has PCI, that has been regulating the use of data and finance, basically all industries. If you've got consumers based in the EU or you've got consumers based in California, you now have to think, keep these regulations in mind in how you're using data. And that also means how you're using data in software development. So that's really the big push for what we're seeing in the industry right now of software teams, QA teams, testers, all grappling with this problem of I no longer have access to production data because of regulations, I've got to find something to test with. So we're either doing this in house or you've got this new industry cropping up of vendors providing solutions.

Definitely. And once again, when healthcare was really not annoying, it was frustrating or scary when an auditor would come in and say, let me look at data and everything. So if they ever found anything that was like, oh, was real patient data or something you would test them with, you would get hit pretty hard. So I assume gdpr, CCPA would be the same type of deal. Why people need to worry about it. You would get fined, I assume, correct?

Yeah, there's fines. And also I would say consumers are more savvy, consumers are more aware of. I want my data to be handled properly. Data privacy is important. We should all be caring about it. And I think that not everybody realizes the way our data is used. We're more afraid of, oh, it's in some database somewhere, and what if it gets hacked? But it's actually just how our data is being used on a daily basis that we need to be more aware of how companies are using our data and how they should be using. What better ways are there for them to achieve the same outcomes with data?

So besides regulations, maybe is there anything else? Is there using a tool that's really specialized in fake data that create better quality data because I guess your test only is as good as your data. So even if you do have a solution, if it's not good data, then garbage in, garbage out, obviously. So anything like that, something maybe like tonic AI specializes in that makes it maybe cleaner data or something like that.

Yeah, there's a lot to talk about there. And I kind of touched on it a little bit in talking about how today's data, it's really big, it's really complex, it's really regulated. And that's. I think a lot of companies have been trying to build a solution in house for themselves. They think, well, how hard could it be? It's fake data. And our CTO has on a webinar we did recently said, I see the world as having two types of developers or QA testers, two types of folks who are trying to source fake data. There's the folks who say, who have never done it before, and they say, well, how hard could it be? And they go ahead and try and build something in house. And then he said, there's the engineers or testers or whoever who have tried to do this before. And they realize this is super, super hard. It's a lot harder than you think. And there's just a lot of complexity involved and a lot of hiccups that come up along the way. And your data is, you know, you think, oh, well, my data is special. I'm the only one who knows it. Well, your data is special. Everybody's data is special. And that's the complexity that is making it so hard to mimic it realistically. So there's a lot of things that you have to keep in mind. I mean, the list is extremely long, but one top thing that comes to mind is, you know, your databases, where you're storing your data, how are you, how the solution that you're building in house, are you able to connect it natively to all of your different data stores? Are you working in MongoDB as well as redshift, as well as postgres, if you're working across those data types? So one thing you want to keep in mind is where your data is stored and what types of different data stores and data types that you need to connect to connect whatever you're building, does your solution that you've built in house, does it work for a data store as large as redshift, or are you using Snowflake? Does it work for NoSQL data? Maybe you've got data in MongoDB you want a solution if you're building it in house or if you're working with a vendor, you want a solution that can work wherever your data lives and also can work with all the different data types that you have. Data, as we all know, can get really messy. You've got like JSON blobs. Maybe you've got some first names in those JSON blobs and you've got first names in another column. Are you creating a solution that is going to create the same first name in the first name column as appears in the JSON blob, or are you creating consistency across your data, not only across data within one table, but across many tables or across databases as well? That's another huge thing to keep in mind and what makes it so complex. And another thing I'll say is not only is our data large and complex and all of that, it's also changing day by day. Like the data that you spin up two weeks ago for testing may not work with your use cases today. And that may be because the schema changed. It may be because you've got just different users and you need to be able to easily refresh and generate data on a very regular basis. So all of that factors into the complexity of sourcing quality data.

So I guess another thing is when people think of fake data, they just think of maybe a QA dipometer testing. I don't know. Are there other types of testing that benefit from fake data?

We're finding that fake data is useful across software development, from your sandbox environments to your development environments, to staging to testing to QA throughout the pipeline. We're finding that developers and testers and QA teams all can make use of fake data. And then even if you go outside of software development, we also see analytics teams at companies needing fake data because they also can't access data. Science teams who are building machine learning models themselves, and they need to train on realistic data, but they can't access the real data. All of those teams within an organization can benefit from quality fake data.

So you did mention there are a lot of solutions out there now. So I know tonic AI is one of them. What's the AI piece of tonic that maybe makes it maybe unique?

Yeah, I'm glad you pointed that out. I would say that is one of our strongest differentiators. So from day one, our focus has been on the developer, on software development and testing. We realized the strong need there because our founders were all working at software companies and ran into that roadblock of, oh, I Don't have any data. I can't access the real consumer data. I need to spin something up. Wow, this is taking a lot longer than I expected. So we came at it from that perspective. But we've also, two of the founders have PhDs in math and software development, computer science. So they also realize that if you really want to create high quality fake data, you're going to have to use some pretty complex approaches. Complex algorithms are going to need to be built. And we're kind of at this great golden age of the burgeoning of data synthesis. So our goal was let's create a platform that enables data synthesis, but for developers, because a lot of those early data synthesis players were much more focused on the data science. Use case like, let's take one table, let's upload one CSV file, and let's make it really, really good. But we know that developers and testers need, they don't need just one table of data, they need a whole database of table of data. And they need multiple databases of multiple types. But they also would like to achieve high degrees of realism. And that's where the AI really comes in. And the ML, we've got generators, we've developed algorithms that make use of neural networks, specifically variational autoencoders. We could really get nerdy on the data science. But suffice it to say, neural networks that can look at data of different types. So you're looking at categorical data, conditional data, location data, and recognize the relationships across those data, across those columns, even across rows like interdependent rows, and create data that mirrors the statistics of that data to really give you the most realistic data you can achieve?

Now, this may sound like a dumb question, but do developers and testers usually know what type of data they need? And does using a tool like this let them know, hey, this is what your normal data looks like. And so you're going to need these types of fields or something like that.

Could you restate that question?

Yeah. Just curious to know, when someone's trying to create, say they know they need test data, what do they need to do in order to generate the test data that the Tonic AI? What does Tonic AI need in order to generate the test data? I guess the question does look at production. Is it just developers and testers guessing on the data that's going to be needed?

Okay, got it. So what we built with Tonic is really a platform to enable teams to create the data that they need. It's not so much you give us your data and we'll synthesize it, we'll generate it for you. We're giving you the platform and all the tools you need to generate your data for your teams the way you need it. So the way Tonic works is it points directly at your production databases. The vast majority of Tonic users deploy Tonic on premises. So the data never leaves, you know, it doesn't come back to Tonic in any way, just all stays within their environments. You can point Tonic to different databases of different types, databricks, postgres, Mongo, like I mentioned earlier, and then you can, it'll basically pull into memory just 100 or so records that you can look at to say, okay, these are first names. They needed this type of generator. These are, I don't know, incomes. They need to be generated on a continuous basis over a statistically representative range or distrib. Tonic comes built in with wide array of generators. We have over 50 different generators that can be used, kind of like building blocks to mix and match specific to the data types that you are trying to protect or, you know, synthesize and fake.

So another weird question is, can you reverse engineer it? Say someone's writing tests using the fake data and they have a bunch of tests, but it's not using a lot of the. It may have edge cases that it's not using the data as much as it can be. Does Faker ever tell you, not faker. Does Tonic AI ever tell you, hey, you have some edge cases you may be missing, or here's some data that you're not using that may be, you know, a risk for you?

So there's a great. This question also reminds me of sometimes we're asked, well, what about if there's messy data? What if there's broken data? Well, essentially Tonic is built to mimic your data, so whatever's in your data is going to come into the output. And I haven't quite explained the output section. So, you know, you point at your production data, you can view and perform ply transformations within the platform Tonic and then you spit out to a net new database, we call it your destination database, that is of the same type. So if you're looking at production data in Postgres, then you're performing transformations and hydrating a new postgres database with the fake data. But so any edge cases, they're going to be in there. Because if they're in your data, Tonic's algorithms are designed to find those edge cases and put in your output as well. But that also goes for any messy data, which ultimately, if you're performing tests, you want to make sure even the messy data is in those tests. So you're catching all use cases.

So how do people consume the data? It goes in the database and they just consume it like they normally would with the data in a database.

Yep. Yeah, it just, you can connect it to your lower environments and away you go.

Nice. Are there any uses that you've seen people use in the wild that you didn't think would be used or didn't think of when it was released? Like come on, I didn't know people was going to use the product like this. That's kind of unique or something we didn't think about.

So a couple of other use cases that we do see people using. One we kind of expected the other maybe is a little bit more. Oh, that's awesome too. The one that we always suspected from the get go was sales demos. You know you want, you need quality fake data to show off your product now that you've built it. Well, if you can get quality fake data for your sales demos, that's a win for your sales team or for your sales engineers. And another use case that kind of came about over time was the more database native database integrations that we added. Now that we're working with data warehouses as well, we work with databricks, redshift, Google, bigquery, Snowflake is data warehouse tokenization. So that's not so much for the software development teams, that's more for the data analytics team. And again it's come about as a result of gdpr. GDPR basically says you can't keep data indefinitely. Right. There's a limit to how long you can store your data. Well, data analytics teams ideally want to be able to store their data long term, perform analytics over time. So data warehouse tokenization is a way of removing all PII and sensitive data from your database in order to enable long term storage and analysis on it beyond outside of the original use for that data.

And you did mention this data is not stored on your servers. Someone would have used tonic and stored locally. I think that's a big point for people that always on prem and off prem discussion. It's in the cloud. How do we keep it safe? You don't have to deal with that at all.

It sounds like, yeah, it's in your cloud, it's in your environments, it stay with you always.

Nice. So there any features you think people don't need to know more about or any favorite features of tonic that maybe you really jazzed about?

I guess I do have a favorite feature and it's been a favorite from the get go. I can't explain why. I mean, I guess we have an open source tool version of this that originally I was like, oh, that's interesting. And now that it's built out, it's really become much more complex. My favorite feature is our subsetter. So subsetting, if people are less familiar with it, it's taking a slice of your data, but it's not just taking like a random slice of your data. When done well, subsetting is taking a coherent slice of your data that preserves referential integrity while massively shrinking down your database. I mean, we're working with customers like ebay that has 8 petabytes of production data and obviously they don't need to de identify 8 petabytes of production data to do quality testing. They just need like 0.1% of that and they've got all the use cases they need. So our subsetter enables them to massively shrink down their database, which already minimizes your data footprint and means that your risk is already mitigated significantly in case of a data breach. But then you can also take that subset tonic. It all happens in the same place, the same platform. The subsetter works in tandem with the generators to de identify your data. So you're subsetting your data and you're transforming, de identifying it at the same time. And what that enables is you can really get very targeted test data sets for your testing environments that are specific to certain use cases or maybe specific even to a particular bug that a user is encountering. The subsetter is designed to target multiple tables if needed, and you can target based on custom where clauses. So maybe you want to say I just need users out of New York City. And then it'll pull all across all tables, even across multiple databases of different type. It'll pull from upstream and downstream tables and databases to really get you all the data you need related to that one specific use case or subset of your data.

Very cool. There are the companies you work with, you mentioned ebay and like maybe success stories or how they use fake data to help them.

Lots of stories. I love our stories. I mean ebay is a great one just because, I mean it's fun to be able to talk about ebay. Everybody, everybody knows ebay and also their story just, it's like just a perfect golden story. Share our use case. Basically, they realized they had a major problem. Staging was broken and the quality of their data was a large part of the problem. They saw this Problem. They said, okay, we've got two choices. Either we just nick staging, we say, all done, we're not using staging environments anymore, or we fix staging and quality data was a huge part of that fix that they needed. And so we connected with them back. I want to say it was in 2020, early 2020, and they saw, okay, this is exactly what we were looking for. We were looking to subset our data, our massive amounts of data. We're looking to de identify it securely. One of their main focuses at ebay is data privacy. They're very, very concerned with data privacy, which is why they didn't want to nick staging, because they didn't. They were not using production data and testing. That's just not a thing. So with Tonic, they're able to subset, they're able to de identify. They use what's called format preserving encryption, which basically enables them to get very realistic data that preserves the format of the original data for improving the quality of testing and staging. So that's one great success story. Partly the reason I like to share it is because they actually wrote their own case study of how they fixed staging on their technical blog. So if you look up ebay's technical blog, you can find that. I think it's called the staging dichotomy is that article, if you're interested in hearing how they did that. Other success stories, Everlywell, they experienced massive growth over 2020, over the pandemic, and maybe prior to the pandemic, you hadn't heard of Everlywell, that at home health test kits, and then over the course of the pandemic offering at home, Covid tests and among many other things that people were no longer going to the doctors for. Their development team just grew extremely rapidly to the point that they couldn't keep up with all the staging environments that they needed to create or all the testing environments that needed to spin up. And their data was constantly changing because they had so many new user workflows, they were rolling out so many new products, the app was constantly changing, and they needed to continuously spin up better sandboxes for development. And that's when Tonic stepped in to help out with getting their data refreshed on demand on a daily basis. They get new data sets every day. They just set it up to kind of generate the data overnight. And so when they get to their desk in the morning, they've got the latest data that they need for all of their use cases.

Very cool. So besides customer stories getting insights, I think you run a bunch of surveys as well, have you any interesting insights from any recent surveys you may have run?

Yeah, we've been attending a lot of conferences for the first time this year. It's pretty awesome to get back out in the field, meet people face to face, have those conversations. And when people are willing, we've been saying, hey, would you just fill out this quick survey for us? So we've gathered surveys at a number of different conferences, which is really interesting because some of those conferences are more QA testing based, some are more Software development focused. DevOps perhaps. But across the board the results are pretty similar. One result, one key question that we ask is what are you using to hydrate your test databases? Are you de identifying the data in house? Are you using a vendor? A third of companies say that they are still using production data. So a lot of companies need to just be mindful if you have sensitive data in there. And is it legal to be doing that still? Just, you know, people aren't necessarily aware of changes in the landscape right now, regulatory wise. So that's one big result that we've seen. Another interesting statistic is we asked who's responsible for sourcing that data? Because basically the onus is on that person, that team. And one, during one of those surveys we did see that 50% of the time it was QA and testing teams that were responsible for getting that data in place.

That's one of the reasons why I wanted to have you on the show. I know that's something I dealt with all the time when I was a hands on tester QA person. So that's why I thought it would be perfect fit. So thank you for sharing that. All right, Kara, before we go, is there one piece of actual advice you can give to someone to help them with their fake data testing efforts or development efforts? And what's the best way to find contact or learn more about Tonic AI?

If I had one piece of advice, I would say do some research, check out what's out there. You know, you're not alone in having to create quality test data. It's a problem all teams are facing right now. And there are a lot of new tools out there on the market. Check them out because they will in the long term, even in the short term, save you a lot of headaches, a lot of time. It really shouldn't be. We just don't think that the burden should be on developers or testers to be creating data, to be de identifying the data in house. It's just too much. The data is too big. It's too complex and it shouldn't be on you to be responsible for the data privacy and data security within your testing environments. Seek out a tool because they're worth your time. A couple of ways you can visit Tonic AI you can reach out to me directly. I would love to hear from you. My email is C H I a r aonic AI. You can also visit our YouTube channel. Just search YouTube tonic AI and you can see a lot of great webinars. We're constantly doing webinars so join us for a webinar. If you see those on our website as well, definitely reach out to me by email. I love hearing from folks.

Thanks again for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com a405 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to test guild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.