Using Synthetic Data for Machine Learning & AI in Python

Sam. Foreign.

Hello everyone and thank you for joining today's live training. My name is Reese and I'm going to be your moderator for today's session. We're going to kick off the session in a couple of minutes or so. We're just waiting so everyone has a chance to join. In the meanwhile though, let us know where you're watching from using the chat or the comments, depending on where you're watching. And yeah, tell us something you'd like to learn from today's webinar. Just to note, today we are using datacamp workspace, so if you don't have an account already, please sign up for one. We'll also be sharing a link so you can code along with us as well. We'll also be generating synthetic data to use alongside the live training. There are comments directing you towards that. And yeah, we're going to be doing that with mostly AI today. I will be posting links in the chats in the comments so everyone can do that. But yeah, sign up for workspace and yeah, sign up for a mostly AI account.

Foreign.

And thank you for joining today's live training. My name is Rhys and I'll be your moderator for today's session. We're going to kick off the session in probably about 30 seconds a minute or so. We're just waiting so everyone has a chance to join. In the meanwhile though, we'd love to hear from you. So let us know where you're joining from using the chat or comments, depending on what platform you're watching from. And yeah, tell us something that you'd like to learn out from today's webinar. Just to Note, we're using DataCamp workspace for the code along today, so if you don't have an account already, please sign up for one. We're also going to be using mostly AI to generate synthetic data. There's a link to, sorry login to most AI today to generate the synthetic data. You want to use the AI ML training topic. So yeah, use that. But I'll be also be posting that in the chat. I'll be back in about 30 seconds or so and then I think we'll be, we'll be ready to go.

Sam foreign.

And thank you for joining today's Live training. My name is Rhys and I'm going to be your moderator for today's session. We're going to kick off the session in, well, pretty much straight away. We've just been waiting so everyone has a chance to join. Two things to note, today we are using Data Camp Workspace and mostly AI for our synthetic data generation and for the code log. So if you don't have accounts for either of those, please get signed up now. You'll have a short, short period to get sorted on those at the beginning of the session, but please, yeah, if you want to join in, please get that done as soon as possible. Brilliant. I think that's everything from me. I believe I will hand over now to our host for today's session, Richie. Richie, please take it away.

Hi there, Data scamps and data Champs. This is Richie. I see we got a lot of people from around the world. Good to see. Nice global audience. So today we are looking at how to use synthetic data for artificial intelligence. So that means we're making up data, and on the face of it, that sounds a little bit like cheating. However, it turns out to be an essential technique for maintaining data privacy and it's widely used in finance and healthcare. In fact, working with synthetic data is an important technique to know about for anyone working with sensitive data in machine learning or AI. And our guest today is Alexandra Ebert, the Chief Trust Officer at Mostly AI, and she's an expert in data privacy and responsible AI. So she works on public policy issues in the emerging field of synthetic data and ethical AI. And she's also the chair of the IEEE Synthetic Data Expert Group and the host of the Data Democratization podcast. So all in all, a true expert in the field. And with that, I shall pass it over to you, Alexandra.

Thank you very much, Richie, for this very warm welcome. And just watching the chat here, I have the feeling I did an entire world tour. So cool to see people join from so many regions of the world. So welcome and really great to see that you're interested in learning more about synthetic data as a privacy protection technology for machine learning and artificial intelligence. So today what we're going to cover is the question that we already popped up in the chat. What is actually synthetic data? Is it cheating? Is it helpful for privacy? We will figure that out. So first we will have a short intro section and then we will dive into the hands on part of today's training. So as Rice already said in the beginning, make sure that you both have your workspace account open as well as sign up to the most AI platform to generate synthetic data yourself. You will also find readily synthesized datasets in the workspace. So if you don't want to sign up or if something doesn't work out, you're already set to follow along, but I would say that's a good thing. To prepare while I'm giving you a quick introduction to synthetic data, maybe Rice, if you can share my slides so that everybody can see. I mean you already know the title. You signed up for this tutorial webinar on using Synthetic Data for Machine learning in AI in Python and as I just said, said sorry, I need to go to the second screen so that I can actually click along. We are going to cover three main parts today. First is the quick introduction so that everybody is on the same page what synthetic data actually is because there are different types of synthetic data. And then the hands on part we will have two different things. First we are going to answer one of the most pressing questions whenever data scientists and data folks first start to work with synthetic data. Is our synthetic data actually truthful? Is it accurate? Is it as useful as real production data? The first thing we are going to do in our workspace on datacamp and the second thing we are going to look in as a bonus exercise is Smart Imputation with Synthetic Data, which is a quite nice feature that can help you to fill missing values that you oftentimes have in real world datasets. That's more or less the outline as mentioned. Please make sure that you have both the workspace open as well as sign up to the mostly AI platform so that you're ready to go. Once we are at that part of our tutorial now I will give you a quick introduction on synthetic data. And one thing that's important for me, this tutorial is really for you you to ask as many questions as possible whenever you want. Richie and Rice will help me to moderate the chat and whenever urgent questions pops up, let me know. And there will also be some dedicated sections throughout the workshop where I can answer your questions. So if you have anything that you're curious about, please let us know in the chat. With that said, why are we actually doing this webinar on synthetic data? Why is it so interesting as a technology? And here I want to share two quotes. One comes from the joint research center of the European Commission that looked into synthetic data for over two years and then last year published a report where they concluded synthetic data is becoming the key enabler for AI in both business as well as policy applications in Europe and also in many other parts of the world. And also Gartner, for example states that already next year 60% of all AI training data is not going to be real data, but synthetic data. So with that quotes we can already guess it's a quite impactful, a quite important and also quite hyped technology at the moment. But to better understand why synthetic data is needed, I always like to give a little bit of context on why organizations are currently interested in replacing their production data with anonymous synthetic data. The problem that many organizations, particularly large organizations, face is that they have significant amounts of data, significantly more data than they had five years, 10 years, 15 years ago. But due to stricter privacy regulations and some ethical and regulatory risks that come with using this data, most often organizations are not in the best place to actually use this data or make it available to the data science teams, AI teams to use for machine learning development or for other parts of data driven innovation. So we have this kind of data fuel crisis where it's a mountain of real world valuable data, but organizations only use a tiny fraction of this. And this is actually something that's not going to get easy in the future. Many of you will, I assume, have heard of the European General Data Protection Regulation, in short, GDPR or California's Consumer Privacy act, or one of the I think 120 plus different emerging privacy laws that we have around the globe. And this is something that is only going to get more complex. Alone in the European Union we have this, what privacy pros fondly call tsunami of new regulation that is upcoming, the AI act, the Data Act, Data Governance Act. And this is actually a similar picture to many other parts of the world. So particularly for large organizations, but of course also for medium enterprises, startups, SMEs, it's not going to get easier to use privacy sensitive real world data. And this is why synthetic data is so interesting. But once you kind of state this problem of having this challenge of having treasure troves of real data, but not being able to use this data while complying with privacy laws, oftentimes the question pops up, well, why not just anonymize your data? And in theory that's a great suggestion because those of you who know GDPR and other privacy laws in more detail might are aware that there are specific sections in those laws that explicitly exempt anonymous data. But the problem is legacy anonymization technologies, I'm referring to masking, obfuscation and the like, they simply don't work in the era of big data anymore. As you can see here on the screen, I think it's pretty obvious that this data is not sufficiently anonymized. Of course, with the unstructured data, the image here, it's super easy for us humans to see because we are visual creatures. But the same holds true for structured data like the financial transactions that you can see here. By the way, our tutorial today is focusing on structured synthetic data. So we're not going to create images, we're going to create anonymous synthetic structured data. But here we can definitely say not enough information was deleted. The privacy is not protected here. But what happens if I delete more information? Is this now anonymous? And I can't see anybody now, so I can't make a raise of hands. But you will be surprised. This is actually still not anonymous, even though the majority of the value in this data was destroyed. Because all of these legacy anonymization technologies are quite destructive in nature. And researchers actually figured out that regardless of how much you delete with traditional anonymization technologies, if there is a tiny bit of real data in the supposedly anonymous data set, then you still have this reidentification risk that your privacy of your customers, your employees, your citizens is not sufficiently protected. And this of course brings you in conflict with GDPR and other privacy laws. To make this a little bit more concrete, let's look at a specific example. One study, for example, found that with credit card transactions, it was sufficient to have three credit card transactions per customer. And if you think about your own credit card transactions, then I would say we can assume that every customer of an organization will have at least a few dozen credit card transactions of a bank. In that context, at least a few dozens of credit card transactions I would say are reasonable. Most often you will of course have hundreds of transactions per customer. And if already three out of these hundreds of transactions are sufficient to re identify over 80% of your customers, then you can see this destructive nature and why traditional anonymization technologies are not fit for purpose in the era of big data anymore and don't give you the data quality that you as a data scientist want to have to develop your machine learning model. And they also don't give you the privacy. And the interesting part about this study was actually not even the entire information was needed to re identify these 80% of customers. Just the date of the transaction and the merchant, nothing more. So here you can really see the limitations of traditional anonymization. And this is something that not only holds true for financial transaction data, you can see the same for health care data, demographic data, and any other type of behavioral or time series data, which is so unique and rich that it's really hard, if not impossible, to anonymize with traditional anonymization technologies and to kind of highlight what does this mean for organizations. We already talked about today, organizations don't only have a handful of attributes per customers, but they have hundreds, if not thousands or 10,000 of attributes per individual customer. At least if you're working in an enterprise context. But with these legacy anonymization technologies, you have this hard ceiling. You can't retain more than a handful of attributes before entering into this reidentification risk. And this brings up this tension between wanting to utilize data and having to protect privacy. And to solve this tension, this is why we have AI generated synthetic data for privacy protection. So now to the interesting question, what is synthetic data? Why do we need it and how can it help with privacy protection? And even though I already said we are going to focus on structured synthetic data, think financial transactions, healthcare data, telecommunication, mobility data and the like, everything that fits in a table. I like to explain the concept of AI generated synthetic data with images because it's just easier to comprehend. The specific images you see here, as you could have guessed, are not real people. And in the era of Dall E2 and Midjourney and so on, it's not surprising to see the stunningly accurate photos that were made by machine learning algorithms. But a few years back, this is already a little bit of an older study from Nvidia. This was really fascinating to see how good you can get with AI generated synthetic data. And what it did in this research project was to train a deep learning algorithm on plenty of really human photos. Up to the point where this algorithm really understood how does a human face look like something like, okay, humans have two eyes which are roughly positioned in the middle of the face, mouth, the nose, these hairstyles, these skin shades, and so on and so forth. And then once everything was learned about the patterns, the structure, the correlations in this data set, you could use the deep learning synthetic data generator to create new artificial synthetic images from scratch, like the ones you can see on screen. And all of those people have never existed before. And it's not a kind of very simple process where you just take the pair of eyes from training sample A in the data set and the mouse of training sample B in the data set and just shuffle it together and say proudly, okay, this is now my new face. It's really generating and drawing these phases from scratch based on the statistics and the patterns that were learned from the original training data. And this is the same approach that you can use for structured data to make sure that you get a privacy preserving synthetic replica of your real world production or customer data like financial transactions. And we at most AI have actually developed a platform that by the Press of two or three buttons allows you to create a highly accurate, highly statistically representative synthetic replica of your original customer data set that it doesn't have any privacy sensitive information in there. And again, the process is the same as what I just described with the Nvidia example. First, let's imagine a large bank has already huge data set, let's say 20 million customers in their financial transactions. But of course, this is highly privacy sensitive data that's not free to use under GDPR and other privacy laws. So if they want to anonymize it with synthetic data and unlock it to make it freely shareable, usable for machine learning, shareable on cloud resources, with startups, and so on and so forth, the process would look like that. First they have the original training data and then they put it into a synthetic data generator in our platform in this example and here, the deep learning algorithm that's part of the synthetic data generator is capable to automatically learn all the correlations, the patterns, the structure of the entire data set. So to simplify, the algorithm basically understands how an organization's given customer base act and behave. And then again, in a completely separate step, once the training and the learning was completed, you can generate an arbitrary number of new synthetic customers and their synthetic financial transactions. And if you look at those two data sets, the real world privacy sensitive data and the anonymous synthetic data, from a statistical point of view, they will be nearly indistinguishable, but they will be privacy safe. So why nearly indistinguishable? Of course, if you want to protect privacy, you will never be able to retain 100% of the information. That's simply not possible from a privacy point of view. But in contrast to legacy anonymization technologies where you stick with the original data and try to delete and distort and shuffle around those parts of the data that you deem to be re identifying, say a Social Security number or your last name or something like that, and then end up with what I like to call a Swiss cheese of data. Where from your, let's say 200 columns, you only have like 2, 3, 4 columns per customer retained. With synthetic data, you don't touch the original data, you only learn the patterns, the correlations, the distributions, and then create a new synthetic data set from scratch where you again have all the 200 columns populated with accurate synthetic data. But there's no one to one relation between any synthetic customer and a real world customer, which is how you protect the privacy. And to make all of this privacy protection happen, it's actually super important that you not only have a powerful deep learning synthetic data generator, but actually also powerful privacy mechanisms in place that make sure that everything that this deep learning algorithm learns is generalizable statistical information and nothing that falls into the realm of personal privacy sensitive secrets. So let's for example, think of a data set where you have, we said 20 million customers and there's one Bill Gates in the data set who has a significantly different amount of income, spending and so on than all the other ones. This person, for example, wouldn't get included. So the extreme, extreme, extreme outliers you wouldn't find in your synthetic data set to protect the privacy. But in contrast to legacy anonymization technologies, you can retain the majority of data. You can retain basically the entire distribution in your data set minus the extreme outliers, which is something that gives you significantly better accuracy. So to kind of visualize this, of course not a highly scientific graphic, but just for your impression, in contrast to this legacy anonymization technologies where you can. Oh, sorry. Where you can only retain this handful of attributes before you enter into severe reidentification and privacy risk, you can finally grow with the amount of data that you collect. And although you will not get 100% of the information that you can retain, you will actually get near perfect data, which is as good as your production data and which many organizations already today use to develop and train the machine learning models on, because it is a very powerful replacement for the data. To kind of sum up what the benefits of synthetic data are first, I think, and I hope it's obvious for now it's eliminating the privacy risks. So once it's synthetic, there's no way back to the original data. You can't re identify your customers if synthetic data is properly synthesized with all the privacy mechanisms in place. Another benefit of synthesizing data is the speed. We work a lot with large organizations. We work with some of the largest US Banks, some of the largest European banks and insurance providers, and they all tell the same story. If they want to access data, it's something that takes them ages if they're quick. A data science team sometimes gets the data within a matter of weeks. But much more common, we hear time spans from like three months, six months or even eight months until they get their data. If they, for example, want to externally share it or put it on the cloud or something like that. So this is something that's super cumbersome if they rely on legacy anonymization technologies because they have to go through this case by case anonymization process with synthetic data, I mentioned it already, it's the click of three buttons, it's fully automated and it's something that can speed up data access from weeks or months to a matter of a few hours, or business data or some organizations even provide an internal synthetic data Lake Hub Marketplace, however they call it, where teams can proactively access synthetic data without having to go through this lengthy data access processes. So it's really a tool also to democratize access to data and speed up data access. Then accuracy of course is another benefit. Because if you cannot only retain three attributes per customer, but can retain all the 200 or 500 or 10,000 attributes that you had, then you of course can be much better in personalizing for your customers, understanding what you really want, and not only developing tools that cater to the average chain and John Doe, but the full diversity of your customer base. So it's also quite an interesting technology, not only to help you personalize and foster customer understanding, but also to be fair and more inclusive because you can finally also see who are the minority groups in your data set in your customer base and what types of services a product they might enjoy. Another benefit and another reason why many organizations turn towards synthetic data is the collaboration aspect. There's so many large organizations out there that want to collaborate with startups or smaller partners, but of course data sharing externally is oftentimes a challenge. So with synthetic data again, this is something that can be significantly accelerated and just speeded up to make sure that they are much faster, not only validating but also collaborating with external partners. And then there are also some other things that you can do with synthetic data to not only replicate the existing data in a privacy preserving manner, but actually use the power of generative AI to improve the existing real world data. One of these examples would be smart imputation where you fill missing values. But there are many, many more and I can actually point you to some of these areas later on when we are in our hands on tutorial. The last slide we're going to cover is the use case, what synthetic data is used for. And then we are off to our workspace to get coding and get in the hands on section. So use case wise, you can really think of synthetic data as an enabling technology. The entire purpose of synthetic data is to anonymize data in a manner that still retains the utility of your original data, but protects your customers or your employees privacy. And the main use case that we see for synthetic data today, regardless of whether it's Healthy care, banking, insurance, retail or even public sector is machine learning because no other use case requires this sophistication of data and this granularity of data. But it's also used for analytics, it's used for digital product development, for external data sharing, as I mentioned earlier, collaborating with startups, AI vendors and so on, even open data sharing, and increasingly also for responsible AI aspects like AI governance, AI fairness and explainability. So one takeaway for you, synthetic data is an enabling technology and it's super interesting to make machine learning in a privacy preserving manner possible. With that said, I think you have a quite good, or I hope you have a quite good introduction to synthetic data and maybe we want to take a few questions now if there are some urgent questions to answer. Richie, I'm not sure. I couldn't monitor the chat and then we can actually, actually enter into the workspace part.

Sure, yeah. So we have a few questions from the audience already and for anyone else who wants to ask a question, please do that now. So first question comes from Karan saying, could you encrypt data on a bit level to anonymize it instead of creating synthetic data?

So basically the GDPR for example, is quite clear that things like pseudonymization or encryption don't count as anonymization. So as long as there's a possibility to come back to the original data, it doesn't count as anonymized and out of scope of gdpr. Therefore the answer in that case would be no. But of course maybe there are some other jurisdictions where there's a different answer with gdpr, which is kind of the strictest one. It's not a possibility.

I suppose once you start publishing the results, you don't want to publish something encrypted, you're going to want to publish like a real number as well.

That would be another thing. And I think one interesting aspect, maybe some of you are familiar with the whole kind of set of emerging privacy enhancing technologies, homomorphic encryption, secure multi party computation, federated learning in some context, what one benefit of synthetic data is is that the output is something that can not only be digested by machines, but actually also digested by, interpreted by, analyzed and used by humans. And I think that's a super not only for the usability, but also from a fairness responsible AI perspective would really make sense if we have humans being able to take a look on the data and for example, checking if it's an adequate representation from a diversity point of view.

All right, fantastic. It's thoroughly answered and so Next question comes from Dipanka saying, how do we decide which model is best for a particular data set and business use case, e.g. single file structured data set, you can either use copulas or.

That's a great question. So I think this is particularly interesting if you want to build something yourself or make use of the open source tools. For example, synthetic data role is one of the most commonly used open source libraries where you have different models available. With tools like mostly AI, you already have a variety of different models inside the product and it automatically automatically picks the best model depending on the data structure. So for example, time series data needs some other elements as if you only had static data. But this is the nice thing about tools like ours that this completely works out of the box. And I also quoted for example the Joint research center from the European Commission earlier, they actually compared open source solutions with our solution and they found that with open source solutions, at least to date, there's quite some pre processing that needs needs to go into the data. I think they figured it was two person month versus again a few click of a click of a few buttons and you have some data that's super, super close to your real data so you can build it yourself. But I think it would also be interesting to try out tools like mostly.

All right, we've got, we'll take maybe one more question.

I would say one more question and then let's synthesize some data, dive into it. Yeah, exactly.

All right, so what's different between synthetic data and scrambled data?

So basically with scrambled data, my understanding would be that you just take your data set, make this move and then have some nicely shuffled around data. So with synthetic data, as I mentioned, it's not related to legacy anonymization technologies where you either shuffle around data or delete some parts of the data, but always have the real data. With synthetic data you learn very granular the patterns and the structures of the original data and then create new data from scratch. Think back to the images. If you were a super skilled artist and you would learn how to draw a realistically looking human hand, you would be capable to draw something where every human being would tell you, yes, this is a human hand, but it doesn't have to. It will not have existed before. And this is the same thing that you can do with structured data. So you will really create new data points from scratch which statistically are representative, but you don't take the original data and just shuffle it around. And this also brings another benefit, which means that you're not tied by your original input. So we have customers who, for example, upload 2 million, 5 million, 10 million, whatever customers, and they maybe create 10 million customers or just 100,000 or just 100,000 college students if they want to do a certain analysis or product development on them. So you're not tied to your original input size. You can really scale this up or scale this down because you don't use the existing data pieces. So there's quite some differences to legacy anonymization and shuffling and scrambling around data. Perfect. I would say we go to our workspace and get our data set. So I hope that everybody managed to. Oh, we are still on the slides. I think I need to share my browser so that you can see something. So here we go. Perfect. Let me make this a little bit bigger. I hope you can see it now. So this is our workspace that we created for today's webinar. And the first thing we are actually going to do is download our data set. We are today working with the adult census dataset. I think many of you already know this dataset stemming from the U.S. census. And what we want to download now is our training sample. So as you can see here, we've already split the data set into a holdout sample and into a training sample. Plus what we also prepared in case something doesn't work with you during synthesis or you don't want to go to the platform, are the readily synthesized datasets that we're going to use for the first part and the second part of the tutorial. But for now, the only thing that you need to do is download the file and go to our platform. If you manage to successfully log in, you should see a screen that's quite similar to this one. Let me know if it's not big enough. I can definitely zoom in and you can then synthesize your data set. So it's, as I mentioned, just a three clicks process. You basically drag your file here and you of course can also use other data sources, but for ease of use, we just use the CSV file and you don't need to change anything. So it's automatically detecting what variables we have here and what data types we have in here. And it's automatically in the background applying all the perfect combinations to make sure that you get the best accuracy while having the strong anonymization. So we're basically launching the job and it tells me this is our free.

Version a little bit just for people on small screens.

Yes, of course. So I think now we can See that it's in progress. But basically what I just did, I uploaded the file and then pressed the launch button, which we can't see here anymore because I already launched a job. I'm wondering now it's actually not okay. This shouldn't happen. So I need to work with the small thing. But you're not missing anything here. Basically what's currently happening is that we see the pending status now. It's retracting and putting this in here. Now I should be able to zoom. So this is now going to take just, I would say like 2 minutes, 3 minutes, something like that. So while we are waiting, we can maybe take one or two more questions or I can also walk you through maybe just in the interest of time, I will walk you through what we're going to do with our synthetic data set once we have it. So let's give the platform a little bit of time to synthesize all of your data sets. But basically once we have the synthetic data set, we're going to download it and then we want to answer one of the most pressing question that everybody has when they start out with synthetic data. Is the synthetic data actually accurate? Is it accurately reflecting the patterns, the structure, the correlation that I have in my real world data? Because if it's not truthful, if it's not accurate, if it's not high quality, then of course you wouldn't be able to use it for analysis, for machine learning and so on and so forth. So one way that's very commonly done by organizations to evaluate synthetic data quality is actually to perform train synthetic test read. This is an approach that was referred to in multiple papers. And here you can see the setup. I'm going to zoom in a little bit. I hope that it's large enough. Remember, we already have our census dataset splitted in our holdout dataset. This is not touched by the platform where we currently uploaded the training sample to synthesize. And this is also not going to touch later in the machine learning process just to evaluate. So basically we have our holdout sample which we keep for later and we have our training sample, the one that we just downloaded and put into our platform to anonymize. Then as mentioned in I hope two minutes or something like that, we should get our synthetic data set and then we want to know how good is the synthetic data set, how accurately is it reflecting everything that exists in the actual training data set. And to figure this out, we are actually going to train two machine learning models. One machine learning model on our Actual training data and one machine learning model on our synthetic data. And then we want to see how good those models perform by scoring them on the actual holdout data. So data that was neither seen by that model nor by that model new during the synthetic data generation process. So that's the setup of Trane synthetic test reel and you can find additional information in this notebook. So we already went through this process. Let's see how we are doing. Okay, this looks a little bit slow currently, maybe because we have so many people on the platform. Okay, I think I will proceed because usually this should already be much further along. But sometimes if you have so many people on the platform at once, it gets a little bit slow. So I'm going to show you how to download the data once you have it. But at this point in time, if your job isn't finished, maybe you're more lucky than I'm and your job is already finished, then you can definitely proceed with the data that we have in the workspace already preloaded. So I will show you also. But basically what you can do once you have successfully finished your job, you just have to press this button and click on download your CSV file. And then you need to unzip it. And to make it work in the workspace, you also need to rename it just to kind of demonstrate you. What you can also see on the platform is an overview of the accuracy of the data set. You can also see first kind of QA report where you can assess the quality of your synthetic data. It's just the very first kind of sanity check, how close you are. So here on the left is our original training data. On the right is the synthetic data set that we just created. And here we have the overall correlation matrix where you can see that just from a visual point of view, it quite accurately captured the correlations that we have in the data set. You can also look into the univariate distributions. And here I hope it's big enough. Here you can see, you can hardly see the gray line which represents the real data because the synthetic data so accurately matches this data. You can also do this for the bivariate distributions. You can also look into some privacy checks, but these are just like on top because the actual privacy protection already happens during and prior to the generation process. So these are just some distance measures to kind of be extra sure that the result is anonymous. But of course, all of this, if you're a data scientist, is just the very first sanity check. It's much more interesting to do this train Synthetic test, real evaluation that I just explained earlier, because machine learning is so sensitive to all the deep underlying patterns that were captured. So let's maybe open the data set that we downloaded. If I'm just going to demonstrate it here. Let's see. Okay. It's currently in progress, but not finished. I want to have enough time for questions and also for the smart imputations. So I'm just briefly showing you what to do. So basically you just unzip this file. I'm doing this now on my second screen. You can't see it, but it doesn't matter because if you haven't yet finished your job, you can definitely proceed with the data set we've preloaded in the workspace. So basically two things to do now, if you're one of the lucky ones who already have a finished synthetic data set, you only need to unzip it and then you need to rename it. Because to make the workspace to make our notebook work, we want to make sure that it has the synthetic data set in there. So basically now you have renamed this file and if you have your synthetic data set already ready, you would just go here to upload into folder and upload it into our Data Camp folder that we created for today's tutorial. If you don't have your synthetic data set yet, then this is absolutely no problem because as mentioned, we have also preloaded it. So here you will find the sensuous synthetic demo data set and you can definitely just proceed by renaming this one. I'm not going to do that because I just uploaded a synthetic data set. But if you just delete the demo part and have Census synthetic CSV, then you can proceed with this workbook. So now we have our real data set there, we have our synthetic data set there, and we have our holdout data set here. And this brings us actually to first importing our data. And then one thing that you can do with synthetic data is just explore it. And here you can see already from the structure, it's the identical structure than the real world production data. So in contrast to legacy anonymization, you don't get this Swiss cheese of data set where you have all these gaps and holes and can only get, I don't know, two, three, five attributes, but you get all the attributes that are available in the data set and all of this information is filled out and you can just generate random samples and already get a feeling for the data uploaded to exploration tools, analyze it, and so on and so forth. Then you can definitely also make use of the nice AI feature that we have here and for example, let you let the AI show you randomly sample, in that case, women of age 30 or younger that have a master degree or, or a professor or something like that. So whatever you want to see from your synthetic samples. And this is by the way, something that's also quite interesting not only for data analysts, but also for product development teams who sometimes want to have super realistic data to populate their products with. So this is something that you can do or you can even plot something. So what I prepared here is a plot where we can see the average age depending on the marital status and the gender. We want to sort this from lowest to highest and also label the average age. So let's see if this works. It jumps around a little bit. I think it takes a little bit until here it is. Wonderful. Here we have the plot, maybe. We ask it to color it a little bit less stereotypical, but it's already super nice. You can see the average age of people that never married, that are separated, that are widowed, and so on and so forth. Usually people that are widowed are already older. And it's basically possible for you to explore the state on a quite granular level. Look into individual subjects or individual records, or also, of course, look at the overall statistics and for example, get things like average age of widowed people versus married people versus whatever. So quite nice, particularly with this AI tool that you have here in workspaces. But this just as a kind of first exploration. What we wanted to do was doing our machine learning. So this is already prepared. We are basically using a small and fast light GVM classifier. And what we want to do here with the adult census dataset, again, I'm assuming that most of you know it, but we have different attributes in this data set. And one attribute is the binary column income, where you can see whether target or whether records have actually a lower income of below $50,000 per year or above $50,000 a year. And what we now want to do is let our model predict if a person is a high earner and earns above 50k. So this is basically the code I'm just scrolling through here because today it's not about creating the model, but actually figuring out if synthetic data is good or not. And then we're going to do the first part, what I explained earlier. We're going to train this light GBM model on our real data and we're going to test it with the real holdout data. So if you scroll here and maybe zooming in a little bit so that it gets higher. So basically we can now. Train our model on the real data that we also use to synthesize data and then we want to also evaluate our trained model on the holdout data. So you can definitely follow along. I hope everybody is here so far and we didn't have any problems. But if there are some big problems, then I assume Richie would have warned me. So I hope everybody can follow along here. So what we're going to do, we train the model on the real data and we want to evaluate our model with the code that was visible briefly above with the holdout data to score and evaluate the area under. Oh, sorry, what did I do wrong?

I think it's model TRN instead of model Tran.

Thank you. I think it was model train. Ah, here train. So. Talking and coding is a challenge. Okay, now we have our accuracy, we have our area under the curve and we also have the plot where you can find the code above. Just a brief explanation of this plot, but I would say from the area under the curve and the accuracy, these are quite nice numbers. But of course the main thing that interests us here is comparing the accuracy in the area under the curve with the model that we're going to train on synthetic data in a bit. But just here what we can see is actually all the holdout records that were scored and how they were classified versus what they actually were. So here we can see in that case, the closer to zero, the more certain our model is that this is a low income person versus the closer to the one, the more certain our model is that this must be a high income person. And here you can see what the holdout records actually were. So here with the high income persons the model was super sure that they are high income. And we can see with the orange cap color they actually also wear high income individuals. So they are classified quite nicely. Also here on the other end of the spectrum with the low income folks, we see that the majority of real holdout records it predicted to or it classified to be low income actually were low income, which is also good to see just a few high income. It misclassified as low income. And here in the middle we can also see more or less a 5050 split. So this is just to visualize the performance of the model a little bit. But the interesting part of course now is can we meet these numbers with our model that was trained on synthetic data and we of course haven't trained it yet. So let's do that scrolling up A little bit. So we are going to do the same thing again, our synthetic model. We want to train the model on synthetic data this time and we want to also evaluate the model again on our holdout data set, Execute that, and here we can again see the scores. And I'm not sure about you, I definitely didn't remember the numbers, so let's quickly scroll up a bit. But our accuracy is, in that case, 88.6. And here we have 89.1. So we can say that's pretty close. That's quite nice. And our area under the curve is 93.7. And here we have 93.8. So we can say that the performance is basically on par or super, super close to each other. So that's already our first kind of part of the tutorial. And the main thing I wanted to show you here is a valuable approach how to evaluate synthetic data quality. Of course, this becomes much more interesting if you test it with one of your real world data sets or if you do it with a data set that you work a lot with, because then you can see how close you can actually get with synthetic data. So that's just the approach and you can definitely recreate this with your own data sets. But I would say we continue with our next part by synthesizing the smart imputation data set because this again will take like two or three minutes hopefully. And in the meantime I'm going to take questions. So just as a transition, the second part I want to show you is how to close gaps in your data set. And what we actually did with the adult census data set that you already used and that we already uploaded to the platform, we semi randomly included some missing values to reflect something that we can quite often see in real world scenarios where you have a data set. But unfortunately they're missing values in there. And this is something that on the one hand can impact when training a model because you need to prepare the model to make sure that can handle missing values. But it's of course also something that you always have to take into account when doing some analysis because it's hard to know which customer segments, for example, didn't provide certain information. Being this, I don't know many different attributes or one thing that we also know from organizations is once they start to capture new attributes, but of course have 10 years or five years or whatever history of customer data, then some part of the customer base will have this information, others will not, or all will have this type of information just from a certain point in time. So Smart imputation is something that can help you to fill this out. And in contrast to more basic imputation methods, you can get much closer to the kind of ground truth of data. I'm going to explain this in more detail in a bit, but I just want to get the synthesization synthetization started so that we can continue. And then I'm going to take questions both on the smart imputation, if you already have some, but definitely also on our train synthetic test, real accuracy evaluation that I just showed. So what we need to do to make the smart imputation happen is basically just change one tiny bit in our platform and we again are going to upload the data set that we already used for the first synthetization run. And then you can see here all the different settings and already when you upload the data set, you are in your data settings and what we actually Sorry, I didn't mention that. The column where we changed the missing values was the last column in the data set, the age column. We also sorted it to make age become the last column of the data set, because currently that's the only way you can use it on our platform in the future. It doesn't matter in which order the columns are. You can then be more flexible in smartly imputing the data. But here with this data set, you have some missing values in the H section. And what we want to do is select now this column and activate the smart imputation, press save and again launch our job. So in the meantime, also our first synthesization finished, I hope for you too. So you can definitely also perform this with your synthesized data set later if you currently just use the prepared synthetic demo data set to see if there are some differences in the predictive performance of the models. But basically now we're going to synthesize the same data set, but we task the generator to fill out all the missing values that we have in the H column. So I'll let the generator do its job and we'll take a few questions now maybe Richie, if you can shoot a few at me until we have our.

Sure. We have absolutely loads of questions from the audience. It doesn't sound to get through.

Okay, maybe just as a kind of advanced note, I'm super happy to to answer your questions. So you can also follow me on LinkedIn, send me a LinkedIn message, you can even reach out or just go on the Mostly I website and put your question in the chatbot there so you will definitely get an answer. Since we only have nine minutes left and can't answer everything. I think I will answer two questions now and then proceed with a smart imputation.

All right, perfect. So next question comes from Naz saying, how can AI vendors differentiate themselves if they use the same puts? If you think of the same element, basically, how do you create different outputs given the same inputs?

I mean, in that case the input would be different because synthetic data in this scenario is not generated out of thin air and you just, just tell a generator, please. Or maybe let me start differently. If you use Mid Journey to generate your marketing images, it's quite likely that everything is going to look more or less the same or will have the same style and that things are going to get repetitive. In this context of generative AI and in this context of synthetic data, what you do is basically not create synthetic data out of thin air, but you use your proprietary experiment existing data as an organization, which means that the synthetic data you work with is specific to your organizations. The results from analysis that you get is specific to your customer base. And therefore I think it's not that much a question about differentiation different input because it's just the data that you already collected. So I would say this, this is not that relevant in this context of synthetic data. But maybe I got your question wrong. So if you have a follow up question again, just go to our chatbot or go to my LinkedIn and I'm happy to also expand on that later. Can you give me a second question, Richie, before we, in the interest of time, need to ask?

I can. We'll move on to Lakshmi's question. So what sort of data sources can you use with mostly AI?

Perfect. I mean, the process of generating synthetic data is something that you can use for multiple different data sources. So you can definitely use it for unstructured data sources like images, videos, audio and so on. But with mostly AI, we focus on structured synthetic data. So you can have categorical data, you can have numerical data. And what we actually excel in is time series and behavioral data, even things like mobility data, financial records, something like that. Because if you build your own synthetic data generated, it's quite easy to get good results for static data, like, I don't know, customer demographic information. But, but it's really hard to have synthetic data that accurately reflects time series or behavioral data. So everything that's time series you can definitely synthesize. And with our platform you can also do short sequences of text so you can't write your next synthetic novel with it. But if you have something like customer support requests or the type of free Text that you sometimes have with financial transactions. This is definitely also something you can synthesize size. Perfect. Then I see we have loads and loads of questions. Again please go to our chatbot or to my LinkedIn to come back to get an answer to that. But we are actually proceeding now. Again, I think our platform is struggling with all the people that are currently on it. So it's not as fast as usual. If you visit it at another time, I promise this will be much faster. But I already prefer prepared it a few hours prior to our webinar. So if I go to my jobs page, this one was the first basic default setting job that we already did that already finished and where I already showed the train synthetic test reel. This one is where I did the smart imputation to populate the age column. And if you look here into the QA report, you should also be able to see this somewhere here. So here you can see the synthetic data. I hope it's large enough. But it's just as mentioned first visual exploration. The synthetic data did not accurately reflect the gray line, the real world data, but it actually smartly imputed the values. And we are now going to look into did we manage to come closer to the ground truth which we have since we deleted the missing values artificially. So we're going to download this data set. You I think will have to wait for your job to finish a little bit or maybe in the interest of time. Of course it would be the same process downloading the data set, unzipping it and renaming it to Census syntheticimputed. But in the interest of time, let's all proceed with this dataset here, which we already preloaded. So we are just going to delete the demo part. So I hope. Yep. And it should then say sensors synthetic imputed CSV. And what we're going to do now here you also have the visualization. So you read our data and we actually want to. Ah, here I'm going to load that. So here we can already see what we also saw in the small window that I showed earlier in our data QA report. But here the gray line again reflects the original data which was the data set we used to synthesize and where we have artificially created some missing values for different subjects in the H column. And this is what our synthetic data just by the press of a few buttons. And now the interesting question is how does the real data look like? Is the synthetic data closer to the ground truth than the data set with the missing values? And to figure this out we will actually plot this, but we have or maybe since we don't have that much time left basically here, I think this should be easy to do for everybody. You need just to compute the average age for the training data set that we uploaded, the synthetic data set we imputed and the raw data set. And what you will see see here then, is that for the training data set, the average age will be 35.7 years. But the synthetic imputed data set will give you an average age for 38.5 years, already three years older than the average age. And if you use the raw data, the original adult census data, where we haven't artificially deleted some age values, then you will see that it's also 38. So much closer to what the synthetic data imputed and therefore much more accurate. If you want to do analysis on this data. And if we're going to plot this, we can also visualize it. So here you can see the red data, which is the ground truth data. It uploads this from our GitHub library, by the way. If you're interested to explore synthetic data more, you will find all the different interesting things you can do with synthetic data, from explainable AI to conditional generation, which means just creating multiple records with certain characteristics, how to do it for multi table, how to rebalance data also in the context of fairness. So definitely check that out. But here you can see the ground truth data and that the synthetic data is much closer with the imputed values in contrast to the missing values. So this is something that works quite nicely for many scenarios. It doesn't work for every scenario. So for example, in our data set we have the different education levels. If we had deleted the age of every professor, then the synthetic data generator would have learned it's a pattern. Professors don't have an age. But if just some professors have omitted to provide you with their age or their income or something like that, then the synthetic data generator is capable to much better pick up some underlying patterns and just by the click of a few buttons help you to impute your data set and get rid of missing values. It's a new feature that we have and I'm super curious if you test it for your datasets that you work with a lot, which results you get. So if you have some feedback, please definitely let me know. I see that we only have one minute left, so I think I will take one question if we have time, Richie, and then please, please reach out to our chat and reach out to me on LinkedIn because there's so many questions and I definitely want to answer them later.

All right.

Yeah, thank you Alexandra. That was really amazing. There were a few people asking a variation on a theme of discussion. I'll post one from Matei saying are there any proceed concerns with uploading data into the mostly AI tool?

So our tool is set up in a way that your data is safe and secure. Most of our customers actually don't use our online version but use it on site. But you can definitely check out all the safeguards that we have in place and of course if you're part of a large organization, also make sure whether you're even allowed to upload your privacy sensitive data. In that case, I think think it's better to get a local version of our tool than use the online version with kaggle data sets and something like that. But it's definitely a tool that you can use with privacy sensitive data to anonymize it either on prem or in the cloud.

That would be sort of bad if the tool itself wasn't privacy skew.

Sure, that would kind of be around the purpose. But as mentioned, particularly for large banks, insurances, which some of the heavily regulated industry industries, they do this on prem or in their cloud environment. So this is a difference I think between smaller players and large players. But of course we also set up a tool that allows you to anonymize your privacy sensitive data because otherwise there wouldn't be a purpose in even offering this tool.

All right, perfect. Thank you once again, Alexandra. Before you all jump off, the next webinar we have is tomorrow. That's a data Camp for business session. So if you're interested in adopting Data Camp throughout your organization, then please do join us tomorrow. And then the next live training is on Tuesday. That's on Data Modeling and SQL. You can go to datacamp.comwebinars to sign up for any of those. So thank you to everyone who asked a question. Thank you to everyone who showed up today. I hope to see you in future webinars. And yeah, thank you once again, Alexandra.

Thank you for having me Rich. And thank you for all of your questions. Really cool to see all this interest. Happy to answer them later.