Analyzing Data with SQL & Python | How to Analyze Survey Data

So make sure you got this open. And we're going to start off going through. You've got links to the paper that describes the data set and the data set itself. If you want to go away and read more about this, we're going to start off by looking at questions with a numeric response and visualizing those with a histogram. We're going to do a little bit of stats for the Mann Whitney U test and and then we're going to look at visualizing questions which have categorical responses using power plots. So the first thing, we need to use some python packages now because we're doing all the data analysis in SQL, no python package needed for that. We're just going to use the Plotly Express package. That's for drawing our histograms and bar plots. We also need the MannetWin EU function from SciPy Stats. That's for performing the hypothesis test. So first thing, let's get started. We're going to do import plot the express and the common alias for this is PX and next we're going to import the manu function from SciPy Stats. I'm copying pasting here because I'm a terrible typist and better to not make mistakes so it's not going to autocomplete for me. Okay, copy and paste then. So here we've got a whole package and then just one other function hit play to run that. So those are correctly imported. Now the next task we're going to import the two data sets. We're going to, if we have a look at the files, the survey responses, these are contained in the Survey_Data CSV file. And then we've also got a Data Dictionary in SurveyQuestions. We're going to import the data first, then the questions. Oh, this is actually out of date. I shortened the name of this file because it was ridiculously long. So the survey data is actually contained in a CSV file named SurveyData. And the details of the columns, basically we've got one column of zeros and ones which means is the current is the the company currently classified as a growth company. So there's an oecd. So this is like an international organization which define gives you an official definition of what constitutes a growth company. And some of them are, some of them aren't. And then each one of the questions, questions are split into different parts. I'm not, not sure whether this is like slightly dubiously translated from Finnish. So question question is like an overall set of questions. It's really like a section, and then row means like the individual question within that. Okay, so we've got lots of questions and then the details of those are described in this data dictionary file. So next task, we're going to import the data set. So we want select everything from this file. So we're going to do select star to select everything and then from. I'm just going to show you what happens if we try and import the data set as is. This is going to go wrong because the data set is in a slightly different format to what we want. So usually in workspace you can just import data sets from a CSV file by. In the from clause, you use a string naming the file. In this case, it's not correctly detected that you've got commas for decimal places. So it thinks these are text columns instead of numeric columns. So we have to do a little bit of cleaning up in order to make this work. There are some code hints here to explain how to do this. And then I'll drop down. So because this is what we have to do is call DuckDB, that's the database that we're using. And we have to call read CSV Auto, that should be a lowercase 8. And we have to set some parameters. So first of all, the column delimiter, it actually got this correct, but I think we have to set it explicitly now. So that's a semicolon, the decimal separator. So the thing that says this is a decimal point that is equal to, that's a comma. So that's these commas here. And then also scrolling to the right a little, we've got some missing, some questions with missing values. And these, they've got a space in them. So the value that we want to be, that we want to consider is null, the null string. And null strut in this case is a space. So these three arguments together. I'm going to run this again. You can see now in question five, part nine, 10, these missing values, they're correctly denoted as null. And then in question two, where we've got numeric and says numeric responses, you can see these are correctly described, these are correctly numeric columns and they've got a period as a decimal point. All right, so that's the data imported. One thing you need to do is, if it's not correctly copied for you, is make sure that the results of this are saved as a data frame named survey. We'll be using this later. Next we're going to import the data dictionary. So we're going to do something very similar. So we're going to do select star from. Let me just copy and paste the name of this file. Now in this case, this uses standard like conventions for CSV files. So that means each column is separated with a comma, you've got a period for the decimal place, separator, things like that. So we can choose default settings. That means we don't need to explicitly call, read CSV auto. We can just pass in the file name as a string. Okay, so if you notice, it's not super helpful just having the name of the question as the column, as the column header. So we need a lookup here to determine what does. You've got the column in the original data set and it tells you what was the question that was actually asked here. So we can read like what, what were the people asked? If you want to know more about this in the files, there is a PDF here which has all these sort of complete details in more depth. All right, so we have imported the, the main data set, we've imported the data dictionary and now we're going to start drawing some plots. So first of all we're going to have a look at the numeric responses. So you can see we had these first two questions, these had numeric responses. So that's what we're going to deal with now. And we're going to take a look. So basically this is the question that was asked. So basically saying if the firm develops the way you'd like it to, how much revenue would you receive? How many employees would have five years ahead? So it's thinking about like in the future, how much do you think you're going to grow? And there are two different metrics for that. So we're going to consider the first part here about employee count. And because we, we've got a lot of numeric responses, we want to know what's the distribution of these responses. And it's very natural to visualize this using a histogram. Okay, so we're going to draw a histogram using the survey data so that we should have got a survey data frame available. We're going to look at question two, row one, and we're going to set an access label to make it a little bit clear what's going on. So we're going to use the function PX histogram and the first argument to this is the data set. And then on the X axis we are setting which column of this data should we be plotting, which in this case is Called question two, row one transformed. And that goes in quotes because it's a string. And then we're going to set the access label to make it a little bit easier to look at. Actually, before I do that, I'll just run it so you can see what this looks like. So we've got a histogram here. And the X axis is a bit mysterious because it's just got the name of the question. So this is where we're going to override it. So I'm going to add in this X axis label. Now in the code hints you can see the, the code pattern for this. So we're going to use the labels argument and if it takes a dictionary. So this is a dictionary with the braces and on the left hand side the key for this dictionary is going to be the variable name. And then we want the text for the X axis on the right hand side. So I've put this in the question what we want. And I copy and paste that in here and run this again. You can see same plot, but now we've got a more informative description of what's happening on the X axis. So let's just take a moment to actually look at the plot and see what it's showing us. So you can see most, the most common Response is between 0 and 100. So this is the expected employee count in five years as a percentage available from last year. So they're taking the current number of employees and saying in five years time do you think you have more or less employees? And actually almost was that. So 48 of them out of 120. So what's that? That's 40% of them expect to have between 0 and 100% of the current number of employees. So 40% of the companies think they're going to decrease their head count in, in five years time. And you've got nearly 30. So about a quarter of them think they're going to go between zero and, sorry, between 100 and 200. So they're going to be somewhere between the current number of employees and double the number of employees. And then you've got a few very optimistic managers who think, okay, we're gonna dramatically increase the number of employees. So in this case between 1200 and 1300%. So I think in five years time they're going to have like 12 or 13 times as many employees. That's like we've taken a load of venture capital money and we're growing incredibly fast. Now one thing I'm not totally clear on There were four responses where they have a negative percentage employee count. I'm not really sure a negative count means of this set in this situation. Sounds like they're just giving up completely or it might be some dodgy data there. So this is worth maybe reading the paper and seeing what, what do the authors think about this, seeing there's more information there. Otherwise there's some questions around what's the meaning of this? All right, so that's pretty interesting. But we also have this column, where's the data set gone? So at the start called growth firm. So this, this is the column I mentioned about. Is this firm currently considered to be a growth company or not? So we might expect that growth companies are going to be more optimistic about how fast they're going to grow over the next five years. So if you're already growing fast, you might hope that it's going to continue. So it's very interesting to look at a difference in the values of this question. Two row one column about employee growth depending on like split by these two groups. And if you've got. So what we can do is instead of drawing one histogram, we can draw two histograms, one for each of these groups. And the way to do this is actually we're going to copy and paste this code, copy that, the command C and then we're going to paste it in here. And we need to add one extra argument. So let's add one more argument. And the argument is called facetrow. And we're going to set that to the column that we want to group things by, which in this case is growth underscore firm. So just notes Capital G, Capital F. This is case sensitive. So let's run this. And now you can see. So these are the non growth companies, the ones that aren't currently growing. So this is growth firm equals zero. The ones where it is a growth firm, that's growth firm equals one. And actually I have to say just eyeballing this, I can't see a big, big difference between the two. It looks pretty similar to me. There's maybe a bigger percentage of companies in this sort of area whether going between 100 and 200 and then between 200 and 400%. So the do seem to be for these growth companies, there seem to be more of those managers thinking that they're going to either double, triple or quadruple the number of employees over five years compared to those that aren't. But otherwise there's not an awful lot of difference appearing like from this Histogram. But just to be sure, we're going to do some statistics to see if there is some kind of difference. Actually, before we get to that, you're going to practice drawing some histograms yourself. So I'm just going to give you a minute or two to do this and you can do the same histogram again. This time you're going to be looking at question two, part two. So it's this plot, but change the question and draw it for yourself. I'm going to give you one minute to have a go at that, then we'll go through the answers. Actually, Rhys, while people are doing that, have we got any questions yet?

No questions just yet. But yeah, if anyone does have any questions with anything to do with codelong, let us know in the comments and we will get those answered.

Absolutely. Happy to take all of your questions. All right, so let's go through the risk, through the answers for this. So again, the easiest thing to do when you're doing two similar things is copy and paste. So I'm going to take the last histogram that we drew, paste it in here, someone called Luke in my workspace actually, and go as well. So Reese should have posted a link in the chat so you can get your own copy of the workspace. You don't need to jump into mine. You can have, you can have your very own workspace to code along in. Reece, if you want to post the link in the chat again. Okay, so we've got the code for plotting question two, row one. We're going to change it to question two, row two. So this is a fairly straightforward transformation. We've got to change it in two places. So I'm just changing the name of the column that's considered in these two places. And the other thing is the title is now wrong. We need to change employee count to revenue. Okay, let's run this again. And this case again is a. A little bit hard to see. So there's one really, really optimistic manager who thinks that actually the revenue growth is going to be, what's that, 150 times what it currently is. So that's tremendous. That strikes me is perhaps it's someone at a startup where they've gone from having a free product to they've just introduced their premium, their first premium product. So they're going from basically nothing to some kind of money because in five years, growing your revenue 150 times. That doesn't happen often, but you can see most managers are a lot more modest. So if they're not a growth company. You've got almost everyone saying, well, actually it's between zero and five times. And then again that's the most common for the growth firms as well. But you've got a few more people where they're a little bit more optimistic. All right, so this is a fairly interesting plot to play around with because of this outlier. What you might want to do is say let's have a look in more depth at, at the sort of lower end. So because it's plotly, we can zoom in a bit. So let's zoom. Okay, let's zoom in. All right. The panning is actually not amazing in this. It's just showing us a little bit closer. One thing that might help is to go, if you want to play around with this further, is to try and edit the axis range. So the X axis you probably going to want to look at like between zero and where's it stop being interesting around 2000. And that's going to give you a little bit more insight as to what most people are thinking without just worrying about, well, what is this super optimistic manager think? So that's something for you to carry on with and play around with later. All right, so one thing I mentioned before is that those two histograms when we were looking at the employee count look really, really similar. One thing you might find though is that there is a statistically significant difference between the two groups. So the growth, the managers who are working at growth companies might have slightly different opinions from the manager. Non growth companies and statistics can pick this up. So if we had normally distributed data, we'd use a T test. But they, the data is very, very skewed. You can see, okay, most people have like a, a number somewhere around here and then you've got this right skew. So a few extreme outliers on the right hand side of the plot. So because it's really not normally distributed data, we can use a man went u test. This is also sometimes called the Wilcoxon rank sum test and it was very confusing. Basically there were three statisticians, Man, Whitney and Wilcoxon, all working on this sort of separately and so multiple names for the same thing. All right, so before we get to doing the test, we've got to write some more SQL to pull out the data. So to begin with we are going to select, select this column of interest, the question two row answers of the one about employee count and we're going to select it from the survey CSV. So we're not looking because we're writing SQL, we're not getting it from a data frame again we're going to have to copy and paste our previous code on how to where are we on importing the data set. So all the way back up here, right at the start you've got the read CSV auto scroll back down to where we are. Okay, so we're reading the survey data again and this time we need an additional where clause because we don't want all the rows, we just want the rows where growth firm is zero. Let's run this. You see before there were 120 rows, now we've already got 58. We can do the same again for getting the the ones that actually are growth firms. Oh, just double check in case it's different for you. Make sure that you've got the results of this as Q2 1 non growth. It's just written in the, in the instructions there. So I'm going to do the same again but this time finally growth firms. So easiest thing to do is we're going to copy and paste. And then the thing we need to change, we want the cases where growth firm is one. So to find the growth companies, just double check that you've got the right value for what this is being to what this is being returned as you want a data frame Q2/1,_growth. I'm going to run this and we've got 62 rows. So we had 58 before, plus 62 makes 120. That's the whole data set. That's a good sign to check that we've actually got the right amount of data. And then in order to perform the Mann Whitney U test, we've got two separate data sets. So we're going to call Man Whitney Whitney U. That's got no typos. I'm going to pass in these two variables. So what this is going to do is, is determine whether there is a statistically significant difference between the average expectations of employee count growth between the two companies. Sorry, between the two groups. And the thing you want to look at here is the P value. So we've got a p value of 0.0 sorry, 0.00. So it's a pretty small number. So in that case it suggests that there's a very high probability that there is some kind of difference between the expectations of growth from the managers between the non growth and the growth companies. I've said this is a little unsatisfying. Now I included this task because this is one of the things they do in the paper. But I got to this point, I was like, okay, so we know there's a difference, but we don't know how much that difference is. Is it important? Why do we care about this? And so it's leaving me, leaving me hanging a bit. I'm like, well, you know that what I want to do more. So this is something that, again, is something you might want to look at in more depth. You might. How big is there? How big is this difference between the two groups? So, for example, you could calculate what is the average or either as a mean or as a median. And you might want to do some more sort of sophisticated modeling. You might want to do some kind of regression to see, well, what affects the difference in perceptions. Do some of the responses to some of the other questions contribute to this? So there are all sorts of questions that you can and make it clear that just run this hypothesis test. It's not really an answer as such. It's more something to annoy you into asking more questions and then doing more analysis. So you're not done after one hypothesis test is really just the start. Now, hopefully I've annoyed you enough to make you want to continue working with this dataset. Go and continue, continue doing some analysis. All right, So I just wanted to give you a little taste of statistics if you're interested in this sort of thing. The hypothesis testing in Python course does have a lot more of this sort of stuff. So you can find out a bit more about, like, what you can do with these different tests. All right, from that, we're going to go back to visualization. This time we're going to look at some of the categorical responses. So most of the questions in this survey, they've got categorical responses. You've got five choices that go strongly, strongly disagree, disagree, neither agree nor disagree agree, and then strongly agree. So you've got five points. Now, the values are encoded as numbers. Let me scroll back to where the data set is shown and you can just see what they're like. So, for example, all of these, these are these questions with the responses from strongly disagree to agree. So you can see their numbers from 1, 2, 3, 4, 5. This is okay for some kinds of analyses, but when we're drawing visualizations, just showing 1, 2, 3, 4, 5 is not very helpful. What we really want is to show strongly disagreeable and so on up to strongly agree. So we're going to need to replace these numbers with the text labels. There is in fact, one more tricky point with this is that sometimes you're going to get questions where nobody said strongly disagree or some of the data is missing. So you can have sort of missing values to deal with. And so we're going to see how to do that. Now, one thing to note, very useful piece of jargon, if you're doing any kind of survey analysis or anything sort of social sciences, these sort of questions where you've got opinionated responses going from strongly disagree to disagree or whatever the categories are. So you've got ordered categories, they're called likert scales. And so it's just a bit of useful jargon to know if someone ever mentions that to you. Because, okay, this is just about opinions, right? So first thing we do is build up a SQL query to get the counts for each response type. And then we're going to use that to draw a bar plot. Now, I've got a file here called agree disagree CSV. So this is going to be a lookup table that's going to take us from Those numeric codes, the 1, 2, 3, 4, 5, to the text values that we need. So first thing we're going to do is import that. So we're going to do select star from again. So this is standard formatting, which means we can just use a string with the file name. Oh, I need to change the value of this to lookup. I think you're probably all gonna need to change this. You want to make sure the return value is a data frame available as lookup. You can see we've got two columns. We've got a column called code 1, 2, 3, 4, 5, column called response. And those are the five different options. All right, next thing. So what we need to do is take this lookup table and we're going to do a left join to the survey data. So first thing, copy and paste the previous code and then we'll add in a left join. Oh, actually I wanted the as lookup here. So we're going to left join and one useful thing, that read CSV auto function that we used before in the from clause. It also works in the join clause as well. So I'm going to go back to where we had this before and I'm a copy and paste that that reading code, the read CSV auto code. And so it's here in the, in the left join code and I'm going to say this one is as survey. Now we need to tell the SQL query which columns we're joining on. So we're going to join on lookup code is Equal to survey dot. Oh, we're going to look at question three, row one. There are a lot of these different questions, but I thought let's start the first one and that's about it. So let's run this. Ah, so we got lots and lots and lots of columns here. There's a bit much to worry about. So let's, rather than selecting everything, let's select some specific columns. So we're going to select lookup, dot response and we're going to select this survey do question three, row one. Let's try this again. All right, so we got 121 rows. And so actually, let me just double check. I've done that.

Right.

We should only have 120 rows, really. So lookup. All right, let's see if this works. So you can see wherever you've got the value agreement, the original value was a four. When you've got strongly agree, the original value is a five. Wherever it's neither agree or disagree, you've got the value three and so on. So we've got all the different responses. It's just now we've got a text version as well as the original code. All right, so we're going to build on this again. Excuse me. And this time we want to get the count rather than just the individual values, because we don't care about the individual responses, we just care how many times they say agree, how many times it say neither agree nor disagree. So next thing we're going to copy and paste. I think this is actually easier if we put each thing that we're pulling in on a separate line. So one thing we have to change is rather than getting individual responses, we're now going to get the count of those. So we want to wrap this survey, question three, row one, and count. We give it a meaningful name. So this is the number of responses. We'll call it N. And then we need to group the results by the lookup response. All right, so here you can see for this question three, part one, we've got zero cases of strongly disagree. So no one said that they strongly disagree with things. We've got 18 cases, neither agree nor disagree. We've got 67 cases of agree, 29 strongly agree, six disagree. So this is great. There's a little problem in that these aren't in the right order. Ideally, we want things to go from strongly disagree all the way through to strongly agree. And the right order, that's not there yet. So we're going to have to include an order by clause. The Other thing, just bear with me a second. The other thing is that in order to draw an easy to interpret plot, we want to include a color scheme. Now for this sort of data, where you've got a central value which in this case is neither agree nor disagree, you've got, you want that to have a neutral color and then you want bars that represent agreement to all be the same color. And you want bars representing disagreement to all be the same color. That's going to make it easy to understand. And then generally the stronger the opinion, the more intense the color you want. So we're going to pick a color scheme that makes sense for that. In order to get the data to make this work, we need to add an agreement statistic. What we want here is evaluate. So where neither agree nor disagree is, this is the neutral things that want that to be zero. When you've got positive agreement, you want have a positive number and the stronger the agreement, the bigger the number. Likewise when you've got disagreement, you want a negative number and the stronger the disagreement, the bigger the negative number. So in this case we want to take the lookup code and subtract three and that's going to give us a, an agreement statistic. So just a little bit of data wrangling there. All right, so I am going to copy this and we're going to do two, two little tweaks and then we've, we've finally got the, the, the data we need. So I told you we had to order these. So we're going to add an order by clause and we're going to order these by the lookup code. So this is like the values 1 to 5. So that's going to give us strongly disagree is like first and then strongly agree is last. And the other thing we need to do is add in this, this agreement value. So we're in the select clause. So this is the last thing we're selecting at the moment. The count value is N. I'm going to add a comma. It's very important. You've got to put a comma between all the things you're adding. Hit Enter. It's going to be a new line. So here I'm going to calculate the lookup.code/3 and call this agreement. Let's run this. Okay. Okay. So one other thing I need to do is add this to the group by clause. Now so you need to, in the group by clause you've got lookup response, you want a comma and then add lookup code and the Reason for that is because we're doing calculations that aren't included in this count aggregation. Let's try that again. All right, so we've got the same, same numbers as before, just in a different order. So here, now you can see the results go from strongly disagree through to strongly agree. So these are all in the right order. And then we've got this agreement level, so you've got zero in the middle. Disagreement is negative, agreement is positive. Now, this is the data we want. We're going to save the results. If you call it Q3,/1,/counts, make sure it's the same, just to just sort of. The following code is going to match. All right, now we're finally ready to draw our plot. So in this case, because we've got counts of categorical data, we want to draw a bar plot and we're going to do some, as I mentioned, we're going to use a color scale. So, so it's called a diverging color scale. It's got a neutral value in the middle and then it goes in two different directions. So to draw a bar plot, we're going to start off PX bar and we've got a few arguments to add to this. So the first argument is the data set we're going to use. So it's Q3 one counts. The second argument is what we're going to put on the X axis, which in this case is this response column. On the Y axis. We want the counts and then we're going to color the bars by agreement. Now, one final. Actually, let me plot this as is and then we'll show you what happens when we give it a different color scale. All right, so here are the results. You can see, as we saw from the numbers, no one strongly disagreed. The most common thing was that, well, quite a lot of people just agreed and then there are a few people neither agree nor disagree, and a few people strongly agree. Now, this color scheme of green, it's a bit of a mild melter, like the rainbow color scheme going from purple to yellow. This isn't very useful to us. The one thing we need to do to tweak this plot is to add in a diverging continuous color scale. So in the cotins, I've explained how to do this. So we're going to use the color continuous scale argument and then the options for this are all contained in PX colors diverging, PX dot colors dot diverging. And then you've got all sorts of different color scales. Feel free. To try a few and see which ones you like. I actually quite like the, the army rose underscore alpha reversed for this. So let me hit play and you can see what it looks like. So in this case you've got a neutral beige color for the neutral response and the positive response is a sort of green. It's like an army green. So this helps for like colorblindness is like not quite the same as the, as the red. Like I'm colorblind, red, green, colorblind and I can see the difference between these. So then you got like red responses for, for, for disagreement and then the, the most intense strong agreement or disagreements they have a darker color scheme. So this works really well for this sort of opinion data set. The Likert scale categorical questions. I'm pretty happy with how this looks. Now before we go to questions, you have got one more option. You're going to do the same again using a different categorical question. So any of the questions in here with from question three through to question six, they've all got the same agree disagree scale. So you're going to go through and take the previous SQL query, put that together with a new question, just edit for the new question and then you're going to draw a bar plot. I'm going to give you two minutes to do this and then I'll go through this last answer and then we'll go to audience questions and actually Reese. So yeah, so you've got two minutes for that. Rhys, do we have many questions from the audience yet?

Yep, we've got a couple. So the first one that we've got is from Eric. He says why is the column the same but it says change the column. I think this was an earlier one back when we were doing the SQL bits. So yeah, why is the column the same but it says change the column? I think that's when we had an.

Error earlier that was almost certainly copy and paste error or bad instruction from editing on my part.

So next one is from Tom. He says, in which case, can we use the Mann Whitney U test when comparing two numerical variables?

Yeah, so the idea is you've got two groups of variables. You want to know on average are they the same or different. So the short term, the conditions generally, I think it works best when you've got slightly larger sample sizes. You need at least two, I think 20 bits of data in each set is the sort of standard and because it's what you call a non parametric test, so it doesn't worry about the shape of the distribution. That means it's got less power than something like a T test, so it's less powerful for detecting differences. But it does mean that you don't have the, the constraint that each of your distributions should be normally distributed in order for it to make sense. So this is useful when you've got two sets of weirdly distributed data and you want to know is there a difference between them?

Got it. Cool. We got a question from Rich saying, can you explain what a left join is? Joins can be complicated.

Oh man, not in 10 minutes. It's when you tend to use them when you've got repeated values of rows and you want to. It works really well in this account situation when you want to know, how many of something have I got? How many instances of something I've got that correspond to different categories. I would say I'm not going to do justice to an answer here. Go and take the, the joining data and SQL course, but actually we've got a ton of joining. Do the same again in Python and R as well if you really want. And it also gets taught in about like 12 other different courses in sort of in, in small chunks because it's something you need to know and it's, it's probably the most difficult thing you do conceptually, except for maybe statistics, like in any of the data analyst or data scientist tracks. It does take a while to get your head around it. If you've come across a vlookup in Excel, it's the same thing. Vlookups are left joins.

Got it.

Yeah, we'll, we'll do a whole session on joining data. At some point you practice different things. I would say you're going to get a better explanation if you take some of the courses.

Okay, cool. I'll be, I'll, I'll find a link to the, to the joining data course and put that in the chat shortly for anyone.

Wonderful. All right, I'll go through the answers of this then. And everyone in the audience, if you want to think of more questions, we've got a few minutes left to do that. Okay, so we're going to pick a different question. We're going to copy and paste the previous SQL query and update it for the new question. All right, so I'm going back to the query. I'm going to take a copy, paste this in. I'm going to do question three, row 13, and it needs changing in two places. It needs changing in this count and it also needs changing in the on part of the join clause. So changing in two places, saving it as Q3, 13 counts. If you picked a different question, give it a meaningful name for you. So you've got a different set of counts here. And then I'm going to go back and again I'm going to copy and paste the plotting code. So it just needs changing in one place here. And we're going to run this. All right, so this one's a lot more balanced. So before we saw there was a lot of agreement here. We see most people don't have, like, there's a fairly sort of balanced thing between strong opinion, like strongly agree and strongly disagree. It's actually unusual. So one thing you find in a lot of surveys is that when someone says a statement, very few people will have strong opinions. So you. They tend to sort of do something for the middle. And in general, there's a bit of a bias towards agreeing. Like a lot of people are very agreeable. Like someone said a statement, I said, okay, I'll agree with it. So strongly disagree is actually like quite a rare thing in survey responses. I guess there are some people who are naturally disagreeable that they're going to respond like that, but it's rare. So when you're designing a survey, you've got to be very careful about how you phrase your questions. And maybe sometimes you have to have questions phrased in opposite manners just to see do people account for sort of biases and how people answer questions, how people respond. Okay, so that's, that's pretty cool. Maybe while we're doing this, we'll, we'll just try another, another color scheme so you can see what it looks like. We'll try Earth. Okay, that's it. You got kind of bluish on one side, brownish on, a little bit muddy in some situations. That's probably kind of cool. I'm not sure it works that well for this sort of survey data. Have a play around. Try some more color schemes, see which one you like best. Okay, with that, I think I'm done coding along, so I'm gonna return to questions.

Okay, brilliant. So we've got a couple of specific questions on workspace. Just a reminder for anyone that hasn't sent a question in that you want answered, now's the time to do it. And yeah, we'll. We'll get to them over the next 10 minutes. So we've got a couple of. Yeah, workspace specific one. So we might. We may need to jump back in. So we've got this one from Tom. He says, is it always that easy to switch from an SQL cell to a Python cell in a workspace. Is there any preparation that you did for this specific session?

So actually my Python slightly stronger than my SQL. So I started writing all of the data analysis code in Python and then I was like oh yeah, we should probably do this in SQL, otherwise it's going to be a bit weird. It is very straightforward. The natural workspace designed to have this natural flow where you pull in data either from a database or CSV file using SQL and then SQL is very good for simple number crunching and then you can save the results data frame continue doing more sophisticated things in. In Python or indeed R if you. If that's what you prefer. So yeah, it's very, very easy in workspace. It's exactly. We just built it to be able to do this. Cool.

Next one is from an anonymous LinkedIn user. They ask the way you. You're directly referring to the CSV file name in the SQL query. This method doesn't work outside of workspace. What do we do in. In that instance?

Well, I obviously prefer this workspace but yeah, so in a standard jupyter notebook all this is based on using DuckDB. So DuckDB is gonna have like the. It's got the read CSV auto function. You just gotta figure out how to integrate DuckDB with your existing notebook. I think there are some document, there is some documentation on that on the DuckDB website. The other alternative is you just ignore SQL completely. If you're wanting to use Python then Pandas has all the sort of functionality built in so you can do everything in Python if you prefer.

Cool. Next question is from Yukon. Nice to see you again, Yukon. Is there any test that can be used for comparing two categorical variables?

Yeah, it depends what you're comparing for. I mean I think you're probably looking for a chi square test, but depends on the question you're going to answer. I would say go and take the hypothesis testing courses because it gives you a pretty clear idea of like which test to use in which situation. I have to say after years and years of doing this, I can never remember which test to use in which situation. And I do always have to look it up before I actually start doing any analysis. So yeah, I defer to the course again in this situation.

Cool. Second of all, we've got a question from Suman. So he asks could we have used a two sample unpaired T test for the hypothesis test?

You can. So really T test is sort of. You're supposed to have vaguely normally distributed data it's going to give you a slightly different P value and I would say run it and see what happens. I don't think you get much of a difference, but in general when you've got this really skewed data, the non parametric version, the Wilcoxon, sorry, the Man Whitney U test or Wilcoxon rank, some test, same thing that's preferred.

Cool. That's it for audience questions for the moment. I've got one question of my own before we, before we, we wrap things up. So if someone's completed this code along without too many problems, what should they be looking at learning next? Like what's, what's the next sort of progression?

Yeah, absolutely. So one thing we didn't get a chance to look at today is how the responses to multiple questions interact. So I did some homework tasks there at the end of the workspace. So for example, you've got two different numeric response questions. You can put those on a scatter plot and see what is the relationship between the responses. In those two cases we have like lots of categorical response questions. You can look at pairs of these and see, you can draw them on a heat map and see what's the relationship between those two distributions. There's also all sorts of really interesting questions like which statements do people most agree with or most disagree with? So finding out some sort of sense of average agreement, ordering them by question, really, really interesting. You can also see like which ones made people like were most passionate about answering, like what made them most mad, like which questions have the most strong agreement and strong disagreements. There's all sorts of stuff you can look at in this data set and then in terms of like what, what you want to learn. I would say please do go and take the analyzing survey data. We've got one in Python, one in R as well. Take one of those courses. I presume most people here, more Python people. So take that course.

Cool. And I've just put that in the chat for anyone that wants to have a look at those courses as well. Brilliant. That's it for questions. So I'll leave you wrap up.

All right, super. Before you will dash off, we've got three more webinars this week, so tomorrow there is a session on data governance strategy. So data quality, data governance, it's tremendously important for like actually having the right answer in any kind of any of your analyses that's gonna be worth turning up to tomorrow. Got some very senior executives come along to that. On Thursday there is a careers based session. So we're looking at how top universities teach data science. So I'm interviewing two people who run master's, pro masters in data programs, talking about, like, well, what skills do you need to know? How does it lead to getting a job? And then on Friday, vertical session from some previous winners of data camp competitions. So we've got four previous winners talking about what they did to win. All right. And with that, I think we're at Time. So I hope to see you all again later in the week, to see you again in future webinars. Thank you very much.