#56 Data Science at AT&T Labs Research

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

This week I'll be speaking with Noemi Dashi, a senior inventive scientist at AT&T labs within the Data Science and AI Research Organization where she does lots of science with lots of data. We'll be talking about her work at AT&T Labs Research, the mission of which is to look beyond today's technology solutions to invent disruptive technologies that meet future needs. AT&T Labs works on a multitude of projects from product development at ATT to how to combat bias and fairness issues in targeted advertising and creating drones for cell tower inspection research that leverages AI, ML and video analytics. We'll be talking about some of the work Noemi does, from characterizing human mobility from cellular network data to characterizing their mobile network to analyze how its topology compares to other real social networks reported, to understanding TV viewership and how engaged in different shows. We'll discuss what the future of data science looks like, whether it will even be around in 2029, and what types of skills would help you to land a job in a place like AT&T Labs. Welcome to Data Framed, the weekly datacamp podcast, exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host Hugo Bown Anderson. You can follow datacamp on Twitter @datacamp and me, Hugo Bown. You can find all our episodes and show notes@datacamp.com Community Podcast this is Data Framed. Hi there Noemi and welcome to Data Framed.

Hi. Thank you for having me.

It's a real pleasure to have you on the show and I'm really excited to be talking about Your work at AT&T Labs in Research at the moment. But before that I'd like to find out a bit about you. So on your website and you sent me a bio. I'm going to read it out because I love it. You're a senior inventive scientist at AT&T labs within the Data Science and AI Research Organization. And I love what you say next that you're doing lots of science with lots of data.

Yes, well that is what I do.

Exactly. So you're working at AT&T now, but you actually have been involved in a lot of other initiatives in the data community. So I thought maybe you could tell me, give me a bit of background, tell me about other things you've been involved in.

Yeah, for sure. I spent a lot of time before becoming involved in the open source space in academia, where I actually didn't have the opportunity or the bandwidth necessarily to work on open source projects that were actually putting me out there in the open source community. But I started becoming more active in the open source space once I became a NASA datanot. And here I started working with NASA's open source metadata, which is basically information about their over 30,000 data sets that they made publicly available. That's how I started getting involved in the data science community and in open source. Recently I also became the co organizer of Women in Machine Learning and Data Science Science Meetup in New York. And here we organize meetup events focused on machine learning and data science topics. And our mission is to provide a supportive community that encourages and promotes women and non binary people in tech.

Fantastic. And actually I've recently had Reshma Sheikh on the podcast to talk about a lot of the initiatives at Wimbleds in New York City.

Right. She's my colleague at Women in Machine Learning and Data Science.

She's fantastic. And so I'm also interested in this idea of being a NASA datanaught. Can you just tell me a bit about this? It sounds really cool. To start with, I want to be a datanaut. So could you just give me a bit of context around it or what the program is?

Yeah, I think every data scientist whose dream was to become an astronaut but didn't make it now can become a datanaut.

That's awesome.

So yeah, when the government forced these government agencies to open source some of their data sets, then NASA open sourced over 30,000 data sets and people don't know about it. So one way they try to promote this is with the NASA Data Nodes program. This is an initiative in which they try to create this collaborative group every year of individuals who are interested or excited about working with their open source data sets. And we have meetings regularly, we have webinars, people can present what they are working on, they can start collaborations based on NASA's Open Source Data sets, can come up with ideas how they can use them. And then there are data scientists from NASA who are actually presenting what they are doing and telling us how we can get involved in certain projects that they would be interested in seeing results in, but they don't have the time to work with. And we also, the chief knowledge architect from NASA, David Meza, he's also very involved and he's very supportive of this community. So you can always just reach out to him and he's going to be very supportive no matter what your question is or what projects you want to work on. So it's an amazing community to be a part of. It's application based. So every year they launch their application opportunity and people can apply and if they get selected, then they can just once they become a NASA datanaut, they will be datanauts forever.

That's really cool. And if our listeners, who I'm sure are really excited by the idea of being a NASA datanaut, at least as much as I am, are interested, we'll include a link to a few things in the show notes as well.

Oh yeah, sure. Well, open.NASA.gov is the first place to go to and then there you can find information.

Perfect. And so the other thing that I know that you're excited about is teaching and pedagogy and data science instruction. Right. Also run workshops in the wild at conferences and this type of stuff, I think. Do I remember correctly with your interest in network analysis and this type of stuff?

Yeah.

So actually my bachelor's degree, master's and PhD and then a five year postdoc all involved network science. So many of my research projects were on understanding complex systems through their network structure. So I was thinking that this is a good opportunity to have these workshops to show how you can do network analysis. Especially because there is this very good Network X Python package out there that can enable data scientists to just analyze the data from network point of view very easily.

I love NetworkX so much and we've actually got two courses introducing NetworkX on datacamp, taught by Eric Ma, who's an old friend and collaborator. He's now a research data scientist at Novartis. But NetworkX is a really great package and the API is really nice as well. Well, I've found. So we've got a few ideas about your background, but I'm just wondering how you got into data science and analytics originally.

Yeah, well, I got into data science long before it started to be called data science and I think this was back in 2006. So to give a bit of context, I did a bachelor's degree in physics and computer science. So I had to write a bachelor's degree thesis on something novel and related to the field, and I didn't know what that would be. I could either decide doing some physics project or some computer science, but I really wanted to find a method that I can combine these two. So I actually was very lucky because I had this great quantum physics professor who is a leader in the research space and he is always interested in physics applications outside the traditional boundaries. And at that point he was working on projects that were focused on understanding complex systems and even more complex systems with underlying network structure. What he was always working on within his projects at the time was to analyze and model these complex systems through some data that he obtained from different sources. This was using a lot of computational physics, which I really liked, and also leveraged data analysis. I found this topic very exciting.

Explains your interest in networks to this day, why you educate around them, your love of network X. And I suppose also as we'll get to some of your work at, AT and T, thinking about, you know, networks of individuals in a society and communication between them and that type of stuff.

Exactly. And that's how I actually got to do my first project, using social type of data, which was from this Erasmus European Union scholarship framework. And here I actually built my first network, which I found really exciting. I built a network of European universities where the connections were built by the students who went from one university to the other.

Interesting. So is that a directed graph?

Yeah, you can look at it either at the direction or you can look at it from undirected way. We actually built both the directed one and the undirected one, because if you take into account from which university the student goes to which university to visit, then it would be a directed one. But if you just want to look at professional connections, for example, I just want to see how universities are interconnected among each other. I don't really care about the direction. So then you just look at the undirected version of the network.

That's really cool. I'm wondering about the data collection and data generating process in this case. Like did you hand write all these universities down and then figure out using another data set, or did you figure out a way to automate it, or how did that work?

So actually this was a fairly small data set. It was only a snapshot from 2003. So they gave us just a small data that we could play with. It contained information. It was basically a matrix version that I received. So each row and column was the university and then the value was the number of students that went from one to the other. So that was the data that I got back then.

Fantastic. What did you do with it then? What were the takeaways?

So we actually revealed the most interconnected cluster, the subgroup. And it was very interesting because it wasn't the top universities, but it was actually someone at the conference mentioned when I presented this that it looks like the universities belonging to the cities where the students can have the most best parties. Yeah. What we basically found was that the connections are very much influenced by the professors connectivities. So the professional network of the professors is the one that basically drives these connectivities within the students, despite the fact that they have the opportunity to choose themselves where they go.

Right. And actually that's. So I actually did a postdoc or the first half of my postdoc in Germany in cell biology. And I do remember a lot of people came through the lab and the institute as a function of professors and researchers and social connections between professors where I worked and professors at other institutes and campuses.

Right.

So then if I recall correctly, you, you worked on something else for your master's thesis. Right.

I got so fascinated by this topic that I wanted to continue to do the same thing throughout my master's degree. So for that thesis, my advisor, the same advisor who eventually I got the PhD with as well, because I was such a big fan of his type of research and work. So he obtains this endrone open source email communication data. I don't know if you know about it.

I know it will.

Yeah, yeah. Okay. So I use that email communication data to start analyzing how people communicate with each other and if we can detect some pattern and build a model. And yeah, the most interesting part was like from a physicist point of view was that basically the communication was like an exponential decay, like a particle decay.

Oh, interesting.

So, yeah, we showed that the later you reply to an email, the less likely it is that you're going to actually reply to it and the probability is going to drop exponentially in time.

Right. Well, anecdotally that seems right for me anyway because there is this. I don't want to go into this too much, but there is this barrier, right? Like if I've left an email a week to reply, then I'm like, oh no, it's got to be. I've got to actually give a proper reply now as opposed to, you know, just a few words like, hey, received, whatever. There is a barrier, barrier there. The other thing that I just wanted to say about the. I'm actually very familiar with the Enron data set in a slightly strange way. A good friend of mine, and I've mentioned this on the podcast once before, a good friend of mine who's a digital artist has done a project where you can register to receive the Enron emails daily.

Oh, cool.

So yeah, I can put a link in the show notes as well. And so I actually in my inbox every day I receive one of the Enron mails. I think today's was like Mark from Legal thanking someone else for like dinner last week or something like that. But it's actually really odd and a very intimate data set as well.

Yeah, I actually haven't followed up to see how the data evolves compared to the fraction of the data that I had back in the day. But actually that would be a cool thing to follow up and see.

Absolutely. And let me ask, is the work that you did for your masters on the Enron dataset, is that on GitHub or out in the public domain at all?

No, I didn't post it back then. Back then I was doing it in C and it was like very long time ago. And in academia it's also not very popular.

Yeah, okay.

No, that makes sense to open source things. That's something that I think academia can work on.

It can, and it wasn't necessarily incentivized back then. And it's generational in a lot of ways. We've discussed that on the show before, but I do think it's becoming more. More and more commonplace. Particularly more and more people are learning R and Python as opposed to Matlab and whatever else there is. Not that Matlab isn't great for certain things. I don't want to say that.

Yeah, well, I always use C. So from there, for me a natural transition was Python. And now since most of my colleagues here use R, it's something that I'm dipping my toes into for sure.

When people ask me why I write Python, my first response always is because I love writing Python code. It's so much fun to write. But today we're here to talk about Your work at AT&T Labs Research. So I thought maybe you could just break down for me what, what the mission and history of AT and T Labs in in general is.

I actually represent ATT Labs Research, so I can talk about ATT Labs research mission because ATT Labs is a very broad research and development division of AT&T. So the mission of ATT Labs Research is to look beyond today's technology solutions to invent disruptive technologies that Meet comprises very diverse and fascinating research areas that range from AI 5G technology to video and media analytics.

Great. So this is stuff that maybe won't be implemented right now, but thinking very much as my Belgian colleagues would call future music.

Right, yeah. So this is the big view and the future goal that we're working toward. Yep.

Okay, great. So maybe you can set the scene historically as well for us briefly.

All right, yeah. So the history of AT&T Labs, if you think about it, AT&T Labs traces its history from AT&T Bell Labs, which is famous for its very rich history and innovation. And as a physicist, I feel particularly honored to be part of the research lab, especially this research lab where several physicists and scientists from other fields as well have been awarded with a total of nine Nobel Prizes for their work done at Bell Labs, which I'm still very amazed by, just to name a few. The Bell Labs hosted extraordinary scientists like Walter Sherwood and John Tuckey, who contributed to the fundamentals of statistics, and Claude Shannon, who is the father of information theory and many, many others. So for me, it's very an amazing opportunity to be here and I'm proud of it every day.

Oh, that's really exciting. And I'm a huge fan of Claude Shannon's work, of course, And Tookie's really interesting, particularly in something we are only getting back to now in kind of the cultural discourse is really thinking about the importance of exploratory data analysis. And the focus in academia and industry for a long time has been on positive results. His focus on actually getting to know your data and all the techniques he developed to do that are incredibly beautiful.

Right. One of the updates that AT&T Lab has is we recently opened an AT&T science and Technology Innovation center in Middletown, New Jersey, which is a museum that comprises this 142 years of inventions that AT&T pioneered in. So you can actually go and check it out.

Oh, great, that's open now. Or about to open.

So it opened at the end of last year, so it should be open. Yes.

Okay, perfect. We'll jump right back into our interview with Noemi after a short segment. This segment is part of a series on guidelines for AB testing. So check out last week's episode with Chris Albon. To get up to date with the lingo there we define some of the terms we'll use today, such as P value. Gosh. Now we're back for the segment called Guidelines for Online Experiments. So I'm here with Emily Robinson, a data scientist on the growth team here at Datacamp. Hi, Emily.

Hi, Hugo.

So we're doing a series of segments on kind of rules of thumb and guidelines for online experiments. And in particular, you've written that there are so many ways AB testing or online experiments can go wrong, but most of them won't be obvious. Now, what I want to speak to you and one of your guidelines I want to speak to you about today is that you should pay more attention to confidence intervals than p values. So can you unpack this for me, perhaps using an example?

Sure. So the first thing I want to make sure that is understood is that there is a one to one relationship with p values and confidence intervals. So if a P value is less than 0.05, the 95% confidence interval does not overlap with zero. So in that sense, you can take the same information from both of them in whether you want to reject the null hypothesis or not. But the reason I say to pay attention to the confidence interval rather than just the P value is you can get a lot of information from it. A confidence interval approximately tells you, okay, what are reasonable, given all the data you've gathered in your tests, basically, what are the reasonable values that your test statistic could actually be in? So the percent change from the control to the treatment in an example. So let's take an example. When we run a test, let's say we are running a test trying to increase subscription rates, and we run it for the seven days that we have planned. But maybe on the fifth day, we're sort of checking some things and we say, oh, my gosh, the P value is less than 0.05 and subscription rate looks like it's up by 40%. Wow. We should celebrate. Well, okay, there's two problems here. One is you're checking the test early, which maybe we'll talk about in another segment.

Absolutely.

The second is, if you look at your confidence interval, what if you see that that confidence interval goes from 1% to almost 80% and your estimated difference is going to be in the center of that? Right. It's going to be around 40%. But this is showing you, wow, there's a lot of possible values that would be reasonable given all the data we have. And this is a huge confidence interval. So maybe I shouldn't think so much that we've really made a change in subscription versus. If you looked at the confidence interval and say it goes from 39 to 41%, you can say, all right, we have a lot of evidence that we did make a really big change to the Subscription rate.

Yeah. Interesting. And I suppose you can even take it a step further and use bootstrapping or some sort of similar resampling technique to look at the whole distribution of the test statistic, right?

Absolutely. So there's a lot of different methods you can use here. And just to give a little preview of what we might talk about in peaking, this is again, kind of helpful for sometimes you do want to stop test early. So let's say it actually was negative 40% that, oh no, your subscription rate might have gone down. If the confidence interval is minus 39 to minus 41, I'd probably say, hey, we better stop this test. I'm pretty confident we're having a real negative impact on subscription rate. Even if we're checking it early versus again, if it was this huge wide confidence interval, I'd say, oh, you know what, I'm not too worried yet. Let's keep running.

Right, so your next guideline is don't run tons of variants and variants of experiments. And we hinted at this in our previous segment, but I'm sure it can be tempting to try 100 different versions of your homepage, for example. So why is this a bad idea?

Well, the first one is it's unlikely you're going to have enough traffic to have that make sense. So maybe if you work at Google or Facebook. All right, you can have millions or tens of millions of people visiting your page, but most sites maybe have thousands or maybe tens of thousands. So if you think about it, dividing those all up between 100 different groups, all of those groups are going to be pretty small. And so you're going to lower your power and your ability to detect a statistical effect. The other thing is often how people will approach a B test with multiple variants is just to check control against each one. So you do say a proportion test on the Control vs. Treatment 1, Control vs. Treatment 2, Control vs. Treatment 3, etc. This will really raise the likelihood of a false positive. Now, there are statistical tests that are specifically designed for multivariate designs, but again, I think it's just usually better to stick with maybe a controller treatment or if you have tons of traffic on a page, up to three treatment groups.

Right. And I suppose the statistical tests and methods that we discussed last time as well, there are sophisticated correction methods out there. I actually don't know what you'd use in tech company, but I remember from my academic research days in modeling and cell biology that the Bonferroni correction was one such method.

Yes. So the Bonferroni correction is exactly a method, but the issue with that is that it makes all of your tests more conservative. So even the one for your key metric.

So you get false negatives then?

Exactly. So you're more likely to miss a real impact that you have. And actually it's interesting. So LinkedIn for example, wrote a paper about, among other things, how they deal with specifically not multiple variants but multiple metrics in one task. And what they do is they apply different thresholds based on whether they expect this metric to move or not. So for their key metric, maybe the threshold is your P value has to be below 0.05 for something you think might change, but it's not your key metric below 0.01. And for these monitoring metrics, that there really should be no reason this changes, but you just want to be sure the P value has to be below 0.001. And they actually talk about how that fits into a Bayesian framework of thinking where you're incorpor the priors you have for whether or not this metric will move from your control to your treatment.

That's really cool. So we'll link to that paper in the show notes and look Emily, I could sit around chatting about online experiments with you all day. We got to get back to our main interview, but we'll see you soon for the next segment on guidelines for a B testing.

Sounds great, Hugo.

After that interlude, it's time to jump back into our chat with Noami. So we've discussed briefly how you know the work at ATT Labs Research, how it thinks about the future of what can happen. I'm just wondering how it relates to the business side of AT&T.

Currently the ATT Labs was founded so it can focus on solving the hardest tech problems that AT&T is dealing with. And the solutions of these problems translates for at&t to improvements in customer service or customer care and and like many of the projects also result in cost reductions for the company like network optimizations and improving advertising and so on.

Before we get into the work you do AT and T Labs research, I'd just like a general like high level overview or kind of maybe you can tell me a bit about some of the current projects at AT&T Labs in general that you find most interesting.

Yeah, actually there are so many and as a new employee I'm just still absorbing all the information and all the new projects that I get to learn about from my colleagues I'm sure but but to mention a few that I'm actually not involved in. But I find very exciting or important. So one of them is creating new products to make a difference in the media and entertainment space, which also helps us build this partnership with Turner that has recently become a division of AT&T's WarnerMedia. So ATT has a lot of TV data and most of the time AT&T is not associated with TV data. But since AT&T owns DirecTV and now Turner, it's a lot of TV data that can be used to do critical and fundamental research in media and entertainment space.

Great, that sounds very interesting. What type of. Do you know much about what type of tools and techniques are used in doing that or what type of outcomes they're trying to achieve?

Yeah, so I think what I can say is that Basically that's why AT&T launched the Zender, which is their new company focused on advertising. I don't know if you know about that. After the acquisition of AppNexus.

Tell me a bit about it.

Last year AT&T acquired AppNexus and it became now Zender, which is focused on improving advertising in the media and entertainment space.

Interesting. And I suppose a big part of that now is thinking about targeted advertising. I think that's something your colleagues are thinking about as well.

Right. So the goal is like we have a lot of advertising that is just distracting and the goal is to provide less advertising but more relevant ones. So there is a lot of research that is going on around this problem.

Great. So are there any other current projects.

That you find very exciting coming out from this advertising? There are projects. There are also very important project that is going around that going on that many of my colleagues are working currently on, which is focused on how to combat bias and fairness issues in these targeted advertisings. So this is somehow related to the first one, but it's also very important. And then I'm going to mention one that has really nothing to do with this. It's completely different. But when I first got to ATT and I found out about it, I was like, wow, I didn't even think that that would be a thing in AT&T. So one of the projects is working on creating drones for cell tower inspection. So this research is basically leveraging AI machine learning and video analytics. And their goal is to create this deep learning based algorithm that is just going to. Once you send out the drones to create these video footages, you're going to analyze this footage to detect tower defects or anomalies. And this will enable automating the tower inspections and it will make it work faster and More efficient. So this is one of the things that. Oh, I didn't even think about that and I found it really cool. Just wanted to mention it.

That's amazing. The use of drones and the idea of using essentially deep learning and AI technologies and video analytics, as you say, in drones, has so many applications. One I've been reading quite a bit about, a bit about recently is in agtech, so agricultural technology. Right. And drones analyzing yields of field crops and that type of stuff. And one of the really cool things is I like this example that if you've got a camera on a drone and you're trying to build algorithms or get them trained and tested in real time, it isn't as though you can like throw the deepest neural net at it, for example, if you're trying to run it on board the drone, for example. So you're pushing up against a lot of technological constraints there as well.

Yeah, and this is so fascinating because for me, drones was, was never connect. I would have never connected it to such an important work at, AT and T, which is like you have to make sure that those towers are working properly and if there's any malfunctioning, you need to detect it in time. And this would be really helping that cause.

Exactly. So this is a nice cross section of different research projects at, AT&T Labs that, that interests you. So I'd love to, to jump in and find out just about some of the projects you're currently working on.

So my projects are also just as diverse as the ones I mentioned above, which I find really exciting because I have the opportunity to study and to work on very different data types and types of data and to try to answer very different problems and to tackle with very interesting research projects. But to mention the first one, which is my favorite, is human mobility characterization from mobile network data. Human mobility patterns revealed from cellular telephone networks can offer a large scale glimpse of how humans move in space and time and how they interact. I find this project very exciting because they can study the human behavior, which is a phenomenon that, that as a physicist I've become very interested in since undergrad. And this project also offers the opportunity to study human behavior through large scale anonymized customer data and leverage the discoveries to also improve our services.

That's really cool. And as we said before, these types of projects really speak to a lot of your interests that you've developed over the past couple of decades as well and what you've worked on.

Last year I was interviewing to find a new job and I was getting After a point, very frustrated that there are so many data science positions out there that I would have to literally throw out all the things I've learned in the past and all the research that I've done because I wouldn't be able to leverage it. And I am so happy that I found this job because I can just use everything I studied so far and it doesn't go to waste.

That's really cool. And thinking about, I mean, the great thing about a project like this from, you know, my outsider naive perspective is you can view it on so many scales. So as you said, you can view it as a network of individuals, but you can also view it as including a lot of geospatial data, which is incredibly interesting, or you can view it on the individual level as well. So there are kind of like a separation of scales there where you can answer different questions at different points.

Yes. I also find it exciting because with this I'm also learning new tools. And for example, I got to learn about nanocubes, when, which is this open source visualization tool for large spatial temporal data sets. And this was actually created at AT&T Labs and it's open source and it's amazing because you can use billions of data points and visualize them real time and you can also query it, like slice and dice your data as you please and then visualize subsets of it. So it's a lot of fun.

Yeah. I've actually seen several demos of nanocubes. I've never used it myself, but I think it was May, Simon Obanek and Chris Walinski who showed it to me originally, but I can't be sure. And actually, as you know, I had Chris, who for our listeners is, I think he's now assistant VP of Data Science and AI Research, where you are.

Yes.

So I had him on the podcast last year and we didn't discuss this in detail, but the first time I ever encountered Chris must have been, I don't know, five or six years ago at a conference. And he actually spoke about characterizing human mobility, which of course is a project you just spoke to. And he gave this great talk, which involves seeing when text messages stopped in a downtown neighborhood in, I can't remember which city it was, but text messages stopped at a certain point and phone calls started being made. And he realized, or his team realized, that this was when all the nightclubs and bars shut and people were calling taxis. This was before Uber and all that type of stuff, calling taxis. So from the data, you can actually see the emergent behavior of populations.

Right, right. And actually they even published a paper. This is why I love my work so much, because we also have the opportunity to publish. And he published these findings. Yeah, it's called Human Mobility Characterization from Cellular Network Data. And it's a publication, I think, from 2013.

Great. That was around the time I saw this talk, actually. So that would make perfect sense. So we'll definitely link to that in the show notes as well. But it's really cool to be able to publish this stuff to make discoveries about human mobility and characterizing that, as you say, but also, as you said, to leverage these discoveries to improve the services that AT&T provides people.

Right. And that links to another project of mine which I'm working on. And I also find it exciting because I can also use my network science background. And that project is about characterizing our mobile network and analyze how its topology compares to other reported real social networks out there. This initially just sounds like fun, but it's also very crucial for us to know how our topology, our network topology looks like because it helps us understand how certain dynamical processes progress throughout the network. And this implicitly also helps us improve our services.

Great. So when we're thinking of this mobile network, generally for our listeners who don't necessarily know a lot about network theory, a network or a graph, you'll have nodes and edges that connect them. So you can imagine on Twitter that. That all the nodes are people with Twitter handles, and connections are formed when people follow each other. Those are the edges. Now I'm wondering what this mobile network looks like. Is it people with cell phones, other nodes, and there's an edge between them if they call each other or message each other, right?

Yes. This is all anonymized. So the only thing that we're doing is using at aggregate level to see connectivities like number of connections and so on. So this helps the. Is understand the topological features. So, yeah. So basically a network has elements, and then the elements that are connected by a certain relationship, you can build this edge among them. And then this is how you construct your network. And the reason why I find this so fascinating is because networks are everywhere around us. And network scientists are many scientists from different fields because it's a very interdisciplinary topic. You have protein interaction networks, you have the brain, the neural networks, you have social networks, you have street networks. So transportation networks, power grids. So we are living in a very interconnected world and everything is networks. So for me, that's why it's so fascinating to work in Network science.

We'll jump right back into our interview with Noemi after a short segment. So I'm back here with Emily Robinson for our ongoing segments called Guidelines for Online Experiments. Emily's a data scientist on the growth team here at datacamp. Hi, Emily.

Hi, Hugo.

So we're talking about guidelines for AB testing. Last time we discussed don't run tons of variants. Your next guideline is related to this. It's don't try to look for differences for every possible segment. Now what do you mean by this and why is it important?

Well, if your test doesn't work overall, it's really tempting to hold out hope that actually it did just it didn't for everyone. So maybe you try to say, okay, our tests didn't work, but what if we look just at new visitors? What if we check only desktop visitors? What about people in the US or people on a certain day of the week? And you know, if you cut your data all these different ways, maybe something pops up significant, right? Maybe it's that you find, oh wow, well, it didn't work overall. But US visitors, we saw an increase in conversion rate. Well, once again here you're running into a problem, false positives by doing multiple testing. So there's actually a great paper called Testing Multiple Statistical Hypotheses Resulted in Spurious Associations A Study of Astrological Signs and Health. And what this paper did was that by testing all these different associations, they found that there was two common diagnoses for hospitalizations that had a specific, that had a higher probability of happening for specific astrological signs. And you know, so this was, for example, that people born under LEO had a higher probability of gastrointestinal hemorrhage. And of course this is not actually true. Astrological signs do not cause certain health issues. But you run enough tests and again, because of these multiple testing problems, something is likely to pop up significant and you'll just be chasing statistical ghosts.

Right. And something related, particularly in this example is that if you discover things in the data post experiment and you haven't designed it for that reason, you actually discover patterns that you see and then validate it using statistics.

Right, exactly. And something you can do is this isn't to say you can't explore your data at all, but rather you should. If you do find something that surprises, think about, can you run another test to help confirm this? And so just making a distinction between running a confirmatory analysis when you've pre specified hypotheses and an exploratory analysis. And with the exploratory analysis, you have to be a lot more skeptical with the results that you get.

Yeah. And another way of dealing with it is actually post experiments, splitting your data. So having a test and train set, essentially.

Yeah, so that's one way, and I'm not as experienced with that, but I think there's lots of different ways that again, as long as you apply some extra rigor to it, you can be more confident in the results that you get.

Absolutely. So your next maxim is check that there's no bucketing skew. So is this the type of thing that machine learning aficionados would call class imbalance?

So actually, in this case it's different. So class imbalance for ML is when you're dealing with a problem that say you're classifying emails as spam versus not, except maybe only 1% of emails are spam. And so it can thrive off some methods if you don't apply any corrections to that. Because, for example, to get 99% accuracy, all you have to do is classify everything as not spam. That can be legitimate. Right. It can be very true that there is class imbalance and you have to find ways to deal around it. Bucketing skew is something that shouldn't happen. And it's also known as sample ratio mismatch. What happens is that the split of people in your A B test between your variants does not match what you planned. For example, maybe you wanted to split people evenly between the control and treatment, but after a few days you find that actually 40% are in the treatment and 60% are in the control. And you can apply a proportion test or chi square test to check whether or not that this would be unlikely to happen during chance. Now, why this is a problem is that again, say your p value is 0.001. It's very unlikely the split would have happened just because of chance. It means that you have a bug, and even if it's just throwing off your split by a little bit, it can really mess with your metrics. You want to go back and try to figure out where is there a bug in the implementation. One good way of doing this is checking segments. For example, is it because of Safari? Even though I just talked about not checking segments, it's a really good way to find a bug. For example, all the other browsers are split evenly, and then Safari is 10% versus 90%. That's a good indication that you want to start looking at Safari specifically.

Right. That's very interesting. So, Emily, that's all the time we have for now, but I'm really excited about having you back next time to discuss more of your guidelines for a B testing.

Thanks, Hugo. I am too.

Time to get straight back. Back into our chat with Noami. So you said two things I just want to kind of tease apart. Briefly, you talked about how the topology of a network can be crucial to understanding how certain dynamical processes progress throughout the network. I'm just wondering if you could give us insight into what topology in a network actually means and what type of dynamical processes, for example, you think about.

So in topology wise, you want to see some basic features of the network, you want to see what is the degree distribution of a network. So that means that I'm trying to see how many nodes I have with this number of connections, how many nodes with that number, and then I'm just building up the distribution. And studies in network science have shown that real networks mostly follow this scale free pattern when it comes to their degree distribution, which means that most nodes have very few connections, whereas you have this small number of hubs which have an extremely large number of connections. And this is something you can see in Twitter too, right? So you have these very popular people who have hundreds of thousands of followers, but most people will have a very low number of connections. And then the other thing that you want to look at when you're looking at topology is how clustered the nodes are within the network. So, so is the network more homogeneous or do you see some more densely connected subgroups? Like for example, in social networks you will see many densely connected subgroups.

That's really important because this can give rise to the emergence of filter bubbles and echo chambers. You can imagine politically distinct groups that really communicate within themselves and read particular types of media, but not from the other side, for example, right?

And that's why it's very important to understand the structure of your network. Because before you start looking at how you can influence, for example, in politics, people, you have to first see what is the type, is the distribution scale free? Do you have these hubs or everyone has approximately the same number of connections. And because these dynamical processes are going to evolve in the network in a completely different manner based on what's the structure of the network. And, and to give an example of these dynamical processes, for example, if it's something that I worked on for a very long time, is cascading failures, which you can see in power grid. So if in any type of network where you have information flow, information flow, you can think of anything like me trying to convince My friend, to buy a product or power grids transmitting from one generator to the other current. You want to see in case one node fails in the system, its failure, how it's going to get transmitted further throughout the network. One thing that we need to take into account when you're looking at these networks that are transferring information is that nodes have assigned a capacity, like how much information can they handle? If one of the nodes fails, it's going to to reallocate its load that the information that it took over to its neighboring nodes. And now those if are going to have higher load than their capacity, they're also going to fail. And this is what they call cascading failure or an avalanche of failures. And this is a big problem because in power grid you have like very milli. Milliseconds to actually try to mitigate that failure. So what you're trying to do is to build a more robust system against that. But these failures are also very dependent on the structure of the network. So at the basis of every network analysis it comes like, what is that the structure of that network?

As you're talking about dynamic processes propagating through networks, it just sprung to mind, I know that Twitter is used by data scientists. So much for thinking about tools and techniques and problem solving and debugging and. And I was just wondering, thinking about if we could see how data science tools actually propagated through social networks on the Internet, which could be a cool project for a listener to do at some point.

There was a research project focused on how tools spread out and how popular they become on GitHub.

Ah, yeah, that's very interesting.

Yeah, so that's related.

You've told us about two of the projects that you love that you're working on. And you told us at the start though, you've got three main projects. So what's the third one?

Oh yeah, the third one I loved working on too. Also something that is brand new and very fascinating topic for me. So many people don't know that AT&T owns DirecTV. And now also with the acquisition of Turner, we have even more TV data that creates for us tremendous research opportunities in the TV advertising space that I'm still learning a lot about. And I find this very exciting because especially when I joined AT&T, even for me it didn't occur at the interview process that I might end up working with TV data. So it's really awesome.

That's really cool.

Cool.

This is a relatively new position for you, as you said. So I'm wondering, we've got a lot of listeners out there who are aspiring data scientists and working data scientists, and I'm just wondering what advice you'd give someone who'd be interested in this type of job. What types of skills and tools would they need to have?

Sure. We are constantly looking to hire, so I'm very happy to share that information for people who are interested. So since we are a research lab, we are looking for people with PhD because we see candidates who possess domain expertise and have research experience. And of course we love seeing people with genuine enthusiasm who are excited about new data and know how to get the best out of it. And of course this requires great technical skills, high integrity using the data, and of course to be innovative, as implied by our inventive scientist job titles. And last but not least, our research lab is a very collaborative environment, so so you can come up with project ideas or get involved in projects with other team members. So a critical soft skill that we're looking for is the ability to successfully collaborate with others. AT&T Labs Research also promotes academic collaborations. We can publish, as I mentioned, and also many of my colleagues have ongoing academic collaborations. So being collaborative in our field is a critical skill that we are looking for, for sure.

So we've talked a lot, we've bounced back and forth between current data science work that you're involved in, its impact on the future of AT&T. I'm wondering more generally what the the future of data science looks like to you.

It's funny that you asked me because I have a complete so yesterday I read the Forbes article saying that data science won't be around in 2029.

Great.

And it's very funny because I have the opposite opinion. So in my opinion, data science has been around for a while and since even before being called data science. And I think it will be around for even longer.

And of course though in 2029 it will be around, but it may not be called data science as well. Right?

Right. So it even before it was data analyst or like data mining data. Right. And then there are other people who are working with data. For example, while I was doing my research, the PhD, what was I doing? I didn't even know what term to give to it. I was just hearing from traditional physics professors that this is not physics and I never knew what to call it. So now we have a term for it and maybe in the future it's going to change. But the job itself, the role, I think it's going to be around for a long time because. Because this field requires sciency, innovative mindset, and I think there will be plenty of opportunities in this field in the future. And I think the part of data science that changes rapidly is the tools that we make use of, from how to ingest large scale data to how to evaluate and interpret predictive models. So this is changing very rapidly and that's why data scientists have to constantly keep learning to be able to keep up with the rapid technological advances. But the data scientist role in itself is going to be around, I think so.

And I think we're also going to see data skills, data literacy and data fluency spread across organizations in really interesting ways as well. I mean, something we're thinking about a lot is what do product managers need to know about data science statistics? What do VPs of marketing know? What does C Suite need to know? Do they need to know, you know, the basic definitions of metrics for machine learning models and maybe a bit about class imbalances for example. Right.

It's very cool because the data science fellowship that I participated in last year, Inside Data Science Fellowship, they also launched a product management fellowship, which is really awesome.

That's really cool. I'd actually love to know more of that. I'm going to look into that because I do actually think the relationship between product management and data science is not so ill defined the way we talk about it is, but it's becoming more and more important. But that's for another conversation. I'd like to wrap up with a couple of questions. I'd just love to know what one of your favorite data sciencey things to do is as a technique or methodology or anything.

For me, the data science process as a whole is my favorite because I always liked these crime novels or movies and I always feel like what I'm doing doing, I'm getting a data it's like throughout the eda, the exploratory data analysis, I feel like a detective finding these puzzle pieces in the data and then in the modeling part I'm just putting the pieces together to reveal the story. So for me it's like, oh, I feel like a detective in a safe space. I don't deal with criminals. But to answer your question, one methodology that one of the several of my favorites is probably it text data vectorization because it's so simple, yet I find it so fascinating how you can so easily extract features from unstructured text data with this very simple technique and you can use this for feature extraction in natural language processing and to build models from it. So I find it really Cool, that's awesome.

And although I agree that when performing data analysis and doing data science, you're not uncovering criminals, a lot of code people write and a lot of process is almost criminal as well. I mean, we're still establishing best practices also. And also I find it really interesting that unstructured data and text data, natural language processing is part of your answer to this because a lot of the techniques and work and research you've done we've discussed today isn't necessarily involved with text data. So that's kind of cool to know that this is another interest of yours.

And actually throughout my openness data set collection analysis, I did natural language processing and I'm also developing a course for Pearson which is called Natural Language Processing for Hackers, which is going to be out hopefully soon.

Okay, great. So my final question is, do you have a call to action for our listeners out there?

Yeah, check out our website at att Labs. It's about.att.com labsresearch to learn more about what we do and how you can get involved. Because we're always looking for young talent to join our growing team. And now I actually shared with you what type of skills we're looking for. So, yeah, we're very interested in new talent.

Fantastic. And we'll put that link in the show notes as well. And for all of you who do reach out, mention that you heard our conversation on Dataframe as well. Well, but Noemi, I'd just like to thank you so much for coming on the show. It's been so much fun.

Thank you so much for inviting me. It was great being here.

Thanks for joining the conversation with Noami. We saw the wide array of work that happens at AT&T Research Labs and what it looks like to do lots of science with lots of data there. Noemi works on a wide array of projects, from characterizing human mobility, from cellular network data to understanding TV viewer shift and how engaged people are in different shows. One of the things that struck me about Noemi's work and trajectory is how she has managed to find a position where many of her interests and skills are aligned. For example, working on human mobility characterization using network analysis, which she has loved for decades, and using principles and theories from physics and framing certain questions as being about the propagation of dynamical systems through networks. Now, now, the takeaway here, dear listener, is not to necessarily learn physics or view the phenomena you work on or think about as scale free patterning, but to study your particular skill set thoughtfully and mindfully. Consider what you're interested in and love doing, and look for work that marries at least some of these things. Now, I of all people know how difficult this can be, but it's good to take at least baby steps in this direction, even if you're not sure where you want to end up. We also saw that even if the term data science disappears or sublimates in the next decade, it will still exist in one form or another, just as it did before the term data science was spawned. It's incredibly exciting to see how data skills, data literacy, and data fluency will see increasing specialization and be spread across organizations and society at large. I for one, cannot wait. Next week I'll be speaking with Skipper Seaboldt about the current and looming credibility crisis in data science. Skipper is Director of Data Science at Sivis Analytics, a data science technology and solutions company, and also creator of the STATS Models package for statistical modeling and comput computing in Python. Scipper is also a data scientist with a bigger beard than mine. I know beard envy is a real thing, people. We're going to be talking about how data science is facing a credibility crisis that is manifesting itself in different ways in different industries, how and why expectations aren't being met and many stakeholders are disillusioned. We'll see that if the crisis isn't prevented, the data science labor market may cease to be a seller's market and we may have big missed opportunities. But you're right, this is not an episode of Black Mirror, so we'll also discuss how to avoid the crisis taking detours through the role of randomized control trials in data science, the rise of methods borrowed from econometrics, and how to set realistic expectations around what data science can and can't do. I'm your host, Hugo Bown Anderson. You can follow Datacamp on Twitter @Datacamp and Megobown. You can find all our episodes and show notes@datacamp.com community podcast.