GDS Digital Innovation US 22 Day 1 Panel

Sorry, Alicia, I know it looks like I'm sitting underneath your headshot right now, but not to be mistaken. Nice to see you all again. So to kick off our panel, I wanted to allow you an opportunity to introduce yourselves and then also say how you are currently leveraging business analytics and predictive analytics. Alicia, I'm gonna start with you 'cause you're right to my left.

Sure. Hello. Hi everyone. So I'm Alicia Outrage and I am a founder and CTO of Byte and Core. Byte and Chord is really my VC sort of studio backend for two initiatives that I lead in the AI machine learning space of building out, kind of taking raw data sets and building out AI machine learning models and curriculum at universities across the country where I'm a professor at Brown University as well as Hunter College in New York and Providence, Rhode Island. And then I work in the AI healthcare space, in the women's healthcare space specifically, building out and mapping an AI model for women's hormonal health. So, that is what I've been doing the last decade and focusing on AI healthcare, education, and the education space now.

Russell.

I'm Russell Hanson. Nice to be here. I'm Chief Innovation Officer at Sama Therapeutics. Sama Therapeutics is an AI neuropsychiatry company where we use AI avatars to do assessments and intake for clinical use cases in psychiatry and clinical psychology. Originally, I'm a scientist, a genetics professor at Mount Sinai, was in Boston at Harvard doing bioinformatics for a long time, and I've been working in AI for 15 years in deep neural networks and cancer vaccines and lots of other things we do with AI in my field.

Good afternoon, my name is Anusha Anantharaman. It's nice to see you all here. So I work in the public sector in the area of education, and my work is centered around innovation, change management, and modernization. So a lot of what Ann mentioned in her presentation resonated very well with me, it's all true. So the way we focus on our work is it's not led so much because by revenues or our KPIs are not driven by profit. So we work largely in the areas of change improvements that can be brought to families and students, and we also have to look at when the funds are coming our way and we have to optimize those times well and we have to align with that and move with what the administration wants. So that's an additional challenge. So we'll talk more about it as we go through all the questions.

We're saving lives from all angles here, from cancer, education, healthcare.

Perfect. Very true, very important industries. We're gonna kick off the conversation about some of the strategies that you were each applying to ensure that the data driving these predictions is accurate and actionable. And I'm interested to hear all your perspectives, especially Anousha. I know that you deal with legacy systems, but Alicia, we're gonna start with you. And how this can be done with limited data.

So I mentioned women's healthcare and building out AI models for women's healthcare. So one of the companies that I founded is The Metapause, and The Metapause is a personal GPS for women to kind of navigate their hormonal health. So that covers perimenopause, menopause, PCOS, fertility. These are all things that either many people may or may not know has very limited data. And understanding. The best data sets you're gonna see is probably talking to other women in a Facebook group and asking what works for them. Because on average, when women have issues related to their longevity or health challenges, the statistics show that it's their 9th doctor that sort of helps solve that for them. And not many women necessarily see a 9th doctor, but then most importantly, a lot of the studies on issues related to women's hormonal health and challenges, they just kind of don't exist or they're very small. So when it comes to small data sets, like that's really what I'm dealing with if a woman has sort of an open ChatGPT box type of experience to just ask a question of why is my skin itchy? How do I— why do—

why is my—

like, why am I losing my hair? That is such a generic— that could be anything. And so what we do essentially, and the key takeaway that I've gotten from that and that I've used in other industries, is that less really, really good data is actually better than a whole bunch of AI junk. And so the way that we structured that in the healthcare space is a combination of, number one, make your X data set Make a really good dataset, even if it's a few hundred really good things from a bunch of experts that have said, this works for my patient, and I'm gonna tell you this worked for my patient really well, and here is like some background on that, some federated clean background on who this works for, these solutions work for. That's number one, get a really good small dataset. Number two, get adjacent information. So that you can then take that adjacent information of, say, hair loss and aging, and then get information about, well, what are the related health issues that could result in hair loss in general? Compare that to be able to generate synthetic data, and then do a Bayesian modeling concept to be able to just create concepts. Created another example dataset that you can test to just kind of make better data, but constantly and continuously kind of check and balance that against your really, really good data. The next thing is federated data. It's like kind of getting really good federated datasets and partnering with other businesses so that you don't necessarily need to make your own data. You just need the insights for like what is— what could cause hair loss in general? Is it— what are the biomarkers that are related to that? What are the tests that are available for that? Because then you can get more of that adjacent data that you trust to be able to then generate good data. And so like that's a whole bunch of technical stuff that may or may not— I saw some nodding heads on just like kind of what you can do. But the key takeaway is get some really good small data sets that you can trust. And then get all the information around it to be able to build more inferences.

Russell, it looked like you were deep in thought. I know that you work in the healthcare industry, as you discussed. So one of the things Alicia mentioned was also chatbots. And I know that we discussed how to have that proper tone, especially in a sensitive industry and with consumers.

Yeah, I mean, Alicia mentioned, you know, a lot of different good points. You know, you really want to be able to Validate your chatbot in kind of a controlled setting before you set it loose. There are lots of examples of chatbots giving bad information and that having adverse effects that can be very serious. Putting a sandbox around your chatbot, telling it what the guidelines are. If there's dangerous information coming out, particularly in clinical psychology and psychiatry, if someone is threatening to harm themselves or somebody else, you need to escalate it away from the AI immediately. There are liability issues as well. But in general, these are very evolving areas. And on the internet, people are doing all kinds of crazy things on character AI and ChatGPT and all the other chatbots. And when you're kind of moving from a consumer space to a clinical space, there are lots of evolving guidelines. Every couple of years, the NIH releases new guidelines on how to develop AI systems for measuring cancer or measuring tumors or other types of diseases. And they want 99% and better accuracy, but things are always changing. And particularly when you're getting into language and behavior, Things are, you know, people's behavior changes, their environment changes, and it's kind of a, you need to be careful.

I love that point that you brought up because sometimes you don't, like there is an answer and you don't wanna say it because A, it'll put you a whole bunch of risk. Like somebody says, oh, I don't like how I look to how my face looks today. Like I look older, I don't like how my face looks. And it's like, well, you can get a whole ton of surgery. Here's how to solve that. Let me go pick out all these things about your face that you can fix and here's the way that you do it. Tap Plus to be able to check out these, these, all these surgeries now. Technically, that's an answer. And if you don't tell, put some guardrails up to tell AI, don't tell the user that, like, hey, maybe tell, ask them like, what, what is causing this? Or like, is this cosmetic? Or is this, did you, do you just feel stressed about something else going on?

Yeah. Or in the case of psychiatric medicine, you know, you know, what drugs should you be taking? You know, your chatbot shouldn't be doing that. They don't have access to your medical history. They don't know what's going on with your brain or your body or your other health issues. It's really a delicate balance to do health advice on a chatbot.

Definitely. In the case of what I'm building, we actually do have a sandbox of the user's medical data, which they can sort of take with them and get guides. On just how to talk to their doctor and making sure that like a chatbot isn't giving medical advice because that's what people are doing. And like ChatGPT has definitely done a great job in setting up those guardrails that he mentioned so that it says, go see a doctor, stop talking to me.

That's where my mind went immediately was in the very beginning of everyone talking about things like ChatGPT. There was a lot of warning signs about it providing health information that wasn't accurate. One of the things too that Anusha mentioned was the legacy systems and working at a government agency. I know that that is something that you also have to battle and are still kind of in your journey of how to use predictive analytics. So how do you start to balance that and what steps are you taking to replace those legacy apps?

Thank you. That's a great question. So one of the things we have to remember is like I work in one of the largest school districts in the country, so we cannot afford any downtime and any operational interruptions. So the way we look at it is like where data's concerned, the quality of data is very, is dismal and it's got to do with the legacy systems that didn't have like, there was a lot of free text allowed, a lot of not too much validations there. So the data we've collected is kind of iffy. So, While we talk about, I'll come to legacy systems, replacing those systems later, but what we've done now is we're trying to improve the data intake at source. So we've been training our schools, we've been training our vendors, our contracted vendors, be it for busing or be it for food supply and facilities. We wanna make sure they're getting the right data at source and we've set up Power BI tables and analytics So our operational areas get data every day and they can make the improved data-driven decisions to improve operational efficiencies, be it reduce the delays in busing, making sure the food arrives to school on time, whatever the case may be. So that's one effort. When it comes to legacy applications, there's so many legacy applications that it can get overwhelming. So we have to prioritize what needs to get replaced first. So we look at the business areas first. We see whether the current application is driving business value. Is it delivering any efficiencies? When it comes to IT, we also have to see what is the risk, what is the cost, and what is the value it's gonna bring to the table? Because when it comes to the government, you don't have a flood of resources and money comes in bits and spurts. Like if there's COVID, you'll get some COVID assistance, then you decide, okay, can I use it on an application? So there are ebbs and flows. So you also have to think of how you'll phase it out in a way that doesn't interrupt operations at all. So you'll build it in different capsules so that if the funding stops and you have to take a pause, it'll still be okay. So these are the things we take into consideration. And then after that, we also think of, you know, does this need to be re-architected or does it need to be rebuilt or replaced altogether? You know, rebuilding is somewhere in between, but replacing is obviously the most expensive option. Or maybe we do some recombinant innovation, like takes, for example, like you, You, something like the busing, let's take busing as an example. So first you'll equip the buses with GPS, and then you'll think of, okay, how can we connect GPS to the routing data and establish that digital connect? And so you think of how you can take old things and kind of give it a little push towards digitization and make it a recombinant innovation. So this is how we think of legacy applications, but the biggest challenge is integration. Like different legacy applications are in different stages of being rebuilt. So how do we make sure that they're all getting connected? How do we make sure that the cutover happens at the same time? So that it is indeed very challenging and working with several vendors across the board because you have to put out RFPs And you gotta choose vendors, they have to be onboarded at the same time and they should all work in sync. So it is very challenging, but at the same time it kind of, it's very rewarding because you get a chance like this every few decades to change everything across the system. So you're literally creating history. So it's, there's a lot of pride in doing that. Doesn't come without challenges though.

Russell, Anusha listed a lot of challenges, whether that be the data quality, the cost, the risk. Something else that we've discussed is the regulations and privacy component. How do you balance that in the healthcare industry in order to keep using predictive analytics?

Well, when we were doing more imaging, you know, imaging cells and tumors and things, it was a lot more cut and dried. And as things have moved towards large language models, I've seen much less guidance from regulators as to what you can and cannot do. Some companies are focusing on subclinical applications, so at a level of seriousness where you should not see a doctor, you're kind of just looking for informal advice. And exactly where you draw the line from subclinical to clinical is usually whether it's having serious effects in your life or the lives of those around you for psychiatric medicine. But it's really— I wish there was— things are changing so rapidly. Every month there's a new deep-seek or model. And it's just really a time of rapid change right now.

Yeah, Alicia, things are changing rapidly. It looked like you were about to add on to that.

No, yeah, I was nodding in agreement because it's kind of like I— so I used to be a senior executive at Facebook. Facebook, back when it was Facebook. I was never a Meta mate. And like, and that was about like when Instagram ads was first made. And so we were navigating the personalization and privacy space, doing moving fast and breaking things, as was said earlier. And then I couldn't talk about my work for 7 years because it was reviewed by Congress publicly. And what I learned from that was that in a rapidly changing privacy ecosystem, you try to do your best. You might not do well. And like the takeaways that I kind of take from that now is that you generally, like you don't just put the answer, put the decisions in the user's hands. Like you need to really kind of be very proactive on securing things and not saving. Information. And a big part of what I was doing at Facebook at that time was not saving information. Don't save the location of a user. You know where they are, you know who's around them. We didn't save it, and we can use it, we can leverage the insights of what's around them, but never save the information. And so what I try to do when I think about the other spaces that I'm working in is don't— like, learn from the insights and like try to be as proactively private as if everything's gonna get locked down as much as you can. And don't ever have, even if the AI thinks it has the right answer or it makes up some answer that seems right, it's not the answer, it's a suggestion.

Well, Alicia, like Ann said in her keynote, to not just go breaking things, how do you communicate to decision makers then to make sure that they're comfortable taking those slower steps and taking actions from predictions?

Well, I'm really lucky that I'm the decision maker. So the hardest problem I have is really the other part of what Anne was saying about getting others to understand what I'm doing. Like, I'm doing all these modeling with my team, and like, maybe they— like, all this stuff is really new, and like, you're learning things as you go with some of this, even if you know what you're doing. So bringing everyone else along has been a learning experience for myself. And I've actually found that the most junior people on my team have such amazing insights to figure out things in this new ecosystem in space. And so if I was going to kind of scale that to your question on how to bring stakeholders on board, as well as how to bring the rest of your team on board, because you can't work alone.. And you can't have the person who knows, who has a PhD in AI, machine learning, engineering, and data modeling, be the only person knows what's going on. You need to have visuals. And if you can't visualize what you're doing in a simple way, something's wrong.

I guess for us, really, we've focused on solving kind of more lower-hanging fruit. So automation, intake, assessments. So if you want to see it, a doctor first and then the next 2 weeks are AI assessments. Kind of just going gradually into an AI process has just been easier for a lot of folks to stomach or handle. So, not having the AI as the supervisor, but having AI as kind of the minion and taking care of repetitive, boring things that people don't want to do and letting the AI take care of those and having the people with the licenses and the credentials to do the high-value work still, for sure.

Anisha, I saw you nodding along too.

Sure. So I think for us it's more challenging. Like, I don't have a situation like Alicia where she makes all the decisions. And data, unfortunately, is— I think the value of data is misunderstood because The quality of data is also understated in my area of work. So it's important for me to talk to my stakeholders about how data is an atomic unit, not only in the age of artificial intelligence and everything else, but it's the single most important driver of value creation. So in all, we are focused on improving, enhancing customer service in the sense that we want families and students to feel very satisfied with our service, and we have strong advocacy from parents. We have strong advocacy from education policy leaders. So, and we have to meet them ever so often to get approval on any contracts or any funds we want to get from them. So they are always up in arms telling us what is wrong with our system. So we, what we do is we try to find out what are the biggest pain points and where these complaints are coming from and what are the, which of these complaints carry the most weight, like what is the 20% that can make 80% of the difference? And then we try to propose an idea about how we're gonna plan to bring that difference along, and because it's very hard to get large funds of approval at the same time, we try to go with proof of concepts, we try to work with vendors that are willing to, go on trial with us. We take a small piece and we show what we're gonna do with it, and then we also add some key measuring parameters to it, and we explain how there's gonna be a paradigm shift. And that's how we talk to our stakeholders. And the other thing we keep in mind always is that we wanna make sure that the core value is not disturbed, the core offering is not disturbed, because that really bothers people who are resistant to change, but we try to add value around it, value creation around it through digitization. For example, like, I mean, like how if we were to think of another industry like the cab service versus Uber, right? So the core offering wasn't disturbed at all. It was just taking a person from one place to another, but what made an Uber Uber or Ovia or several other companies successful is how they connected with the consumer, how they connected with the rider by giving them a lot of information ahead of time, or how they used predictive analysis or machine learning to kind of make them feel that they're giving them personalized offerings. So these are the kind of examples we use with our stakeholders about how we're trying to create value around the core concept without changing the core concept that's because the core concepts are part of a chancellor's regulations or policies that have been written far back.

I love what you said because that actually reminded me of what at Brown University where we had to— because I'm not the president of Brown, so I couldn't just say, hey, let's do this. But they were building out an AI literacy program. So I was making that curriculum with them. And I said, you know what, we used to have radio stations where students would go work on the radio and they would learn about new media and they would be able to be employed in the entertainment industry because they knew how it worked at a broadcast company. That doesn't exist anymore. So this AI literacy curriculum, like, you should have a startup, an AI product that they work on, radio, college radio style, like group project concept. And then I found the radio station, so like, and this is an example of find the innovative or sort of pocket of money that's sort of a legacy money, or like there's a business reason that like that department or group needs to innovate, because they just had money sitting there for the college radio and media department that was not used, Because Gen Z doesn't know about radio. All they do is just type on the— well, type on TikTok, swipe on TikTok. So they're consumers. They don't create. So long story short, I found the innovation— that group in the radio and media department, they had that money. They invested in us building out the visual prototype. And then we demoed that to the students and got the students bought in. 'Cause all the old people like me and the professors knew what radio was and what college radio was and how we learned about media. The Gen Z students, they saw, oh wow, I could work on a personalization and recommendation engine and it could connect me with other people and I can build out and make money for our school and our department to do stuff in the clubs using this concept to sell advertising and know how this works works. And so that course is going out, and that's something that— I mean, granted, it took a year to do, uh, and it only took about a week to build the, the prototype, but that's how we can actually, like, make something visual, like with your example, and, like, show it. And you can do that really fast now with AI.

I also want to take a quick pause here and let the audience know that they are free to ask questions. We have some microphones roaming, So if anybody has a question, I am looking out for any raised hands. Does anybody want to ask something yet? Are we feeling a little shy this morning? I guess it's afternoon, huh? Long travel day, huh? Russell, I'm going to toss it over to you real quick then while we wait to see if anybody in the audience has some questions.

Well, I was just wondering, you know, we've been looking at AITAs, so a teaching assistant who does office hours and, you know, can be equipped with like the the materials for the book or the lectures or whatever. Have you seen any good examples of an AITA being adopted? And so we were sometimes joking about, you know, you have Einstein teaching you physics or Winston Churchill teaching you diplomacy or something. And whether the cognitive aspect of talking to Obama about politics or these notable historical characters this would be really good for education or too distracting or—

I built my own ChatGPT agent for my students, actually, because they kept asking me the same questions over and over again. And I just took the recordings from all of the classes. I was like, yes, this is when the midterm is, this is when the final is, here are the questions on it, here's the information, and here's the link to the slides, and then incorporated that into Canvas. Which was our, like, it's kind of like Blackboard. It's like the virtual classroom product. And they liked it, actually, because they were, some people were shy to talk to me. And they, like, I have a, nobody would go to my office hours, but they would sneak up after class and talk to me, because they didn't want anybody else to see.

[Speaker] Yeah, exactly. We have the same thing that happens, or we're investigating with clinical trials and AI avatars. So if you have a therapist, the therapist charges money per hour, and the patient has to pay. And there's kind of this issue of wanting to save face both for the therapist and the person, so that takes time to develop a therapeutic rapport. And so to what degree is an AI better than a person because the person doesn't have to pretend they're something they're not, and the therapist doesn't have to, you know, keep inviting them back in next week.

Outside of the productivity component, I'm also hearing this involving external people, so consumers, and then there's this balance of how do you make sure that your consumers are comfortable with the predictive analytic component? Are you transparent? Are you not? Do you find that there is a challenge with that?

Anousha?

I think the best thing is to involve the end consumer in that experiment because they are, it depends on what drives your business. Like my area of work is driven by consumer satisfaction and even if they are biased, but if they're loud enough, they can affect your life. So what we do is like we have certain, we work with a lot of vulnerable populations. For example, students in temporary housing, students who have just been displaced. We had a whole lot of students that moved from Texas last year to New York. So we have students in foster care homes. So they are our vulnerable populations and we wanna make sure that we have enough room for them in our school system. So we use a lot of predictive analytics there. We follow the numbers closely. We see what the trends are, what the patterns are, and also the geopolitical climate and what we can expect affect in the near future. So we kind of include those representations or advocates for those populations, and then we explain what we see the trends are like, and then we see if it resonates well with them and they kind of agree. And if they feel that we are skewed or our data, predictive data is not in line with what they think, we kind of correct it. But for us, it's about taking the end user along with us.

We do have an audience question on the right side. I think there's a few here.

We have a lot of questions.

Yes. Ramesh Natarajan. This question is to Alicia. Thanks for doing the great service.

I'm here. Hi. Great. I was like, I can't see.

I'll probably stand up. Menopause is not geographically restricted, right? So how are you planning to scale your organization is the differentiation given that you said you're looking for a very small data set? Because once it goes out of US, then it could be a different data set you need. So how are you planning to scale it up?

Is the question how am I— it's a small team and how are we planning to scale? Got it. That actually brings up the kind of related to what she was saying, where it's bringing the user along. So the women's health and wellness the wellness and longevity community is highly integrated and on board. Like, that is the first cohort, all of the quote unquote biohackers and like women that are like trying to actively solve this problem, that they're going through it right now. They're solving that in Facebook groups. And so now we're essentially giving them sort of their personal copilot to kind of prompt engineer the model and build it themselves. And we're saying that like, hey, we don't have all the answers. We're your copilot. This is your personal sandbox for you to ask questions. This is secure. You can plug in your, your information and then you can take it with you wherever you want. And like, kind of get guidance and guidance and suggestions to help you feel more comfortable when you're talking. So that's like kind of the scaling on the data side, scaling on the sort of infrastructure and security side and all of that is really kind of So that's kind of why we aren't live to the public right now because we're, it's very like kind of building from the start at a highly enterprise security level, integrate the integration and set up very small focused cohorts focusing on specific symptoms to get really good at it and partnering with boutique clinics and doctors.

I know there is a couple hands to the right, somebody in a blue shirt.

Yep.

Thanks for using nice microphones. So quick question. As I was sitting at lunch, I was having conversations around, you know, most people that maybe haven't even moved data to the cloud yet, right? Which we were talking about in 2007 when AWS came around. So as I was speaking with most people, they're trying to figure out where they're going, how to start their AI journey. And most, some may have not started their cloud journey. What would you offer to some of these people that are looking to get started in their AI journey? Because it sounds like you guys are well on your way. What would you offer to those that are just getting started?

Anusha, I'm gonna toss this to you because I know that you are a little bit earlier in your journey.

Yeah, yeah, yeah, yeah.

Sure.

I mean, we're still hybrid cloud, so, I mean, so yeah, it is, I think you should start with, like a very clear implementation goal, and again, a problem that's widespread in the organization or that has a lot of weight both operationally and politically. Solving that problem would give a lot of people sense of satisfaction or peace. So look for a problem like that, and it's a small problem that's gonna make a big difference, you know, that kind of a problem. And be very clear about what the goal is, because then you could get into more trouble than you can imagine. So you, what is the goal here, what is the end goal, how would you measure the success? And then once you've set that aside, I think the goal should be to get a set of clean data in place. That is a big task, more taxing than you can imagine, because we are all dealing typically especially in my area of work, we're dealing with data coming from disparate systems and of different quality. So we decide that, okay, this is the area that we're gonna clean, start with clean data, and we have a clear idea about what that data means to everybody, and I'm getting into details because it's important, even starting from the data dictionary to everything, everybody has a clear meaning about what that data means, that's what that set of data, we have to make sure that it covers covers all the use cases, all the scenarios, so that the end result is not skewed in any one direction. And everyone's understanding of that is the same, like the end goal is the same, the data covers all the scenarios. And then we start off with a proof of concept, like I said, it's like, or a pilot. Say, okay, we're gonna cover this, and this is gonna take this long, this is how much it's gonna cost, what it's gonna cost, and they were gonna pilot it with this audience. I think you have to start small, and you gotta convince people because the investment is pretty high, and you gotta keep convincing people with every stride in that process so you keep the investment coming in continuously. I hope it helps.

We have time for one more question. I know that you've raised your hand a few times in the front. Are you okay without a microphone, if you don't mind yelling it?

[INAUDIBLE] I just had a question. All of you are in education in some way, in K through 12, university education. And you had mentioned that AITA, that intrigues me. I'm part of education myself. I'm curious as far as the future and the personalization of education and if you see a point at which we see AI working with each an individual student as an assist within their university?

[Speaker] Yeah, we have a couple of projects on that over the last couple of years. There's some companies working on education in Africa, and one of the interesting things that they're working on is figuring out a way for the AI to read the student as to whether they're focused or unfocused or what their particular learning style is. So we see there's kind of a, a feedback loop of figuring out what kind of student it is and tailoring the education curriculum to that student and also being able to maybe read their biometrics or whether they're playing with their hair or doodling or something. So you need to kind of keep it interactive and using a camera you can see what they're doing as well. So yeah, I mean, ideally you'd be able to generate curricula that's most suited to the student's learning style and, you know, all the other things that you want to tailor for that person.

I have a very brief answer for once on, on just, uh, in my classes I let students use as much AI as they want, but all my tests are verbal.

Very interesting. Alicia, Russell, and Anusha, thank you so much for having this conversation. I thought it was interesting too when Anusha, you were talking about leadership And Alicia's like, well, I am that person. If only we could all have leaders that were ready to implement change. Thank you to my panelists for having this conversation. They will be out and about, so you guys feel free to say hi.