How the Govtech Industry Should Bring GenerativeAI Into Government

All right, TGIF, and happy December 1st to everyone out there. Gosh, it's hard to believe, Joe, we were just saying, you know, Happy New Year seemed like a few months ago, and here we are now rounding the year off strong. Today we're gonna dig into everyone's favorite topic as we look at what's trending across the country. There is no shortage of conversation and editorial and new features on generative AI, And in order to talk about it, we thought let's bring on actually a company that is leading the charge as it relates to this. And so we wanna look at how the industry should be approaching generative AI in government. And here to help us dig into this important topic is Bratton Riley, CEO of CityBot. Bratton, thanks for joining our virtual stage here.

Thanks for having me, gentlemen. It's good to be back with you.

So we want to nerd out on GenAI with you, but we want to kind of dig into the hard questions. Everyone talks about, you know, the, the nice, you know, kind of great things that you can do, that what I call the party tricks. But, uh, we got a lot of, uh, a lot of meat in today's show, so I'm really excited about it. But first, let's learn a little bit about you, Citibot, and the work that you're doing in the GovTech market.

Thanks, man. So first of all, we build chatbots, multi-channel chatbots for customer service, typically for local government. We do it across channels, so we do it via text, through the local government website, through social media channels. We're about to add WhatsApp, and then we also take an enterprise approach to government. As I tell cities and counties, the last thing you need is another siloed software system that doesn't connect to another system. So we we put a lot of emphasis around integrations into the systems of record. That a city or county government might have. We gotta respect their tech ecosystem and let the chat data flow, you know, through the system as the staff is accu— is accustomed to, you know, experiencing. That's very important. And, um, so we do it in 71 languages now too. And we are a mission-driven company. Our mission is we wanna serve all residents regardless of how much money they make.

Mm-hmm.

what their skin color looks like and what language they speak, that every resident deserves to have a high-quality engagement and customer service experience with their government. And we're just helping, you know, kind of hit the easy button for these local governments to kind of execute on that.

Yeah, we were just joking, it's December 1st, we're getting into the rush of the holiday season, everyone's looking for the trending gadget, the cool gift that's gonna stand out. Well, the The trending top topic in state and local government right now is generative AI. Just came back from an event here in Sacramento last night. CIOs were talking about generative AI. Really interested to get your take on what are you seeing around generative AI adoption across the state and local landscape?

Well, it's, you know, it's interesting. You know, we started in this AI world 7 years ago, and I joke, it was like the first 4 years we were selling a product that Governments didn't even know you could have, which is not a great sales proposition, by the way. But anyways, you know, I think with the announcement of GPT and the, you know, it becoming a word of our vernacular, and that was about a year ago almost to the day.

Yeah.

You know, as you all know, so much has happened and that has really increased awareness about AI and how it can be helpful. which it can, and how it needs to be safeguarded in proper ways, especially as we're talking about a government use case. So it's very exciting. You know, the technology's evolved so much over 7 years, and we've just tried to be on the ride and kind of try to stay ahead of it as much as we can.

So as you think about, you know, generative AI, there's so many different flavors of it and so many different, you know, new advances that are there. When I often start probing people into like, what is the killer app of generative AI? You know, it kind of is like the party trick stuff, right? Like, I had to write a poem for someone, or I had it, you know, create a children's book, or whatever it might be. But in your opinion, as you think about like generative AI in our market, like, what is the killer app for government?

Well, you know, we're using, you know, a GPT-based model in our product, and The reason why is we feel like it's the best in the marketplace. But, you know, we believe that these language models are going to become commoditized. There's going to be many of them, you know, just like there's Google and Bing and, you know, other search engines. We believe, you know, and obviously a lot of the big companies are working on it, but also, you know, some open source technology options are also Bubbling up to the surface, so you know we kind of structure our software where we're not codependent on any language model. We don't want to be because we just want to let the technology evolve and make sure that ours is you know kind of using the best engine that's there. But so that's what I would say. You know, believe it or not, I I don't I might not spend as much time researching all these companies and models as one might think because we're pretty busy over here. But that would be our, you know, observation at this time.

So as you see that evolution taking place, you see the advancements happening across the solution set, how is that contributing to or, you know, changing the constituent experience, particularly as it relates to, you know, the chatbots of today versus the chatbots of yesterday?

Yeah, so as I like to joke, all governments are on some kind of continuum to become a more resident-centric organization. Some haven't quite figured that out yet, but they're still on the continuum. And so, you know, how we like to approach them is just like, what's your goal? You know, and is your goal aligned with ours? Now, we're soon to build an internal product for the City of Modesto, California, for internal chat and their hosted SharePoint site. But for, you know, from an external perspective, it's like, You know, what are you trying to do? You know, and for us, it is we want to help local governments serve more residents, save a bunch of time in serving more residents, and save the residents a bunch of time in waiting to hear back from the government. Because sometimes it's not fast, and sometimes it is. Just depends on the size of government, how much they've invested in, in customer service. So as it relates to You know, our vision, you know, we're, we're looking at this through the lens of how can we help governments more equitably show up for their residents. And that's, you know, the, the, the digital equity term, in my opinion, is very overused and misappropriated, in my view, because we kind of associate it with Laying broadband cable around the country, which is very important. But it's like we like to talk about digital equity in terms of equitable access to the governmental institutions themselves. You might have the best Wi-Fi in the world, but if you're not getting equitable access to government, then then there's work to be done. So that's our approach, and that's why we're on these different channels and why we have brought in all these other languages, and we can track the languages that are being spoken for each city. It's really like. You know, y'all know I grew up in city government. My dad was mayor of Charleston for 40 years. Like, I understand the, the importance of building an equity-based government to build trust. And so, and Deloitte did a study a couple years ago and said a really good way to help build trust with residents is to create positive digital experiences for them. So that's exactly what we do. But in terms of how the technology has evolved, You know, we started with glorified, you know, targeted Google API searches to a city or county website. It wasn't AI. We built conversational AI with that for residents to be able to report service requests, potholes, missed trash pickups, etc. But so much has evolved, and the way it really We think about it in terms of the user experience, of course, first. And then we also think about it in terms of implementation and what are we asking the governments to do to help us implement our product.

Mm-hmm.

And so how it's evolved is, you know, we can turn on a generative AI QA model ingesting the government website. Pretty fast, like in an hour, which is pretty amazing. So, and it's, you know, answering thousands and thousands and thousands of questions. So prior to that, you know, we were using these machine learning models where we're bringing in the FAQs that a city or county might have, and, and the ML models would figure out how to create, you know, 50 different ways of paraphrasing the same question that would return the same answer. And that's very clumsy. It's very clumsy. So we don't have to do that anymore. We don't need an FAQ from local governments. We don't need them to do anything. They've already built their knowledge base. And as we tell local governments, no one wants to navigate your website as hard as you try, as hard as you try. And it's the nature of local governments. There are these collages of agencies that provide these amazing services that don't have anything to do with one another. They just happen to be associated in a local government, like booking your kiddo to the winter hockey league, or paying a parking ticket, or fixing a pothole, or getting a business license, or finding a lost dog. None of these things have anything to do with one another. And so Being able to take the existing knowledge that is being presented through a website and to be able to turnkey answer thousands and thousands of questions, you know, is a major shift forward. So, you know, some of the older chatbots— we never did this. We've always built our chatbots custom for each customer. We don't put that responsibility on our customer. We don't put the responsibility of having to maintain it or make it better on our customer. That's just, in my opinion, that is being insensitive to how busy the government staff is, whether it's on the IT side or the communications side, et cetera. No, we've got to be able to build a custom turnkey product and integrate it into their backend. Like, that's our responsibility. And then we maintain the responsibility of fixing the product and making it better and all that as people are using it. So now, you know, from an external case, we can control the data coming in. And we only use publicly available content generated by our customers.

And that's it.

And that's how we control the environment. And, and we can turn that around really fast. But then also what we do for our customers, just like you might have the best technology in the world, But if people don't know about it, they're not going to use it. So we help our customers brand the product to their brand, build a marketing and awareness campaign, let the residents know, hey, here's how this, this great way for you to interact with your government on your time. And so that's a big part of what we do. But the, the speed of implementation and the quality of the Q&A. Uh, is really amazing. And, and, you know, generally speaking, you know, cities or counties that have a 311, they track incoming, you know, data around what are people calling about, and it's usually about 70% questions. Uh, in our product, we're seeing more like 80% questions. So it's just like, you know, we talk about service requests and 311 and reporting potholes and abandoned vehicles, but most people are just trying to get answers to questions and they don't necessarily want to talk to anyone. They want to get it fast and move on. And so we're able to deliver, you know, a high-quality product that does that now, given the knowledge base that governments have already worked on. They shouldn't have to build a separate knowledge base for chat or have the responsibility, if it's not working, to to make it better. That's just— that's more anxiety on government staff who have enough as it is, uh, and, and the last thing we ever want to do is add to that.

Yeah, amen to that. There's a ton of, uh, you know, workforce challenges that Joe and I have talked about over the years as agencies haven't reached, uh, you know, back to pre-pandemic levels at the local government. We have a question here actually that just came in on LinkedIn that I'd love to get your thoughts on too. I mean, as we think about GenAI as a new predictive model using ML data, do you think, you know, we can create a sort of GovTech semi-open source database with previous threat models, resolutions, real-time zero-day threat and analytics? So I guess that's kind of part one. Part two is, you know, the responsibility dynamic. Is it, you know, responsible to create it as a SaaS, or should it be, you know, free and open source for state and local government? So maybe we'll— break that into 2 different pieces and kind of get your thoughts on the first side as we think about, you know, the, the, you know, open source nature of this and, and, you know, sharing kind of those types of models with other jurisdictions. Where do you see that kind of fitting into the, the, you know, kind of growth of this technology?

Well, I think, you know, I could take an hour to answer this question, but I think in general it's It's how do you enable these models to be government safe?

Hmm.

And that means you got to take out the hallucinations, you got to take out presenting any kind of personal information to an AI API. And in our case too, it means you have to source every answer back to the government content. You have to. The people need to know where the information is coming from, and then you know if they ask a question about how to get accessory dwelling unit permit in Roseville, California. Well, we'll provide a direct answer with a four-step process to do that, but then there'll be the link where you can click on it and then click on the link in there and start the application process. So we we're not trading. We're just not trading in content that would, you know, have any kind of nefarious intent. We're just using the website. Now we can use PDFs too on websites. As I joke with local governments, I long for the day where there never exists a PDF on a local government website, but that day is not today.

Yeah.

So, and there's some important content that gets put into these PDFs, so we have to be able to To search those too. But I think, you know, in terms of a threat, we control that by, you know, we're in the AWS cloud and, and we don't expose any private information to any AI API, period.

We don't do that.

And we don't, we don't transfer our data to any third party, period, end of conversation. The data is our customers' data.

Yeah.

So those are some of the safeguards that we have in place that make it government safe, and that should be the responsibility of the vendor to present products to government that we have already tested. We've been messing with the GPT API for 2 and a half years, but there's a reason why it took us 2 and a half years, because we had to apply our own, um, our own, you know, criteria for what we consider appropriate use case for the government. And so, all right, so how about the second question that was open source and—

Yeah, I think the second part you kind of tackled there too is you talked about the role of industry with deploying this as SaaS versus kind of the open source side. I mean, I think, you know, we've kind of seen that dichotomy as you look at, you know, some of the moves that some of the larger models like Meta have done versus, having, you know, these types of solutions packaged up and made turnkey? And I know, you know, as you said, it's kind of the responsibility of the industry community, in your opinion, to do that. But anything else you'd kind of add to that as you think about the delivery mechanism, SaaS versus, you know, open source, free, state and local?

Yeah, I think it's the ultimate question for a local government, you know, IT department in this case might be, you know, buy or build. And, and if we build it, who builds it and who maintains it? And what happens when they move to go work in the private sector because they can make a lot more money there, or whatever it is? It's like some governments will use open source to build their own. But, you know, we— generative AI is a component of our product. It is not the only component of our product. And so When you're trying to figure out what is the goal, what are we trying to do, you know, we see that more as a SaaS play. Even if you are using open source generative AI technology, it's, it's, it's a piece of the overall pie.

You know, you kind of broke some of it down just a second ago, but are there additional considerations or challenges that you see from the, from the government side around generative AI potentially moving too fast or finding the best, you know, the best place to start? What are the things that you see them weighing or addressing before going all in?

Well, you know, some cities like San Jose have presented a generative AI policy framework, and they've done a really good job with that. And so if you're a local government and looking to have a generative AI policy, check out San Jose's. I think you can just Google San Jose generative AI policy and it'll get you right there. You know, I— we separated into the internal use case, Joe, and the external use case. I think a lot of these policies are being driven around the employee utilization of it. In the work that they do on a day-to-day basis? And how do you, you know, if you're using like a pure language model, GPT, like, how do you take content that is being delivered to you that you can't source, and you don't know if it's true or not, and apply it, you know, or how do you edit it? Or how do you reference check it? you know, to use for internal use cases. Maybe it's creating content for the public, or, you know, there, as y'all know, there's thousands of use cases.

Mm-hmm.

I think a lot of it comes from this internal concern about, you know, employees using the technology in ways that, that you can't trust. So, you know, we'll leave that to local governments to figure that out. Um, we just, You know, I believe that when governments are looking at a generative AI policy, they should ask the question, what's the data going in? Is any PII or anything considered private information being exposed to a generative AI API? That's a no-no. And then are the answers sourced so that the residents can trust the content? And those are the 3 biggie biggies, if you will. And so again, we weren't going to push this product out until we had figured out how to check those boxes. And so just, you know, there's a government we're working with right now that's developing a generative AI policy. And so they don't want to use our generative AI until then. And that has its own— You know, life cycle of generating policy. But it's you know what I want to tell them is look, what are your goals? Make sure the vendors are meeting those goals, but don't slow up the the the innovation process by getting bogged down. And you know, we started a committee to look at AI, and they met for a year, and then they came up with something. And by that time. Something big's happened in the marketplace where a lot of the study that was done the previous year is now irrelevant, you know what I mean? So I think government should really be careful about privacy. They should be careful about, you know, you can't— you just can't— you can't use a product that you can't guarantee is trustworthy. So you just can't, um, not in the government use case. And so those safeguards just need to be in place.

Let's double-click on those safeguards because I think it'd be interesting to get your approach and maybe some best practices on approaching things like bias and AI hallucinations and all of these new things that are there. How are you thinking about that in the context of what you're building too?

Well, let's talk about hallucinations first. These language models default setting is To try to answer every question, whether it's right or wrong. And that default setting does not— it's not applicable to government. You can't have hallucinations, period. And so in our product, we intentionally default to I don't know answers, because the last thing we want to do, and we just won't because we're only using You know, content created by the government. So we won't return a wrong answer, you know, unless they've got some content buried in their website from like 2003 that they didn't realize was still on there. But anyways, so the hallucination thing is just an absolute no-no. You can't do that. And so we've, you know, we default to an I don't know. And then we ask the resident. to then communicate directly what it is that they need because we were unable to, to answer their question. And so we'll get the contents of that direct communication that gets routed into government, and we'll take a look at it with our customer and say, hey, is this a one-off, never to be asked again? Let's don't worry about it. Or is this a good question that we should get you to throw some content on your website about because it's going to be asked again? And I'll give you an example. In Mount Bellevue, Texas, a resident asked, "I just got a Tesla. I have a certified Tesla engineer that's going to install my exterior wall charger. Is this something I need to get a permit for?" Might not have thought about that use case before, but it's going to question is going to be asked again. And in Mount Bellevue, you do need a permit for that, and that's a revenue source to the city government. So it's like. Let's make it easy for residents in the future to get that question answered so they can pay for their permit and do it the proper way. And it's a revenue source to government. So then we help them with pushing stuff into their knowledge base, their website, based on the messages, the questions that we can't answer. And we'll always do that because every resident deserves the instant gratification of a response. And no resident should ever get to a dead end in chat. Never, because if you get to a dead end in chat, you probably won't use it again. That kind of defeats the whole purpose. So those are some of the ways that we handle that.

We have another audience question that came in, Bratton. How are you measuring and quantifying the delta on the citizen experience outcome? So effort, emotion, you know, it says recent studies point to a considerable level of dissatisfaction with chatbots. delivered by government. But, uh, what are you seeing out there in the data?

Yeah, I think, I think that's accurate. I think, you know, chatbots have given chatbots a bad name, uh, in large part because in, in many cases it's a DIY project for, uh, staff and government, which is just— why, why are we putting that on government staff? So, um, but the technology is there now. Like, I know even in a year, you know, our technology's improved so much. And so I think the chatbots of chatbots 3.0 or whatever it is, you know, using generative AI, certainly in our case, it's— they're working very well. So I forget what was the second part of that question.

Yeah, just what are you seeing in terms of your results or data?

Well, the results in our data is We're returning an I don't know response a lot less than we were. So those communications that come into the city that are questions that we haven't answered, we have dramatically reduced those, like above in some cases 20%. Now it's down at 7%. And there invariably will be that 7% 'cause people ask questions to their local government.

Mm-hmm.

that have nothing to do with their local government. And I know that because I grew up in a mayor's house and we— the phone would ring between 6 and 9 PM at night and people, constituents, would be calling the mayor and not everything was about city government services.

No, right there with you. So, you know, earlier you actually brought up my hometown, Roseville, California. You talked about San Jose just just a moment ago. Uh, you know, maybe you could share a couple stories with us, uh, right from the trenches, right from your travels across the country, in terms of how you're seeing generative AI, AI in general, kind of working well to, you know, do just what this— the last question was about— improve the customer experience.

Yeah, well, we've, we've just launched— we haven't even announced it, um, just because we're gonna put it on enough websites to make a big deal about it, but If you go to the Roseville website and, and mess with it, you'll see how it works. And so Rosie is how we branded it. And Roseville has one of the best communications teams in the country. They're amazing. And some of them are my really good friends. But I think just the, the, the way we measure it is by how many more residents are we serving? And how much time are we saving the government and its residents? And, and, you know, we see increase in chat volume over time. So it's not this one-hit wonder that's like, oh, this is so cool this year, and then it loses favor the next year. It's like we're building an audience and we're building, um, you know, volume of customer service communications for all our customers. And so, you know, they're able to serve more people and and save time doing it. So Roseville, Santa Clarita, we're about to push out Arlington, Texas. It's over there and a lot more about to flip over to our new system.

Exciting stuff. So, you know, for those agencies that are listening in too, I mean, we often get asked questions around where do I even start, right? Like, you know, maybe my employees are starting to look at it, but, you know, maybe their website's not in their current radar, but like how would you advise that they look at generative AI, you know, where should they start? Do they all need to go, you know, test every solution that's out there? Do they need to, you know, have policy in place? What's your, what's your thoughts on this, Braden?

Well, I would just say this to every government: push the ball forward in trying to make decisions at the department level around how do we improve the resident experience. 'Cause that's not historically how this has gone. And, um, so if you were thinking in that way, then what's your goal? What is your goal? What are you trying to do? You trying to serve more residents? You're trying to more diversely communicate with people? Like, what are you trying to do? Define that first before you start looking at all the bells and whistles and everything. Define your goal, and then go see if AI can be a component of helping you achieve that goal. And that's, that's really, it's not a technology question. It's an organizational culture and vision question first. And that's what I will beat like a dead horse to all these governments because it just, you know, Everyone needs to do better at creating better resident experiences with their government. Everyone. It's all— we're all a work in progress. We're a work in progress on our products, um, and, and governments are a work in process in terms of how they, you know, take care of their residents at a higher scale and create positive experiences for them.

Well, speaking of communication and doing that homework, where can our audience go to learn more about CityBot and all the work you're doing in the market?

Well, there's our website right there, citybot.io. You can book a meeting with me, um, and just know that if you want to have a non-salesy meeting and just talk about AI or talk about any issues that you're having, just— we're just happy to be helpful. Um, you know, we're in this for the long haul, and, um, you know, we don't know everything at all. But just happy to be helpful. And if you know, if your goals are around, you know, expanding your customer service, digitizing your customer service models, using channels that your residents are actively using every day, you know, then maybe you want to talk to us about how we can help you out there too. But again, we're all in this together. We're all custodians of of good technology. good outcomes and just having honest conversations with each other. That's the way I look at it. So kind of that's the approach that we take in working with these amazing government professionals that we get to work with.

Bratton, this has been a phenomenal episode. It's so great to have you back on with us and joining Dustin and I this week. We look forward to tracking what you continue to do. out there in the marketplace. And personally look forward to see what you're doing in Roseville. That's always cool when, you know, the people you work with come to town.

Yeah.

Until we see you next time on another episode, have a wonderful weekend.

Thank you so much, Joe. Dustin, thank you so much. Thanks for having me.