AI SEARCH FORUM | Unlocking the Power of Synthesis AI

Hi, my name is Susan Strom and I'm the Chief Client Officer here at Knowledge Architecture. And with me today is Chris Parsons, our founder and CEO. And you are joining us today for an AI Search forum where we're sharing stories of how our clients are using AI Search to help evolve and grow their firms with knowledge management.

I will hand it over to Chris.

To tell us a little bit about joining us today and what we'll be talking about.

Yep, great. Just to start with, if you haven't seen Synthesis AI Search, it combines next generation search technology with AI to instantly generate answers to natural language questions driven from content on your intranet. So it's kind of like combining Google search with ChatGPT, but for your firm's knowledge. What you'll be seeing today from our panelists is beta version 1.0, which we released into public beta in December of 2024 and is currently being tested by over 80 AEC firms and is live to all users at six of those firms. One of them is with us today. We're currently building beta 1.1 of AI search, which is focused on advanced employee search capabilities to find people by skills, licenses, certifications, roles, project experience, all that kind of stuff. So for the folks that are clients of Knowledge Architecture on the call, it's important for you to know you can deploy AI Search to as few as one user to start and slowly expand your usage at a pace which feels right for your firm. So then you have time to improve your content as well as understand the strengths and weaknesses of the tool. The folks on this call will tell you all about how they went through that journey and figured out when the right time was to add more people the content. They cleaned up all those kind of things. But we're going to start by looking at AI search in action at each of our three firms. And so we're going to start with Shepley Bulfinch. So Jess and Jim, can you please give us a short introduction to Shepley and tell us what you do there?

They moved the button on me. All right. Hi everyone, my name is Jim Martin. I am the CIO at Shepley Bullfinch. We are a 180 person firm at this point. I recycled this slide from our one of my K Connect talk last year so apologies. Not exactly accurate or but it was easy to get at. We have six studios including a virtual studio. We are doing hybrid and it is been pretty interesting. And Finch is our intranet and has been around for just over 10 years. Jess, how many people I think we have like 30 people right now in the beta search. Accurate.

Yeah, I think that sounds right.

Who are you by the way?

Well, I'm Jess Russell. I'm our design technology manager, which means I manage the technology we use for design, but really I do a lot of research and development, figuring out processes and software we're going to use. I do a lot of data analytics for operations. I do a little bit of finch. I do pretty much anything that the firm needs at the time.

That is a true statement. All right, well, I have been kicked off with giving people just an overview on actually what AI search looks like. So I'm going to start and give you guys a little bit of a, kind of a preview. So the first search that I have selected for you is time off because everyone needs to take time off. We actually recently, in the last, well, in the last two months, we switched over from a PTO based time off policy to a self managed time policy. And it's been pretty interesting because we have so much content in our Internet from the last 10 years that still thinks about PTO that we are, we are going around doing whack a mole, trying to identify all the places that it exists and trying to kind of correct the trajectory for it so that it actually gives the information that we want it to give people up front. So that's been pretty fun, but it gives a pretty good summary of things and then it kind of shows you, well, here are the pages or the posts or the videos or whatever resources that it used to pull those things. And then you still have access to looking at the individual types of search results as well. So for people who are not here for the things or they don't see what they need in the summary that's generated, they can go back and find it elsewhere. A couple of other interesting examples I was hoping. It's difficult to know what's going to be interesting to whom, but how do we create work plans? At Shepley Bulfinch we have created a bunch of resources around that stuff and it actually pulls from a bunch of different pages as well as a video that was done in Bird Feeder, which is our learning and development effort. And it just kind of goes through and does a search summary and you'll see that it actually gives you steps which is pulling from a lot of the content. And we've been bird dogging this pretty hard to try and make sure that the steps are in fact accurate. And they've been really good so far. So that's been very Encouraging. Another example is the hybrid policy. There's. This is very new in Shepley's 150 year history. This is the first time we've ever been a hybrid firm. And there's a lot of questions around what does that actually mean? And how are all the things that we're doing impacted by the fact that we are now doing them in a hybrid manner? So we're doing a lot of that stuff and again, it's doing the search summary stuff and we are looking at, well, where is it pulling that from and is that actually the right stuff? And so far it's been pretty good. Our AI use policy was interesting because that actually didn't exist. We wrote it and then forgot to post it, annoyingly. But once we put it up there, it actually does a pretty good job of summarizing what our AI policy is at Shelby Bullfinch. I'm sorry, the work plan one I already did. So this one is. This one's really interesting. This is just searching for trellis. Like this is simulating something that someone who's on the boards is trying to do a trellis and they're trying to figure out what the firm knows about that stuff. And it pulls a whole bunch of random content from the Internet, some a post that was made, which is an Ask Finch post, a bunch of other kind of random content, and then just kind of pulls it all together and provides links directly to the actual place where that stuff lives. Ask Finch is our ask and answer process. We've been doing that for probably eight or nine years at this point. And it has been incredibly helpful in getting people to share the knowledge that they actually have on the platform. And the AI search looks like it's going to be an amazing tool in order to daylight some of that stuff that has happened six or eight years ago. We're also concerned about, well, is that information still accurate? Because 10 years is a long time in the world of architectural technologies. So we're kind of taking a look at that. But so far it's looking like it's in doing pretty good shape. This one is interesting. So we did a search for mass timber and we share things that we know as a firm, but also things that have been shared with us or with the rest of the world. We bring things to people's attentions. So in some cases it's actually pulled projects that are not our projects because we have shared webinars that included examples of said projects. So we are trying to figure out now, like what do we need to do in order to tweak these things so that it's clear when it's pulling a resource that isn't a Shepley project, but actually some other project. So a book drop is another good example of that. It's just we've had random posts through the to the ages of people asking about what a good book drop is. It turns out that it's apparently very difficult to get an archetypically, architecturally, aesthetically pleasing book drop in the world. So that has been something that we've talked about a lot on Finch. Photoshop is another interesting one because we have moved to Affinity instead of Photoshop. So when you search for Photoshop, it actually brings up content that tells people, oh, you know, it's available under these conditions, here's what it's for. And by the way, we moved over to Affinity and here's some of the places that you can go to find information about that. And then finally InDesign, we tried to go to Affinity, the Affinity Publisher product, but it didn't really work because we have to collaborate with too many people who are using InDesign. So we had initially done some of that work to get everything moved over, but then had to move back. So this starts to articulate for everyone. Oh, well, here are all the decisions that are being made. So it's doing a really good job of doing the summary. And then the summary helps you look at the results and you can kind of decide what to do from there. Hopefully that's helpful for some.

Nice.

I think, Jess, you had a couple examples you wanted to show.

I don't know that I have a ton. One of the things that was interesting and I probably don't even need to pull it up, but when we were announcing AI Search the firm in one of our town halls, there was also we do a Shepley Spotlight at the end of a town hall, someone either like a new hire or leadership or someone within the firm does a little short presentation about me. And that week it was one of our principals who's been here for decades, and she had made some mention about a black flag, which is a decades old tradition of this flag people would hang over people's desks in Boston when they turned 30. And so I went ahead and popped that into the AI search and threw a screenshot in there because I knew there was a Finch post about it. Recently, someone did a digital black flag during the pandemic. But you can see on the side there, Jim's pulling it. There's a couple inside the Architect Studio, which are like interview we did like 15 years ago where they were mentioned. So we've run into a lot of things where you think there's nothing on the site because it's not explicitly written out. But having just decades of content in there is like uncovering little side stories.

And the really cool thing about it is that if you click on the, the actual thing, it'll bring you directly to the place in the town hall where that content actually exists. So because it's reading the transcript so you don't have to be directed to a resource and then all of a sudden you're stuck with an hour long or a 45 minute long recording that you've got to find the place for. It brings you right there, which is amazing. Cool.

Thank you. Shepley, Kathy and Andy, can you please give us a short introduction to Mead and Hunt and then tell us what you two do there?

Absolutely. Hello, everyone. I am Kathy Schumann. My role at the company is employee experience and Communications Director. So largely this is an internally focused role. I spend a lot of time on our Internet platform, the Station. We've also been at it about 10 or 11 years. Pretty much similar time frame, I think as Shepley Bulfinch and. Thank you, Jim and Joss. I always learn so much when I get to hear you guys speak. Mead and hunt is about 1500 employees. We do work nationwide. We have about 50 some offices nationwide and I'd say about 260 fully remote employees. So obviously we spend a lot of time trying to use our Internet to stay connected, growing company. We are in the middle of a huge ERP process improvement project. We'll show you a little bit about that later. And then some of our er, rankings. We do. We mainly work in 10 markets and then quite a few number of service lines. You can see sort of our markets there. And then we do everything from architecture to urban planning, construction, traffic engineering, everything in between. So a big multidisciplinary firm. And then I'll hand it to you, Andy.

Yeah. So Andy Knopf, I'm the CIO over here at Meet and Hunt. I'll be rounding 35 years in May, so been here a pretty long time. Yeah, Kathy kind of hit upon the points. We're very progressive on AI, spend a lot of time with that. We were really excited to get this integration and we'll show you a couple things on what we're doing on AI with our policy and our training. And I think that kind of goes hand in hand with this. If you Give people that background. It kind of dovetails into what's going on with this. We call ours the Station, but I think having that base really helps with everything else.

Cool.

And do you want to show us some examples?

Yeah, I definitely can do that. Let me share here again. So we're pretty excited with what is in the system right now, and we're even more excited with the possibility of having some integration with our help desk system, which is maybe coming here in the future. But we get a lot of questions about just really simple things, like.

We'Re not seeing your screen. Sorry to interrupt you.

Oh, sorry, I forgot to hit the share button. Rookie mistake. Okay, here we go. So here's our. Here's our intranet. We call it the Station. So a lot of really good help desk, you know, questions come in here. You know, what. You know, what do we have for a password? You know, so instead of having a help desk on what we have, you could type in that search and you can. You can find it really quick. You know, I have a new phone. This happens all the time. How do I get my phone set up to get Outlook? And then, you know, you have all these extra resources that you can click on. So if you want to go and pick the secondary resources, it'll come up and it'll give you all that information about how, you know, how you get your phone on there. So instead of having a help desk about that, type it in here and get it. So we're fielding a lot of our help desk requests through there, and we're trying to get our help desk down. But, you know, there's other things that you can. You can definitely do. You know, qa, qa, QC policy, you know, what does that look like? Where are all the documents about it? If you know, how we had it before, it was very static. You know, how the answers came back. And this one, you know, will bring the examples of all the different, you know, documents that we have out there. So the searchability, you know, what we're getting back now, we have all, you know, 1500 people on this system. People are loving, you know, having that extra, you know, like, chat, GPT or copilot feel to everything that they're doing. And we're definitely seeing an impact on our side, especially on the help desk, that they're. They're getting these. These answers, you know, here, rather than putting in a help desk request for it. So. So there's a lot up there that you can do. Here's one on our copilot, and I Think this really helped. We did our AI policy about a year ago. So we really defined what our AI policy was. So everybody got a foundation on that. And then we have a 30 minute training that we make everybody do before they get a copilot license. Cause we want people to have kind of a stake in it. So we do a 30 minute training. Once they do that, then they can submit and then they get that copilot license. But really when they get that copilot license and they do that training, they're learning prompting. And once they get prompting, they know how to ask good questions. So our users are actually, you know, doing really good, you know, prompts right now to get the information they want out of our system because they kind of had that co pilot background and they. And they've got that. So we're really focusing on prompting and getting good prompts out there for people to get their information. So that's a little bit on mine. Back over to Kathy if you want to add something.

Yeah, I've got a couple prompts I can share here. Got fired up here and the button. Okay, so you guys are seeing this. Okay.

Yep.

Yeah.

All right.

Perfect. Yeah. So when I get. Here's one here that. So we started with a smaller group. We have. We're the. We're the company that rolled out to everybody and we can get into that in a bit. But the smaller group, I queried them on what their favorite results have been. So we have a team, our architecture and building engineering team is have. They have a ton of robust information out here. In fact, way more than I knew. So the top results has revealed quite a bit of items and this was one of their favorite. How do I issue construction document changes? And I think what's great is in this summary we often see a lot of step by step. We've done some of that in the back end and he did some of that in the materials he created on these pages. But I'm finding a lot of the searches are pulling and putting together this sort of step by step on its own, which is super great. I mentioned we were going into a ERP implementation. This has been very widely communicated in a lot of different ways here at the company and ongoing. This is quite the process. It's going to take a while. So we've been doing our diligence on that end. But it is sort of hard to keep up with what's kind of the latest. Right. So this has been a great summary of. It is mostly based, it looks like off of one or Two posts. But you know, if you really. What I like about these filters is if you do want to take it down another step. So maybe last updated, for example, if I just did February 1, 2025, I can take it down just a little bit further. There's only been three posts since then. It's going to give me the most recent stuff and I think we haven't done as much education on this part yet. I would like to do a little bit more, get people understanding about those filters. You know, same with something with authorized contract signers. This was notoriously tricky to find out of the results, which is kind of all managed out of legal. This is a team that has worked more in documents than I would prefer, and I'm going to be pushing them a little bit more to pages and guides and that sort of thing. So it's still pulling some really good information from their documents. But it's going to be helpful to be able to share this with them in terms of improving overall content. But it really is pulling quite a bit of good stuff, I think. I just have one more. What does Mean Hunt's corporate giving policy? This will be something I get pretty frequently, maybe for a proposal for marketing. And they just want to add some information there. And we do have a nice guide out here on the station. But I ended up rating this earlier today and put a reminder in that feedback. So this is. I actually do have a better piece of information I could put out and add to this. So I'm going to make sure I get that out here. So if people are pulling proposal information and they want a nice summary or rewrite, they can take it from there. So I actually use the rating button to give myself a reminder. Excuse me. So that I go back and make sure I add that when I have a little bit more time. So I think that's all I have for searches right now.

Thank you, Kathy. Sure. Well, and then Jillian, maybe you can introduce us to RIOS and yourself.

Yeah, sure. Hi, I'm Jillian, Jillian Hutchison. I work at rios. Our sort of company tagline is we design everything from cities to teacups. And we're, you know, it's memorable. We. I don't have any snappy slides like the other guys, so I'm just going to do some hand waving. We. We're an international office, so we have three offices in the US as well as three outside of the US So a couple in Asia, one in London, and we're ranging around 275 employees at the moment. We do, you know, Some of our main professions that we carry is architecture, landscape, interiors, experience, design. We have fab labs, so we do in house fabrication as and myself, I'm the project operations director. So I dig into sort of global standards. I do sort of business integration strategies. I am the de facto knowledge manager and also oversee a bit of our learning and development strategies and all that. Let me share my, or I should say our intranet is called Blue. Hoping I'm sharing the right thing. And so you guys have already done some of these intros. So we've done some searches about. I'm going to share ones that have been rated. So these are some of our top rated results and how to send a revit model to a consultant. So Andreas, our BIM manager, shared this one with me saying this was a very accurate result. We're seeing the sort of top results as they should be. Same thing with our cto. He was testing sort of disk, how do I clean out disk space? And same thing, he rated this as a high rate. But what I'll start to dig into is some of the other kind of subsequent searches that we've been getting into. So one of them was around the global brand guidelines. And so, you know, it's good that it was just showing up as a document, but Bobby even took it a step further and asked for the hex code. So maybe this could be a bit more useful for someone in the moment when they're like, okay, I just need to know whatever this code is and use it right away. So that came up accurate, which is great. It would be cool if it actually showed the colors, but maybe that's like next level addition. And then another one. We started going meta and asking what is loot? So what if someone was new to the company and they're like, what is this thing that I'm on? What is this Internet? And it actually gave a pretty accurate summary. So that was cool. And then Bobby also started talking more and doing follow up questions. And so cac, which is our climate Action Committee, is our sustainability group at rios. And so he wanted to start digging in because his thought is that people are going to start using this to get very specific answers and how to build things basically. And so he started digging in not only what should a waterproof membrane be placed if in a warm climate? He started going in more specific. What about Los Angeles? Where would a waterproof layer go in Los Angeles? Because it didn't get specific enough what type of waterproofing membrane should be used. He started going deeper and deeper and he actually rated this one highly. It came up with an accurate answer in this result. So that was pretty cool. And in this, you can see it's pulling from documents and videos rather. We didn't actually have any wiki pages or guides on this. So he was really kind of testing the extent of how it would pull the answers. One other thing that we've noticed is sort of the scale of the questioning. So sometimes we were thinking about different users. So if someone was using one word versus multiple words versus like a longer sentence, and so I started testing what that would. What that result would be. So say someone's new to the company and they type in onboarding. So when that happened, we got like a variety of different onboarding material, but maybe not exactly what someone needed as they're starting. So then I said, I'm new to rios, and we actually have a homepage for onboarding that's called that. And this was more accurate as a result. And then I also went further with that. So I'm new to rios. Where do I start? And then we found the results started dropping off again. That top level page wasn't there. And so this was something that we were thinking about, or I've been thinking about when onboarding new people is kind of training them in how to query in question that if you start getting too long, it might start pulling bits and pieces from that query in different ways. And so the more kind of directed and specific you could be, the better results that we were finding. Yeah, that was the last one. Okay, those were those ones for me on that section.

Awesome. Thank you all for kind of giving us a kind of quick dive into what you're doing. I think we want to definitely make sure to talk about content management and rollout and onboarding and those kind of things. So, Gillian, I think we want to stay with you and just kind of talk about how you prepared content, whether you were adding content, updating content, removing old content. Just talk to us about what that process was like and at Rios.

Yeah, sure. Because when we moved to Synthesis 6, which was in May last year, that was like a great opportunity for kind of cleaning up content as we started with that. And so this was actually a really great progression to then as we're testing the AI search, see how that new content cleanup actually worked, and we could see where it was pulling from some of the old stuff that maybe we need to still move over. And I think one of the core things for us to understand was the search hierarchy and how that kind of filtered the information. Or the results. And so understanding that the best bets was kind of the most powerful element. Then wikis, then documents, then videos, and sort of understanding that sort of search order also helped us understand what to start cleaning up. I'm going to share again because I do have some visuals of the cleanups. So one of the things that we realized was around videos, the Monday morning meeting is our all hands that we do every Monday. And originally we just shared zoom links. And so we found that the results were skewed because they weren't obviously there was no transcript, it couldn't search anything within the video. So we started uploading these as native content. So that was one big push was to really advocate for people. And I've been a bit of a gatekeeper, making sure people are uploading things natively so we can use that content. Another thing was about, you know, new or mic like if there's presentations, whether they were video or, you know, PDFs or something, really migrating that content into wikis or guides. And one of those instances was our technical. Russell, one of our technical directors did a series on fire resistance. He ended up making this into a series of guides. Again, this just helps with the results because this is a bit clear step by step. And if wikis are going to be pulled ahead of PDFs, this is a better way to control and even look through that information and clean it up.

As you go.

Along. That vein of moving PDFs into guides, we also had our landscape manual. And so that also got pulled over into a guide. And what it ended up doing, it inspired some of those landscape technical directors to create more. So this was our original manual and they ended up developing all of this because once they got into loop and using it, they were just like, this is actually really easy and much easier than laying out a PDF. So it was cool to see that knockoff effect. One other thing we learned was just about nomenclature. And so because our our revit training was nested in a way, we realized that the search doesn't pull from the hierarchy. And so because he has training designer, he thought, okay, you know, it's going to pull from this. But he found the results didn't because on the page he didn't actually have training. And what he wanted was this as a result. So he ended up going and putting training into all of the these pages so it would pull an accurate result. And this just came out of some of our testing of content. And a similar thing with the Deltec PM training because we had just a name of a Library as Deltec PM training. It wasn't actually pulling these because we didn't actually say deltec Vantage Point or something in these videos. They're just, you know, short clips. So what we ended up having to do was rename all of the videos with Deltec in them so that the search recognized that as a term. And I think the last one is just like the consistency of sort of nomenclature. So, again, I was doing this onboarding testing, and originally I didn't have Project in the name of this page. And so it was telling new hires they needed to have a meeting with all the leaders. And at kickoff, I'm like, no, no, no, that's for project. Project managers. And so I ended up adding Project Onboarding here. I put Project Kickoff before here. And it was a great way to even kind of clean up the consistency of how we were using Onboarding or Kickoff throughout all of our pages as well.

Yeah, I think that's one of the things I've heard throughout the beta from folks is cleaning up the content to make it easier for a robot to understand. Also makes it easier for a human to understand. So just more clarity and kind of context is useful for all of us.

Yeah. It was a great kind of opportunity for each of our content managers to start realizing how they were even writing about their content on their pages. And it gave you an opportunity. Opportunity just to clean that up a bit, be more consistent.

Very cool. Jess, do you want to talk about the same thing, but at Shepley?

Yeah, sure. So we had a lot of similarities. I'm going to go ahead and start sharing. So we recently switched to synthesis 6 late last year, and through that process, it was basically me moving every page we have. So I am aware of everything that's on the Internet. I was being cheeky a couple minutes ago because I was like, how do I show how many pages we have? And I know the AI can't count that, but I did a Finch trivia in September. So as of September, we had at least 650 pages. It's probably closer to 700 now. So we kind of decided, like, prepping for the AI search is not going to be feasible, and we're just going to actually turn it on beta, roll it out, and then use the AI search to highlight problems kind of like Jillian said, where, like, pages need to have something set explicitly.

Interesting.

With town halls, that doesn't always pick up a town hall unless someone said town hall in the town hall. Like, it doesn't always pick it up in the title of the video. But synonyms have been really good for us. Like, we had problems with people trying to find questions about expenses and expense reports. So I now have one that's like reimbursables, like expenses, like everything you can think of. This one might call expenses. The synonym actually really helped a lot with that. But for what we did, really before we kicked it off, was that we had to move all of our video to synthesis native. So one of the things about being somewhere and responsible for something for a really long time is some of your best ideas years later are your worst ideas. So we had our entire learning platform basically built into a couple channels on teams, because then teams automatically dumps the recording in a recordings folder and then Stream lets you play it, and you can actually go back on the back end as an admin and see views. So this is really helpful for understanding who's going back and watching the video. But that also meant that every page we had was an embedded video hosted somewhere else. So we ended up moving just for our learning Library alone was 250. But there was another like 70 to 80 videos that are in posts, and we're still finding them too. But the way I actually pulled that was I took the data out of the API, the OData connector, and then I searched in the content block, Anywhere that said SharePoint or Stream App or Vimeo, and then that made a list. And then I had a couple people just like trudge through and upload everything. And then again, it's so much content that we didn't really check the transcripts, but we did try to find bad transcripts to search for and try to get it, like, oh, if this is bad and I search for the right thing, can't find it. We actually just couldn't even find bad transcripts, so we let that one go.

Cool.

Amazing. Thank you. Kathy, is there anything you wanted to add in terms of preparing content for AI search?

Yeah, I mean, I guess I'll just say we've done a lot of similar things. I have a laundry list of things that need to be updated and edited and, you know, nomenclature changed exactly the same. We had the side benefit of moving into this ERP implementation. So last summer we had four data management interns in house that were helping prep our vision data. But the side job that they got was to clean up our posts on synthesis. So we were able to have them go through. We had some strategies in place there. It was a lot of heavy lifting for them, but kind of fun to be able to get a lot of that cleanup done early. Of course, we're still finding a decent amount of that. But we were. We archived some things, we deleted some things. It's very evident what's still out there now and what's great in the whole feedback process is that we're able to get that cleaned up pretty quickly. So now I can just make a very simple, easy delete. I just think a lot of it was gone. So that was a huge benefit was to have some interns sort of focusing on that piece. Other than that, the other thing I've noticed is we've gotten in the results. There's a decent amount of content I didn't know was out there or had forgotten was out there. You know, my forgotten resources I'm revisiting now. So we have an office page. It's very robust, or office pages, I should say. So if you go to visit an office, they had like a list of food recommendations. And I just typed in a search about which ones. Where would you recommend I eat when I'm at the Dallas office, or something like that in the search. And I'd forgotten we had a small business minority document out there for businesses that people could go to near our offices. It was in an Excel file. So again, it's just sort of identifying your resources you haven't seen in a while. So my list of things to clean up are probably longer than what I've already cleaned up, but it's fun to investigate that.

So it sounds like a lot of it was feedback driven, too. You have a long list that's helping you prioritize where on the list to start.

I absolutely love the feedback feature because I feel like there's a lot more engagement for people and that's a big part of our strategy and why we just decided to go fully live to everybody. But we need them. And it's kind of part of our. Our onboarding and orientation when we do teach people about this. So it's gotten them very engaged.

Yeah, I mean, actually, that's a great segue. Why don't. Why don't we just stay with me and Hunt and maybe Andy join and talk about how you kind of rolled out through different phases, through different cohorts within Meat and Hunt, and then how you guys made the decision to go live for everyone.

Yeah, so we started out pretty small. I think it was just Kathy, myself, and maybe one other person when we really first started because we didn't really know what it was going to be all about and how good it was. And I mean, right out of the gate. I was very impressed with the look and the feel and what was brought to the table. Like Kathy was talking about with anything on the AI side, it's garbage in, it's garbage out type deal. So you really have to do a good job job of parsing out that stuff that's not, you know, that's not relevant and get rid of it because AI will mine that out more so than the regular search. So it's going to find a lot more. You know, if you got garbage out there, it's going to dig it out a whole lot. And that's what we learned on the copilot side too, is we really have to be strategic about what we're telling it to find and make sure that, you know, that we clean it up. Because the downside of this is if you just have a lot of old information, it's still going to mine that up. So that's probably the only downside of this is, you know, making sure that you have current information and, you know, spend the time to clean that up. But I mean, digital hygiene should always be at the forefront. I mean, it's just huge. You know, if you just never clean that up, it's, you know, it's never going to possess good results. So we, you know, we ramped this up pretty quick. And I know I was pushing Kathy, you know, I'm like, I love this. I'm going to turn out right away. And she's like, oh, let's just pump the brakes a little bit. Let's go a little bit slower, which is, which is good. You know, we, you know, we had, you know, some champions that we gave it to. There was about 25 people. And that's normally how we do a lot of these things, is we'll start with a very small group. We'll have a champion group of about 25 people. And we actually did that when we rolled out Copilot. We had about 25 people. And then there's a couple other stakeholders that we'll throw in there. Maybe somebody had a group letter or C level person we'll throw in there just so they know what it looks like. And then it was like, we got a lot of good feedback. It was probably about six weeks of testing it thoroughly. And then it was just like, let's just rock and roll. And whenever you roll out something this big and new feature to the whole company, I mean, usually there's inundations on the help desk. I mean, for any. It doesn't matter. You're Always going to get a lot of people asking questions. And I mean, it was, it was very seamless, I'd be honest. I mean, I don't think we got, you know, more than three or four help desks, you know, like, what is this all about? You know, how should I do? I mean, it was very, very seamless. So for us, you know, when you're, when you're pulling the trigger for that many people, you know, you tend to be nervous. But this, this worked out very well for us.

Right on. Appreciate you sharing all that context. Jim from Shepley, anything you want to add to kind of how it's different about maybe how Shepley approached it than what we heard at Mead and Hunt?

Yeah, I think for us, we didn't, we, we have like more of a traditional. We're starting a little bit slow, trying to learn some things, trying to help people understand what things can do and can't do and all the rest of that fun stuff. We didn't really have the, I think cojones is the appropriate word to just kind of turn on live for everyone. But we started with an alpha group, as we typically do, and then we expanded it to a beta group and then we entered our kind of open beta where Jess announced at Town Hall a couple weeks ago, anyone who wants to jump in can jump in. And we were waiting for the avalanche and the avalanche didn't come. I think ultimately our problem was that we, we put guidelines around what we were hoping for for people who were in the beta. And we said scary things like, yeah, we'd like you to give five pieces of feedback a month. And people were like, I don't know if I want to commit to that. So we have had much better luck going to people directly and just saying, hey, here's the kinds of things we think would be interesting and we would be interested in your particular feedback around whether that's, you're an expert so you know whether or not the answers that are being delivered are going to be accurate or not, or you're a noob. We really need to know what you're going to be asking so we can evaluate from a noobs perspective what is, what are the answers you're getting and are the answers adequate or not, or leading you to where the answers might want to live. We did some other kind of interesting things like there's, there's an old critical knowledge transfer project that we did in 2017, 16, something like that, and we actually had three one hour recordings of one of our principals Jennifer Oliver, who is, you know, an expert at healthcare planning. And she was talking about, you know, how she actually does her thing, how she talks with clients, how she approaches the problem of getting them to articulate what they need, not what they want or not what they think they want. And so we actually created a private community and just used a filter function and just said, all right, I'm going to ask you some questions, but I only want you to use this private community. And that was pretty interesting. I would say the nature of the interview was such that you're not getting super kind of rich. This is what Jennifer thinks sort of thing. This is, this is an AI that is tuned for search. And so it's telling you, okay, well she said these things and here's where she said them. And then you can go and you can hear the context around them, which is great. But we also tried creating a. Some public. Sorry, some private libraries and putting a bunch of RFPs in them. Think there was a question in chat around that and that was a little bit less successful. It was amazing in that you could ask it, hey, write me two paragraphs on this topic that is referenced in several of these RFPs. And it did a really good job, but it couldn't do things like give me 30 blog post ideas that I could write about the topics in this thing. And I think that is primarily because it is tuned for giving you the right answer, not tuned for making stuff up, which in my mind is actually not a bad thing. Because if I'm, if I'm asking my intranet for answers, I don't really want a lot of hallucinations if I can help it. I want it to. I want to be able to control the content that's on there. We have other tools available to us that seem like they're better at the give me 50 ideas for or whatever. And that's actually what we're using Copilot for, among other things. And it seems to be just much better at the super creative, like taking a lot of liberties with, with, with maybe what you might have actually documented.

Yeah, I think you're right. I mean, we've seen like, like with Shepley, we've seen clients putting hundreds or thousands of past proposals or project sheets in and getting answers out of them. But we have deliberately not said invent things. Yeah, right, because it's supposed to be search. So we'll see what the future holds. Maybe we'll do something like that too.

But yeah, it is interesting though, I would just throw out there based on our experimentation, you don't need to put hundreds of thousands or hundreds of RFPs in there. We actually got some pretty good content out of just five. And it seems like you need to make sure that the in the entirety of your conversation target is included in there somehow. But you don't need to have a lot of redundancy necessarily, in order to get something valuable out. So I don't know that the quality won't go up if you add a substantial number of more documents. But if you're interested in testing and kind of getting started, I would just, you know, shoot for small content and figure out what you can get out of it. You'll learn a lot from it and it'll be. Make it. Make the rest of the process easier.

Cool.

And just before I move on to Jillian, what are your thoughts about going live? Have you set a date? Are you waiting to see some more development? Like, what's your thoughts on that?

Well, my data is sooner now that I know that Meat and Hunt has done it and nothing bad happens. That's always encouraging. But I think ultimately we are in open beta right now, so we will. I don't know when we will push. Push it out to everyone by default. I think we probably could do that right now if we chose to, but I don't know the value, and I'm still. I think there's still some. Some value for us as the people who are rolling this thing out to be a little bit more controlled and get a little bit more of a commitment from people who are joining it so that we know, okay, well, they're going to be trying it. There's a little bit of backend analytics that I'm still hoping to get from the platform around search, so we kind of understand what people are searching for, regardless of whether they put feedback in for it. And I think once we get that, I'll be a little more comfortable because I'll have a better handle on what's actually happening and we can go ahead and do proactive remediation for any weird things that we're seeing or ask genuine questions about people's experience with it.

Awesome note for the audience. Search analytics are going to be going into private beta in March. Gillian, do you want to talk about RIOS and kind of your rollout phases and your kind of thinking on how you kind of roll it out to all users at some point?

Yeah, totally. So it really started with me, then I invited two of my fellow nerds who I know like to dig into things and test things and I think once we started to understand how it worked and like I was talking about earlier, that sort of search hierarchy and we kind of wrapped our heads around it, then I invited in another smaller group of maybe eight people and we. And again, it was people. I selected people from all different disciplines that I knew would kind of dig around and test because I really wanted to get those ratings. And I, I know people all think and approach things in a different way. So for me, kind of gathering a spectrum, some content creators, some just users and so. And that went well and we got some ratings out of that, which I showed earlier. And then I actually opened it up just recently to our community managers. So these people are the ones kind of managing their own areas. Some of them were included in that earlier, but I haven't seen a lot of movement from those folks. But I'm meeting with them today actually to get their thoughts on it. And then I'll probably like give it another couple weeks and then we'll open it up to the office. Like if there isn't any sort of big, I don't know, question marks that people have, I think we've got enough stuff sort of cleaned up that you might as well just open the gates, see what happens.

Cool. Cool.

Yeah.

I want to talk about onboarding and how you guys are onboarding new users, but I want to do a quick lightning round from the Q and A. Are you guys adding more synonyms? Have you found yourself doing that to support AI search?

Yes.

Okay.

And best bets too.

A lot more best bets.

You mentioned expense stuff, Jess, Is there more than that you're doing?

Yeah, I was gonna pull up that page, but it's not actually as impressive as it would look. You would think. But there is around the like smt. So the switch to self managed time where kind of sifting through the fact that there is 11 years of content around PTO and posts about it. So like the synonyms we know works. And so I am collecting now just every time it pops up with feedback like putting that in and then not just what someone was searching, but like everything I can think of that someone might search into that synonym as well. So treating like the best bets is great. I want you to find this page. But treating the synonyms of like, I want you to direct towards this word.

I think one other thing that we're looking at with great interest is what does SEO look like in AI search land? Because Jess also has a really interesting example of an idea that she grabbed from EUA's presentation last month where we basically had a bunch of posts about Microsoft Planner and using it for tasks and things like that. And we had some Microsoft videos embedded in it. And the search results weren't fantastic, but she actually went through and asked search to summarize specific posts like was shared by eua and it gave us a really good summary which she then turned into an actual. Yeah, an actual page, which then amplified the search results, which is really interesting. So it's got us really thinking about like how much of the content that we need to deliver is really to assist the AI, but also would be beneficial for our people as well. So we're starting to think about that a lot more. And that kind of is in addition to the synonyms and the best bets, but it's also like developing content specifically to hold to fill gaps that the AI needs to have filled that a human being would not in order to make the AI much more useful. So we're trying to get our head around that.

Part of that too is treating have it generate a description for like an hour long session. You did drop that description on the page that you've now read through and verified that that's the right information. So now you're using the AI search like summary to then be a source of truth the next time someone searches that. So because you can't see the result everybody's getting, you can at least kind of like feed a result you would like the next person to get.

That makes a lot of sense. Kathy or Andy. Any, any, anything you want to chime in on this kind of synonym. Best bet, giving additional context to search discussion.

I'm going to just take that idea that she just showed and start doing more of that, you know. No, right now I'm just making small changes as we go, but I think these more intentional changes are what's going to be really important. But because we have so many users in it right now, it, you know, there's a lot of things coming out. So again it's just prioritizing what we can tackle first.

Great.

I'll do one more question, but I'll just answer it before we go back to talk about onboarding. Jerry asked, is searching within hierarchy expected to be a future improvement of AI search? And we think so, is my answer. There are some unintended consequences from inferring too much up of a contextual hierarchy tree that could lead to unintended consequences. So it's something we are studying and kind of looking at the most responsible way to do it. Okay. On onboarding, let's Have Jess talk about onboarding folks to AI search at Shepley.

Jess, you're muted.

Sure, thanks. So, like Jim said earlier that we did an open beta recently. And so this is like, onboarding has kind of developed over time, but mostly I've just been dumping all the information people need to know about the search in an email so they can go back and look for it later. But I love to put things about Finch on Finch. So now in the getting started with Finch, there's actually the AI search beta guide. And part of it being an open beta is like, anybody can stumble on this page and then know that they can reach out to get started with it. But like Jim said, we kind of said, like, hey, try to spend 30 minutes a week for a few weeks and we'd like to see five feedback a week. And then all of a sudden people are like, I can't commit time to that right now. But so basically this goes through, like, how it works, what it's going to do well, what it doesn't do well. And then we're updating this list as you guys update your list. We're always, every time we talk about it, talking about our AI policy of like, verify anything. So we've got 11 years of stuff on here and you can't even really use the filter of like, oh, show me stuff in the last two years, because that might still be out of date. And maybe you're skipping content from like five years ago that is still accurate. So it's really just like a trust but verify situation. And really our people are just kind of used to that when they're searching Finch already anyway. But we do ask people to send five star reviews as well as feedback. That's bad because you can look in the back end and see what people are sending. And I love getting email notifications of what people are looking at and what they're getting. So one example Jim was talking about, like, we found better success actually reaching out to individual people and then telling them why exactly we want them to do it. So we had a hard time getting emerging professionals to actually want to sign up for this until I started emailing them directly. And I'm like, I want to know what it's like for you to, like, have questions and get answers. And we had someone who like, absolutely dove in. So, like, in her first week sent like eight things. And, like, they're all legitimate things that she needed to do that day. So things like, how do I get something off my desktop and share it via OneDrive, right how do I do an expense report? How do I write a field report? Like, all of these things are legitimate questions she's having during the day. She's not just experimenting at this point. And I find that really exciting. When you, like, give people more direction than like, we're just testing this AI, then they know what questions to ask. And then I kind of wanted to just poke in here because you know how much I love data. Of the 25 people we have probably so 16 of them have been sending feedback, which is pretty good. I think some people, it's like they went in a little bit, they kind of fell off. Some people are in there because they're like department managers and I'd like them to test, but they haven't gotten to it yet. But we are actually finding, like more recently, people are sending about three to five feedback a week. So that request of five actually wasn't too much of a stretch when we start looking at the averages.

Amazing. Kathy, can you talk about your step approach in one minute? Is that possible?

Yes. So quick share. I think I have it. I don't have it up yet. Yes. So this. Sorry, Taking the next step. Okay, so here we are. This is sort of. This is the approach we are taking to roll out an orientation. This is just a design that we had somebody do in graphics team because one of my testers had said we really need to make sure we're managing expectations, what it does, what it doesn't do. And so I said, let's think of it like a step. And then I just kind of made this little acronym, I guess, of that, where I just basically wrote out our intentions and our expectations for them. Right this, our intentions, that every piece of feedback is important, and patience. And here's where you can check those amazing release notes from Knowledge Architecture and just the engagement and continuous improvement piece. And so this is where this is sort of the model that we're using, in fact. And I just wrote a note, we have a March 27th all company meeting. We will address this in a slide at that point too. But it's a kind of a constant reminder of the expectation. So I think this has been part of the support of getting everyone up to speak again, knowing that we gave everybody access without maybe having it as clean as we could have.

Amazing. You did it. You hit the brief.

Excellent.

Obviously we could just keep going and going on this discussion because there's a lot happening here. But I'm so grateful for all five of our panelists, Jim and Jillian and Andy and Jess and Kathy, for sharing your time with us. We're going to send the recording of this out on our website and sent out via email next week and we'll put it on our online community. So thank you for all the questions. I see that Susan answered 23 offline and I know you guys were chiming in in the chat. So thanks everyone for this conversation. We hope you have a great rest of your week and we'll see you next month.