AI SEARCH DEEP DIVE | Preparing EUA for Synthesis AI Search

Well, hello, my name is Susan Strom and I'm the Chief Client Officer at Knowledge Architecture and with me today is Chris Parsons, our founder and CEO and Andrea Mead, the Knowledge Specialist at eua. You are joining us for AI Search Deep Dive which is where we share stories of how clients are using synthesis AI search to help evolve and grow their AC firms through knowledge management. Chris, I'll hand it over to you.

Yeah. I am super excited to have Andrea Mead from EUA here with us today. Andrea is a knowledge specialist which means that she builds and maintains a comprehensive cross disciplinary knowledge base at EUA. She also supports EUA's continuing education program promoting growth and development and she is dedicated to implementing sustainable knowledge management strategies and using technology to strengthen the firm's knowledge sharing culture. I have had the great pleasure of working with Andrea and her team on their AI Search implementation and EUA was one of our first clients in the AI Search beta and they have worked both tirelessly and joyfully with our team to help us improve both the product, but also to help prepare their content to take full advantage of of what AI Search has to offer. So Andrea, please, thank you for being so generous with your time and your knowledge and sharing with our community and for sharing your AI Search journey.

Thank you so much. What a wonderful introduction. I'm going to go ahead and share my screen so if you can confirm you are seeing the right one. Yes, we are. Looks great. Some. So thank you all for joining me today and Susan and Chris for inviting me to be here. It's such a delight to be a part of this community. As they said, I'm a knowledge specialist at eua. I just celebrated my six year work anniversary with the firm earlier this week which is really exciting and hard to believe. I spent my first three and a half years at EUA as an ambassador which is a little bit like an administrative HR hybrid role. And the last two and a half years I have been in this knowledge specialist position and I couldn't be happier with the shift. As Chris mentioned and as any knowledge manager or specialist can probably relate to, I wear many hats at the firm. If I had to serve it up in a few words, I manage our intranet called Connect, which I will reference a lot today. That includes auditing content, leading community champions and training them to maintain their respective sections. And ultimately I find ways to more effectively and creatively collect, organize and share our knowledge. Like Chris mentioned, I also support our robust continuing education program called euauth. So EUA is a fully integrated design firm. We have 264 employees in five different office locations. We have three in Wisconsin, Milwaukee, which is my home base, and the largest of our offices. We have one in Madison and Green Bay, and then we have an office in Denver and one in Atlanta. Our projects cover seven major markets spanning all across the country. So a little bit of background. Our partnership with KA started back in 2012. I was not around UA for that, but I have learned also that we went live with Connect in 2013. So we've had Connect for quite some time and thus there is a lot of information and knowledge available on our intranet platform. This length of time, having Connect, it certainly plays a role in how we're preparing our content for AI Search, and it also speaks to the importance of maintenance of that content, which I'll talk a little bit more about later. We joined the private beta in October, the private beta of AI Search in October of 2024. And like Chris mentioned, we've been fully immersed in testing and providing feedback ever since so that we can strengthen our Internet content before releasing it firmwide. One thing that's really clear to me is that this technology is changing quickly and we are learning and growing with it in real time, which is pretty awes. So with that, I'm going to jump into some live demo of AI Search. I just want to show just off the bat how powerful this tool is and showcase it working well. And then I'll talk a little bit about how we got here, what we learned along the way, and then what's next for us. So going to switch my screen here. All right, so here is our Connect homepage. I'm not going to talk really about the organization of it. I'm sure many of you know that we kind of have it bucketed by practice areas or technology branches, bdm marketing. I'm just going to jump right into the AI Search and start showing it off a little bit. So I might ask it something like what do I need to know about the bxp? Most of you are probably familiar with that, the BIM execution plan. Right now it's going to generate a summary. Here you can see it's giving me a great summary. It's breaking it down into different sections. And then here, when you hover over these citations, it's telling you exactly where it's pulling this information from. So if I go to this page, I get the rest of the information right. But what that search summary is doing is it's combing through all of this stuff and giving me kind of the Highlights what's on this page. This page includes a downloadable document, some naming conventions, and if I want to understand what's in the model expectations of the bxp, I kind of get a little summary there. So this is just a great way of getting people to the information that they want quicker. Another one here is eua license to practice in Utah. This is a new resource that we added to Connect recently. We were getting a lot of questions. Well, that's what you get sometimes with live demos, sometimes it doesn't work. This is a new resource that we added to Connect. We kept getting questions about where our firm is or is not licensed. So what we did was we created this guide of that information. So you can see here, yes, it's licensed to practice in Michigan and it's citing this page. So I'm going to go to this page just to kind of show you how we packaged up this information. So here you can see we're licensed in both architecture and engineering, who the architects of record are, which is helpful to know. And then if somebody's here, they might just want to page through the different states and see what's what. So this is kind of a great resource for people to have at their fingertips instead of, you know, having to reach out to somebody all the time. Okay, this next one I'm going to show this was one that I had already submitted before. Instead of recreating it, I'm just going to show you a really, really cool, innovative way of interacting with AI search and your intranet. So I searched what's our approach to historic preservation? You can see that it is combing through video here, several different documents. And then it gives you this kind of highlighted set of information here of how we approach historic preservation. And then I asked a follow up question. Can you rewrite this as a narrative? And bamboo, it provides this incredible narrative that you can now use for marketing purposes or, you know, however you might want to package this up and send it out. This is just a really sophisticated way of interacting with our knowledge base. I love that example. All right, I'm going to search a couple more here and then I'll get into our process. I'm going to search affinity. So some time ago we switched from Adobe to Affinity. And so when I search Affinity, you kind of get this nice summary. You see we got the three different branches of it and then citations to match what we did on Connect 2. I'm going to veer a little bit and go to these pages. What we did for our applications is we're developing this resource library of all of our software applications. Right? Each of the pages kind of have the same framework that when you expand this section, you've got documents, you've got videos, you've got templates, and you've got external resources and FAQ posts related to that application, related apps, hashtags and support resources all built into this page. You'll see here the breadcrumbs, the lead back to an application library. So we utilized an image library to create this so that we could have the logos here match each of the applications. We only have 52 here. This is a work in progress. But what we've done is we've added this custom field here, which is a URL. So somebody can come to this application library and they might not know. They don't know what they don't know yet. So they might say, okay, I want to see what applications we have available for graphics. They filter that list and then they're able to just focus on the task at hand. And then they can go here, hover over this, and then they're taken to that page with all of the information on that application. That was just an aside, but it's a way that we packaged information about these applications that is really working well with AI search. All right, and I've got one more example here for the quick hits. Autodesk email change. So I just got married. I need to change my email in Autodesk. I don't know how to do that. So I come here, I search. It gives me step by step instructions on how to complete this, which is incredible. Right now you can see that it's citing that document there, which is a guide on how to do this. So I just want to go and compare this to regular search for a second. So if I were to just use the regular search and not the AI search, you still get this, right? This is the document I'm looking for. But somebody has to know that and they have to click on it and then read through the document, right? This is like if you're traveling from New York to LA and your means of getting there is you get on a bus and. And then you get on a train, and then you rent a car and then you get on another bus and then you're finally in la. Well, I think of the search summary as your direct flight to the information. This just gets people directly to the information they're seeking without all of that other nonsense in between. I just think it's so sophisticated and ultimately it just gets people to where they want to go quicker. So that is kind of my initial demonstration. Now I'm going to flip back to my PowerPoint here. So if you've talked to me at all in the past three months since we've had access to AI Search, you probably heard me say this Connect is only as good as the content we give it to work with. You can have really powerful technology, but it has to be coupled with strong knowledge management. I think of our intranet as a garden, right? It requires weeding out outdated information, planting new relevant content, nurturing the ecosystem with regular updates and maintenance, and then ultimately you can harvest the benefits. If you have a well maintained knowledge garden, so to speak, our users can get access to accurate information and the retrieval is quicker, which ultimately leads to increased productivity and innovation. We want to create a thriving ecosystem where knowledge can flourish and AI search is a great tool to make that happen. And ultimately what this is really about is how we used AI search as a catalyst for auditing our content. So after we got past that first flurry of testing, we refined our approach a little bit. Many of the testing, you know, the search terms that we were doing would drum up old posts, duplicate documents, as to be expected if, you know, we've had Connect for over a decade, so we certainly had a process in place for auditing our content. But using AI search allowed us to accelerate that process and to really refine it along the way as well. In one of my conversations with Chris in preparing for this webinar, he said, it is the nature of AI search to find the most relevant content and sometimes the most relevant thing you have is outdated. And that turned out to be really true for us. So we, we used AI Search as this catalyst for auditing our content moving forward. So getting started, right. Once we got access to AI search, it was important to set a foundation of how we were going to use this technology in a way that complements strong knowledge management. So what we did, I've said this already, but we just jumped in with testing right away. I used the KA search term list as a starting point. I also just used my general knowledge of, you know, questions. People come to me for that. I typically point them to our intranet to find whatever makes sense to you, just start as. I mean, that's all you can really do. Whatever makes sense to you, just start testing. Another thing that I learned really, really quickly is do not underestimate feedback. Feedback is so key. If we don't know what to fix, if there's not participation in feedback, how would we know what to fix. And similarly with knowledge Architecture, if they don't, what's working well or what could be improved, they're not going to know where to place their priorities. So it's really, really, it has to be a symbiotic relationship between administrators, between end users and the technology. And I'm going to come back a little bit more to the value of feedback later, but I can't, I can't stress that enough. Another thing in preparing for AI Search, preparing our content for it was using tools like stream administration. So we were able to leverage that to clean updated information really quickly. And here's an example that I bet most of you can relate to, which is annual health benefits enrollment, right? So this is something that happens every year and we have all kinds of resources to support this topic. Posts, documents, pages, what have you. And leading up to this event, we're typically sharing temporally relevant information by way of posts, right? Open enrollment is happening next month, open enrollment is next week, Open enrollment is tomorrow. And as you can imagine, as soon as the phase of enrollment is complete, those posts become obsolete. So what did we do? We realized really quickly that we needed to clean this stuff up because it was just really cluttering the valuable information that people were finding. So with tools like stream administration, you can bulk remove posts related to a topic, especially if you're using, you know, consistently using tags. So we used benefits enrollment 2024, right? So now I can go back and delete all those posts tagged with that in one fell swoop. So we really used tools like that to help prepare our content to really shine and not get bogged down with dated information that's really temporally relevant. And then finally we created a pilot group. This was really important. I recognized at the beginning of this process that I can only manufacture so many prompts before it's not relatable. I needed real people asking real questions on the platform. I set expectations for the group and let me back up. So first it was kind of just me and my, my boss that had access to AI search. And then I opened it up to the rest of the technology team because that's the team that I'm a part of. These are my teammates, and they're familiar with the admin side of Connect between design technology and information technology. Those two groups have the densest knowledge bases on the Internet and, and they're the ones most consistently auditing and editing, editing their content already. So I knew I could lean on them to participate and provide valuable feedback. And I was certainly Right about that. But that was kind of, you know, that was just the technology team. I wanted more. I needed more real people asking real things. So I opened it up to our Air&D group, which we already have established here at ua. I invited them into this pilot group to provide even more valuable feedback. This is a group that includes representatives from various departments, from office locations, studios, roles, tenure. It was important to us to have a mixed group so that we could understand what people are searching for, how they're searching for it, and ultimately if they're finding what they need. I will say we got pretty good participation, but it could be better. I might even add another phase before rolling this out to the firm at large in sort of like an opt in situation where people can decide to join the group because that inherently sort of suggests that they'll be excited to participate. This is where it gets a little bit more interesting. During this process, during all the testing, we realized, okay, maybe we have some of the information on Connect. It's there, but it isn't packaged in a way that's working in concert with AI search. So I'm going to do a little bit of show and tell here. Here are some key takeaways what I learned leveraging AI search to get our content ready. So when possible, I suggest using posts to promote information, not introduce it. And what I mean by that is, for us, Connect has been around for over a decade. It has served a successful means of communication during all of that time. But that also means there's a lot of information for AI search to sift through, right? Posts are a great way to promote information that already lives on Connect to draw attention to it rather than information being presented for the first and only time in a post. Post should be used as a means to an end. And here's an example of this. So prior to AI search, the source of truth for information on changing your network password was a post from 2019 by an employee who is no longer with the firm. So even if that information is still correct, is that really what we want to promote as the source of truth? With how rapidly information changes, does that inspire confidence in the user that this is the most up to date material? The answer to that is no. So what did we do? We took the information from that post along with the information from two to three probably documents and we repackaged it. We set it up on a page on Connect rather than a post. And then this way we have it. I'm going to do this. How do I change my network password right so this way people can see the last updated on pages, which I'll show you in just a moment, and they can trust the validity of it. And then we don't have multiple versions of documents floating around both on the network and on the Internet. So now when I go to connect and I search, how do I change my password? I get this really great summary. Okay, first you have to consider are you in the office or are you remote? And then the steps, the subsequent steps from there and you can see that it's pulling from this network password page. And when you go here, you've got a few different options. So this all used to be a post and maybe two to three documents and instead we kind of put it all together in one page. Easily digestible and works really well with AI search. Okay, I'm going to abandon my PowerPoint for a little bit and just tell you the next points and then I'll go back to it so I'm not bouncing around too much. The next thing I learned is that Q and A format works really, really well with AI search. Now I think this one is kind of self explanatory, right? Like by mimicking the style in which people are going to search, it is more effectively finds the information. So what do I need to know before using the Matterport? Now, the Matterport is a piece of equipment that people can use. They can check out and go and scan spaces. So what this is doing is it's giving me this great search summary, right? It's citing exactly where it is. It tells me the accessories I'm going to need, what to do before I use it, any additional training, what the instructions for that are. And then if I go to the page that it's citing, you can see it's laid out really nicely. We've got a cheat sheet. And then it's really pulling from this FAQ section. And of course our team is fielding questions all the time. And if they're getting more questions, they're adding it to this page so that the next time somebody might have that same question, they can just come here first. I'll do another example of this. How often should I audit my model? So this one is going to give me pretty straight to the point. Best practice, audit your model on a weekly basis. Say I want to know more. Okay, I see. It's this whole audit page. I might then seek out more information and see what other questions I might have before I even have them. Right. So Q and A, it works really, really well. I'd say as much as you can leverage that. Highly recommend. Okay, the next point I have is kind of one of my favorite and it's to use video chapters. I learned this is also going to be a plug for the feedback tool in search. So one of my colleagues created this great page about acc Autodesk Construction Cloud with all kinds of sub pages to explore. And as he was testing AI search, he realized that the information was there, but AI Search wasn't presenting it as he'd expected. So this is great. Okay, so this was his original kind of feedback on his AI search summary. So he said this was the prompt. My project is an acc. How do I open it? Detached, Right? And now at first glance, this looks like it's giving you step one, step two, step three. But when we looked at the page. Now this is an old version of the page. I'm going to show you the old version. And then how we repackaged information on the page to really work in concert with AI search. So this was the old page. And to just briefly give you some history, what it's saying is that there are three different workflows that a person can do in order to open a detached model, right? They can either this is the first workflow, this is the second workflow, and this is the third workflow. But what AI Search was interpreting this as was step one, step two, step three, which is not what we wanted. So what we did. What I first did was I went to the video and I forgot to mention, he also included a video on here which goes. It's a quick video. It goes through each of the different workflows that somebody could do to open a detached model, right? So what I did was I went to the video and I added chapters. I added chapters for the first workflow with the timestamp here, the second workflow timestamped here. And I titled it exactly how he had it and Workflow three here. And then I just reformatted the page ever so slightly. So I said workflow one and then I linked that specific timestamp section. Same with Workflow two, Workflow three, still have the video here. So it was just a slight adjustment. And then now when somebody searches, they see this. I use the same prompt. How do I open a detached model in acc? Now it shows me. It says you can follow three different workflows. Workflow one and then the steps, Workflow two and the steps. Same with Workflow three. So it was just a slight adjustment that really made a difference with how people are consuming the information. And then also the validity of it. Right. The previous way the user would have been like, this doesn't exist. It tells me to click ellipses. And that doesn't exist. So it's really important to, you know, had he not tested that and provided feedback, I wouldn't have known that that was an issue. And we couldn't have. And we wouldn't have been able to adjust that piece of content. So another example of using video chapters was we had a continuing education course called Boma 101 and I wasn't able to attend. So I want to see if I can get a recap. And what it does really nicely is it gives me the course overview and then it's giving me these different sections. You can tell in the bold text that it's broken down by sections and each of these are citing a different chapter in that video. So if I missed it, or if I had to step away for a few minutes and I missed the part about deliverables, I can come here to AI search, ask it to give me a recap, and then jump straight to that section of the video using the citations, which is pretty cool. Okay, moving on to my next point. My next what we learned thing here is that best bets are also tried and true. We continue to use those as, you know, sort of to enhance visibility. So best bets you can set up. So if somebody searches Handbook or something, I can have our best bet come up. That I suppose I can just show you that, can't I? If somebody searches handbook, it's pretty likely that it's going to find our EUA handbook. It's going to give me a definition, and you can see here that it's a best bet. So this gives users also some confidence that we are promoting this as an item that we want them to find. So I won't spend too much time on best bets, but I highly recommend continuing to use those because it just really enhances visibility and drives users to. To resources that you deem important. So the final piece here of what we learned is to prioritize pages over documents. So this is a kind of an important one to us. At eua, we've been really heavily documented. We use documents to document how to guide, right. How to do this, how to do that, and then we upload those documents to connect. And while that's great and it works fine, inherently, AI search works better with pages. That's not the only reason why we're going to prioritize pages over documents. It's not only for enhanced visibility, but we, because we find this to be a more sustainable method of capturing and updating knowledge moving forward. I'm not saying that we're going to go back and turn every document we have into a page on Connect, but we're certainly going to take that into consideration when adding new material to connect or if we're getting questions on a specific topic, like maybe we want to take a look at the documents we have uploaded and perhaps make it into a guide or a page. You know, if we continuously have to update a document and re upload it to Connect, that's more steps for a community champion to take. And that's honestly more likely that they're not going to do it. So instead they can go straight to that intranet page, update it there, and again, it captures recently updated. So if I go to a technology page, let's see here, you know, I can see that this page was Last updated on January 16th. That's pretty good that that inspires confidence in me that this material can be trusted. And then we also don't have multiple versions of documents floating around both on the network and on the Internet. So another example of this is at eua we use workspot. Now workspot is kind of a cloud desktop situation and we have a lot of, there's a lot of troubleshooting, there's a lot of content we've had to create regarding workspot for people to work through their own issues and troubleshoot different things. So what used to be probably seven to 10 different documents, we evaluated that and since we were getting so many kind of like support tickets related to workspot, we were like, maybe this is something we want to consider condensing into one spot that people can consume this information. So I search workspot troubleshooting. I got some great ideas off the bat here on what I might want to do. And then a person might go to this guide and they might see something like, okay, yeah, I do want to know more about multiple monitors. What do I need to consider that? What are the pro tips? Workspace and browser? So it's just kind of considering the way that you're packaging information in order to for it to be delivered to the end user in a more streamlined fashion. Another example of this is how do I check out equipment? We've got some great guides on this. Again, it used to be a document and of course documents work well, but this is just, this is just better. This gives it to you right away and then you can see exactly where it's coming from. You can see that this information was updated on January 7 and get the full download of what's available. All right, now I'm going to go back to my PowerPoint here. So what did we learn? I talked about Q and A format. I talked about utilizing video chapters, continuing using best bets and prioritizing pages over documents. Now I wanted to get some testimonials because you heard a lot from me already about, you know, why this, why this tool is useful. But I think it's important to hear from other people too. So Maxim at our firm, he's an IT specialist. He's also very passionate about AI in general. He says using AI search helps save me time rather than using standard searching methods. Of course, he likes to think of AI of IT as a tool to help me solve potential business issues. Now, specifically with Maxim, right, he's in a role where he's fielding support tickets all day long from people. He can empower people to use AI search in order to self serve their issues a little bit more. Of course, we don't want to eliminate entirely the human interaction, but if somebody has a quick issue that they can try serving it for themselves first. Sorry, I don't think I was showing my PowerPoint. Then they can kind of self serve a little bit more. And then he can also use the knowledge of what people are struggling with to influence kind of the content that we're serving up on Connect and ultimately point people to that. Another testimonial I have is from AJ's Dant. He's our content writer, which he's really been instrumental during our pilot group. So he says AI Search is now my go to tool. It returns tailored, context specific responses from our archives, saving me hours of manual search. Connect is a goldmine of project and team information. And aj, as a content writer is often digging through years of past proposals and other documents to create custom project descriptions or service overviews. AI search not only streamlines that workflow, but it also empowers potentially other people at the firm to craft customized content by providing them with a strong foundation. It's really a game changer for how we deploy our collective knowledge. And I'll give you a really creative example of something that AJ did. So. Let me just switch here. So aj, this was before we had filters on the AI search, but what he did, I thought this was really clever. He searched for can you please summarize the content of the following posts? Right. And he copied and pasted. So there's a little bit of legwork that he did to do this, but I thought it was really creative and innovative. He copied and pasted the specific titles of the posts he wanted AI Search to comb through. So what it did was it spit out this incredible summary of each of these specific posts and they're all cited here, which is incredible. And then he asked the follow up question, can you please turn these summaries into a narrative about Eway's community engagement in these offices? He was really specific about using first person plural, do not mention specific team members. And then bam, you have this incredible summary. Something that, you know, might have taken him an hour to do. Just took him five minutes. I just thought that was a really, really creative way of using AI, AI search, and one that I in my role would not have thought of. That's why it's really important to have different voices participating in this because, you know, I'm not working with our content every day, with our marketing content every day. This was awesome. All right, So what's next for us? It's going to start with some AI training. We are launching an AI based training series for the year which is going to incorporate AI Search on Connect. It's not going to be solely both based on that. We're also going to incorporate Microsoft Copilot too, because that's part of our practice. But we're going to be communicating this through multiple channels to people. We're going to have, we're going to be announcing it in February and then start the series in March and then every month thereafter. So we're going to have monthly sessions. We're going to have training on AI Search and then my colleague Maxim, who I mentioned before, is going to kind of own the Microsoft Copilot session section of that. And we're also going to crowdsource topics from staff so that we can help them solve real issues using AI. It's really important to have that engagement from people because it's not as simple as if you build it, they will come, right? You have to show the value and that takes consistent training and collaboration with end users in order for them to see the value. Of course, this technology makes it really simple because it's impressive and cool and it's fun, but it takes that, you know, you kind of have to continually show the value and get creative with how you can use this tool to make your job easier. And then we're going to have the full scale launch, which we're planning to do. By full scale launch, I mean rolling the AI Search out to the rest of the firm beyond the pilot group. So we're anticipating doing that in March, if not sooner, but I think March it's going to be because that will coincide with our first training session. And then the final piece of this is just maintaining content quality. This is really important because you want to make sure that the content that we're presenting to people is useful and valuable and correct. So what I have already done is I meet quarterly with our practice area champions and community champions to sort of review analytics and relevant. Well, and then I'm going to also add reviewing relevant search feedback. I'm going to roll that into the agenda as well because then we can really dial in what we are, what we have available for people and how we might make it better. That's where that feedback really comes in handy. And also with the proper training, people will know that if they provide feedback, it's going to help, it's going to elevate this platform beyond what it is today. So that maintenance piece is really important for moving forward as well. And we're going to make sure to give that its proper attention. With that, that is the end of my PowerPoint, so I'll stop sharing.

That's amazing.

Thank you.

Tons going on. Take a breath. No, there was really good discussion in the chat. We've got some, some questions kind of teed up, but thank you for such a thoughtful setup for everybody. Julie Weston asked how feedback was collected from your user group.

Feedback was collected mainly by using the Feedback tool built into AI search. I really wanted people to use that tool so that it was somewhere that I could go back to. And I'm not sifting through emails and teams messages from people. I really wanted them to use the AI search feedback tool because then also Knowledge Architecture sees it and they can see if something's not working as designed or if maybe it's a priority that they for us, that they want to take into consideration. So as much as you can use that built in Feedback tool which all users have, that's what we did. Cool.

Julie, I can see that you raised your hand, but we can't enable your mic here, so you're going to have to put it in the comments or question. We'll come back to you. Can you show it, Andrea? Feedback?

Oh, sure, I'd love to.

I haven't seen it yet.

All right. Okay. So you're seeing Connect.

Yep.

So Feedback tool is in the settings gear under Search optimization. And then what it looks like is this sorted by the timestamp and of course like any directory or library, you can adjust those settings and then what happens is if I click on this, so this is a recent one from Tony. I click on it and I see exactly what he saw. So I see his question and I see the search summary and I see what it was, the citations as well. So then I'm able to kind of take that into consideration. Here we go, take that into consideration. And then what's great too is the users can provide a summary or a little description of why they're rating it that way. You know, I would have expected to see, to have seen this instead of that or this looks great. Sometimes I use this section also as sort of a checklist. So like when I was searching benefits enrollment, at one point I said this is good but need to clean up old posts related to this, to this topic. And then I can come in here and once I've reviewed it, once I've done that action item, I can check it as reviewed and then I'll just show here this is. If you were to scroll all the way down to the bottom, there would be a section that says reviewed. So I can go back and see all of the things that I've reviewed that I kind of checked off my list, which is a really great way of kind of keeping track of what you're doing, what you need to pay attention to, what's a priority versus what's not. It kind of keeps things organized into nice buckets for you that way.

Right on.

Cool.

Maybe we should stay in search optimization. Well, kind of. So Joe wants to know how are you preparing to launch AI search company wide without search analytics to inform you on what staff are looking for and if they are successful or not?

Because we have feedback. So without search analytics you still have a way of seeing if people are finding what they need, if they are providing feedback. So let me just search Bluebeam. If I search Bluebeam, it's going to give me what I need. Okay, now say this is exactly what I needed. You come to this rate button here, you rate it. I say five stars and I say this is great. Okay, you can make it anonymous, anonymous if you want, but I'm going to encourage people to not be anonymous just because if I have follow up questions I want to be able to talk to them and get to the bottom of what that might mean. Now say this isn't great. Say I, you know, I would have expected to see blank, whatever, fill in the blank, whatever that might be. So without search analytics, you still have a way of seeing what people are searching for and if they're finding what they need, but they have to use the rating system, they have to provide feedback. So that piece of it, just that educational piece of helping people understand that this rating system is, this rating tool is the way in which we know what's working well and what we need to work on. So it puts a little bit of the onus on the users to provide that feedback. But then once it's there, I can go into search optimization and take a look at what people are finding, celebrate successful searches, and then pay attention to the things that, you know, perhaps aren't performing well, like EUA line patterns. So using the feedback tool is how we're going to sort of manage that feedback or manage the understand what people are finding and not finding and then we can base our priorities off of that.

That's a great answer. And just to build on that. And Joe, you and I have talked about this, but I'll say it for more broadly. We are building search analytics. We're building it currently. It's not that far off from coming into the product which will augment what you have in the feedback. It'll just show you all the queries, queries with zero results. You'll be able to understand what users are searching the most. So I'm just dropping this in the, in the chat, a link to our release notes that kind of talks about search analytics a little bit more. Let's see. So a question from Melanie. Do all of your marketing and communications and business does all of your marketing, communications and business development content live on your intranet? So I'm guessing past proposals or boilerplate language or project sheets, that kind of thing.

We have a good amount of it on Connect, so we do have a whole section of previous proposals and interviews. So you can see we've got 711 documents here. We're actually BD and marketing is the next community on Connect that I'm going to be working with the marketing and BD group to revise. It's one that we haven't paid attention to in a while. So we're going to be talking about those things like do we want to have all of these previous proposals or do we want to maybe do it up to a certain year or something? So that's still kind of in the works. But it has proved to be quite useful when searching for this marketing material especially too when you have, you know, at first we were kind of thinking like maybe we don't want all these super old proposals and interview materials. But because you have the filters with AI search now, I can take a little more control of what is being presented to me. So if I only want to see it citing proposals from 2020 and beyond, I can apply that filter to my search result and then that other stuff is a moot point anyway. So right now, yes, I would say a lot of our marketing material lives on here, but this is a community that we're going to be spending some time with in the next couple of months to just audit and make sure what we have here is useful to people. So TBD on that one.

That's great. Colby asked a question and I'll take the first swing at it. And then you might want to editorialize on it. So Colby asks, will the AI tool also search through directories? Say I wanted a list of employees licensed in a certain state or wanted to search project data on project pages. So we can do that today in kind of rudimentary form. It's really good if you just search on the name of a person or the name of a project or a client at synthesizing information from both the directories, but then also unstructured content like videos and posts and those kind of things. So that is what it was kind of designed to do in the current version. What we're currently building now is a more advanced project and employee query support. So, you know, how many projects have we done over X square footage in Wisconsin or, you know, who are all the licensed PEs we have in the state of Michigan? That is what we're actively working on now, and that will largely leverage directory data. And so, Andrea, I'd be curious, kind of like your, your kind of impressions or thoughts so far on employee and project search, like, what it's doing well and like where the line is on where it's. Where it's been useful.

Yeah, I think I've had. I haven't been doing as much of those queries. I think aj, whom I mentioned before, has been kind of searching on that. And I'd say so far, I don't think it's been accomplishing what he's been trying to do. You know, he'll say, how many can I see all of the proposals that such and such was mentioned in, and it's not exactly giving all of them, but we also have so much content on here that I'm not sure it could list out all of the proposals that a certain person has been mentioned in. Yeah, I guess. I guess I can't really speak to that as much. It's been working when I have, like, I had that firm licensing, you know, it recognizes that I have Scott Ewan on this page, but this isn't a directory, so I guess I haven't been playing around with that a whole lot.

You raised a really interesting point, Andrea. So we designed, kind of the initial release is meant to be more of a knowledge based search, meaning I want the best answer to this question. And so the example you just gave where I want all of the, you know, all of the examples where someone was mentioned a proposal we have not designed AI search to handle yet, but that is exactly what we're working on to be able to handle both. I want the best answer and I want all the things, whether that's all the employees, projects, mentions in a proposal, videos that have a mention of this thing. Like that's the next kind of major query type we're going after. Joe. Oh, this is a good one. Joe wants to know, how did you communicate to staff that you were deleting their old post content to improve AI search? What was the reaction to this? And have you edited the content of other people's old posts?

Yes, I have. I have edited content of old posts, if it makes sense to. And I guess I want to be really clear that I'm not deleting posts with really meaningful information. Like you can even see right here. Sometimes on Connect we talk about where there's free food around the office, right? We've got on the terrace refrigerator, there's free snacks. I'm deleting old stuff like that to just make it more clean. Not that AI search is necessarily gonna pull, you know, a post about free sandwiches on the terrace, but it might, it might mention something. Sometimes people get clever with their titles or something that includes some architectural term to talk about how there's free cupcakes in the third floor kitchenette. So a lot of the stuff that I'm deleting is stuff that isn't relevant. Yeah, I want to make it really clear that I'm not just like bulk deleting stuff just because, you know, I'm not going like delete every post prior to 2020. Like there is definitely some human intervention in that. And I'm making sure that old information that I'm deleting is really okay to delete. And in some cases I'll run it by our DT team. Like if before I delete a post from somebody that's not at the firm anymore but might have really relevant content, I'll be like, hey, Tony, can you look at this post and tell me if it's still relevant? And then what he might do is say, yeah, it is. And then I'll say, okay, how about we mine that information from the post and add it either to a document or make it into a page and then get rid of the old post. That person isn't here anymore. And, you know, having old posts, like I mentioned during the presentation too, a post from 2016 doesn't always inspire confidence in the end user that that information is still relevant. And even if it is, though, you know, they might look at that and be like, there's no way. With how rapidly technology changes and information changes, there's no way that this post from 2016 is still true. So we're, we are diligent in how we're addressing these things. And then some of this stuff, like I mentioned, with the benefits, enrollment. Right. I've talked with HR about all of that. And in fact, they're the ones that are like, can we delete this old stuff? Which I think is great. I haven't just deleted anything without sort of making sure I'm doing the right thing first. But I've talked to all of the kind of practice area champions or community champions and just said, hey, this is what we're doing. This is why, you know, I'll demo AI, search for them and say, this is what happens when we have old posts on here. And then I empower them to do the deleting because they're the subject matter experts in this case and they can make the calls if something should be deleted or not. And if they have questions about how we can translate that information into a page on Connect, that's kind of where I step in. So it's kind of a, you know, kind of a little dance.

Yeah. And I think Andrea also joined. They were like the third firm in the private beta, and we in the beginning weren't very good. Well, I shouldn't say weren't good. We didn't even try to weight content to give it to search. So since then, you know, we put in weighting based on content types. So, for example, posts rate higher than documents, which rate higher than pa. Sorry. Pages rate higher than documents, which rate higher than posts. We also wait for freshness. So something from this year is more relevant than something from 2016. And we also wait on best bets. So there's a variety of things that we're looking at to try and get the most relevant content back to search. And I think that's helped a lot. I think within the community there are some things. I saw a comment in the chat about putting expiration date on temporary posts, that's something we're considering, but we're kind of the first order of businesses. We're prioritizing the algorithm to make sure we find the most relevant things first. So that's good there. I think we've got a couple minutes. Danny wanted to know the timing of your cleanup. Andrea. So did you do it when it was just you and Ken and a couple people from Design Technology? Did you wait until you had more users in to get feedback? Have you done it continuously? How'd that work?

We just. Right off the bat, I just started right away. Like when I said I started testing as soon as it would drum up an old post from 2013 and if I knew it wasn't accurate, I just deleted. I just deleted it. And then. And then I got a little more strategic with it. Right Then I would meet with the pilot group and. Or just, you know, it was kind of on a in the moment type of thing. Like, we didn't really have a whole lot of, like, meetings specific to AI search. It was just in the moment, it was all happening so quickly that we just wanted to kind of get ahead of it. So in the moment, we were just doing it. And then I taught people how to do it themselves and how to use the tool to sort of wade through that outdated content. Like, like you said, Chris, we knew that the waiting of the search would eventually sort of refine itself, but we really wanted to take this opportunity to clean it up. You know, if we can start with a baseline of just really rich current information, it's only going to be better for us. And then it's less of a massive cleanup as we go and it's more of maintenance.

Amazing. Yeah. And maintenance, that's really a lot of the journey.

That's it.

All of us that have worked in knowledge management know. Well, Andrea, I can't thank you enough for sharing your work with all of us. This is really impressive. I can see in the comments it was very active. Probably didn't see it while it was going, but, like, lots of thoughts there. I think people really love what you're doing. The recording just for everybody of this intranet tour will be posted on our website within the online K community and sent out via email next week. You can learn about upcoming events on our website. I'm going to drop the link in the chat and we look forward to seeing you at our next event and in the community and online. So thanks everybody. Thanks again to Andrea. Be well.