Responsible AI Frameworks Every Founder Needs to Know with Fion Lee Madan

I actually met my co-founder at Founder Institute spring 2020 cohort at Kitchener-Waterloo, which is about 2 hours away from Toronto. We had our in-person session with 30 people in the cohort 2 weeks before the pandemic lockdown. I met my co-founder 2 times in person. Then because of the pandemic lockdown, everything was moved online. It's the first time the world has been doing these accelerator online. A few weeks later, we were down to 11 people in the cohort. That's when I look around the room and say, hey, David, my current co-founder, our CEO. I said, David, you are the smartest in the room and you have the best idea and it's something that I'm passionate about and technically I can contribute to, I asked him if I can be his co-founder and the rest is history, as they say. 5 years later, we're still standing. And if you've read or thinking about joining startup or starting your own startup, I'm sure you've read some of the stats about the survival rate of startup. I'm here to help every bit, even if it's a little bit that I can to make sure you have a better chance of succeeding, starting with how do you build responsible AI from day 1? My background is in computer science and I picked up an MBA along the way. I started my career actually right before the internet bubble burst. The AI transformation we're currently facing is actually the 4th wave of digital transformation that I've experienced in my career. So happy to talk about that some other time. But today I want to focus on what we do here at Ascendian. So I'm officially the technical co-founder and chief product officer at Ascendian. We've recently acquired a company called Eng AI based out of Stockholm, Sweden. Formerly, we were called Fairly AI. Now, with the joint company, we've rebranded as Asanion. Asanion came from the famous sci-fi author Isaac Asimov. If you've watched the movie I, Robot with Will Smith, Asanion essentially are the good robots. We want to help companies like yours build good AI. How do you do that? We provide a reliable fast track to AI governance and beyond. Why you need it? Accelerate AI adoption and minimize AI risk. So some of you folks that are already in the space probably have seen the recent MIT report that says 95% of generative AI pilots at companies are failing. One of the reasons is, according to Gartner, and for those of you who don't know Gartner, they are an industry analyst, So they find that a lot of companies that are actually succeeding, and 45% of them are actually keeping their AI projects operational for at least 3 years. So clearly, some people know what's the right thing to do. And if you actually read the Gartner report, it put AI governance at the top of this technology stack for the, for their AI maturity. The other thing that is moving very quickly and why people are talking about AI governance is the fact that for the very first time in, in history, the technology is moving very fast with a new LLM, large language model, coming every other week, for example. But the regulatory side of things are also moving very quickly. This is our global AI regulations tracker. It has over 130,000 viewers and subscribers now. And as you can see, all over the world, there are AI policies, guidelines, and regulations that are being proposed, passed, or in effect. And some of you, if you're operating in EU or are planning to sell to the EU countries, there is actually an AI regulation called the EU AI Act. It's passed last year. This year's already enforceable for the large language model provider as well as what they call prohibited use case. But next year is when every single company that wants to sell or operate in Europe needs to be compliant to. And it is very much like GDPR, Data Privacy Law. It's the same governing body that came out with GDPR and EU AI Act. And that's why a lot of people in the industry are looking at the EU AI Act as the gold standard for AI regulations around the world. Very much like GDPR became the gold standard for data governance around the world. And the fine is actually significant. It's 7% of global turnover or €35 million if you're found to be not compliant. So why AI governance matter for your startups? So EU AI Act is a good example of that legal compliance. So a lot of startups, right, And we are still a startup ourselves, even though we're 5 years old. But if something that you don't have to do, it kind of falls lower in your priority list. But when it comes to legal compliance matter, right, that is not an option, right? It's a must-have. You must do, otherwise you cannot sell, you cannot do business. And lawsuits are happening at over 1,000% increase rate right now. In the States. So preventing costly mistakes before they happen, right, by doing AI governance to foresee what legal compliance requirements you need to have in place, especially if you're in regulated industry. And I can tell you right now, right, with the EU AI Act, for example, if you're building any AI product or services that touches human subjects, whether it's HR use cases, A lot of educational use cases, a lot of AI companions, right? They all touch human subjects. So those are already considered high risk. Investor relationships. So we're increasingly seeing that investors are being more sophisticated now, want to understand where your training dataset is coming from, for example, when you're building an AI product. Because as you also may have heard or read, right, all the large language model developers are being sued. Most of them, I wouldn't say all, I'll correct myself, being sued for copyright issues. So there's twofold. One is the risk appetite of your organization and the customer you're selling into. There are companies that choose to continue using OpenAI, for example, knowing that they are being sued for a copyright issue, but there are companies that chose not to. So that becomes a question for your investor, whether they care, what's their risk appetite of investing in a startup that may be using data that are copyrighted. Those are things if you know that this is potentially issue, you would want to make sure that if you're thinking of training data, make sure you understand where the training data's license come from and whether it is an open source license that you can use freely or there's actually copyright issue. The last thing is customers. Because there's now awareness of these legal compliance issues, the EU AI Act is definitely one of them, and then different countries are adopting their own version of EU AI Act right now, especially regulated industry that are used to dealing with different regulations like banking, healthcare. The procurement team are already asking some of these questions like, show me that you're doing proper AI risk management, show me that you're compliant, show me your data provenance to make sure you're not using copyrighted material, right? So we're seeing that adoption at the very top-tier consulting companies, the big tech companies. And because these big tech companies are now buying, acquiring other AI solutions vendor from startups, same as the consulting companies advising their clients, their Fortune 1000 clients who are purchasing these AI products and services. There's a whole flywheel going on in the regulatory compliance space, AI governance space. People realize that it's no longer 2020 where you can build something and push it out and have no consequences. And then with recent losses like Character AI, some of you might have read that a 14-year-old boy committed suicide because the character AI encourage him to, right? So all these lawsuits is also bringing awareness to not only the legal community, but also the investors community, as well as the customers or the people advising the customers, the consulting companies. So all of that, if you can prevent it from day one, why not, right? And I know a lot of you, maybe it's just building in low-risk use cases, not the high-risk use cases that I mentioned that's touching human subject, right? And in those cases, you really just have to prove that you're not a high-risk use case, and then you can continue doing your business, right? So the EU AI Act does require you to do an assessment to show that you're not high risk, right? But if you're high risk, you're actually required to do third-party audits. So all those things is why you need to figure it out right now and allocate appropriate budget, especially in high-risk use cases, to do that third-party audit if you want to sell into the EU. So you can feel free to ask me more questions later. This is very high level, and yeah, there's definitely nuances on, you know, specific use case, industry regulations, jurisdictions, etc. So AI governance is one piece. The industry we're in, because we work with, you know, the very thought leaders, the people who are running at the forefront, right? And people do realize that a lot of these AI governance are very static compliance-driven checklists, right? It's how some people tend to implement them because they think that it's the same as GDPR where you just have to say, oh, I have a system to check consent for data privacy, right? Like your opt-in, opt-out checkbox, right? That's one of the GDPR requirements. Make sure that people are opt-in. And then you're set. But no. So the EU AI Act actually, for example, asks if you've done the proper testing, right? So in the past, if you say if you've done the proper testing, then you're done. But with EU AI Act, you actually need to show how you've done the testing, how do you know the test results are good, right? And because fundamentally AI systems are probabilistic, so for those of you who may not be thinking about this issue, Traditional software, 1 1 2. If it doesn't equal to 2, you know it failed. You cannot go to production until that's fixed. But with probabilistic AI systems, and now thanks to ChatGPT, everyone understood, even if you ask ChatGPT the same question, 1 1, should it equal to 2? Sometimes it gives you the right answer, sometimes it doesn't. So it really depends on how you ask the question and how many times you ask the questions. Hence, this whole new concept of AI assurance become one of the hottest research areas right now. We are part of that. I'm actually personally involved in 3 different global AI assurance think tank and working groups and pilots to solve this issue, figuring out what to test, how to test so that you have the confidence to we approve these AI systems to go to production, right? Back to the character AI scenario, right? Like thought leaders are definitely trying to solve this. So if you're dealing with any AI compliant companion, AI coach type of AI therapist, AI mental health type of application, right? Those are already deemed high risk. And like definitely there's a lot of thought leadership going on that I'm happy to share. I think I put my LinkedIn in the chat. Feel free to connect with me. But back to what's the definition of AI assurance, right? Very much like AI governance, depending on who you talk to, you know, people tell you what it means differently. But standards definitely emerging, right? How do you do testing for AI is another word for it. Of course, you need to do testing before production, before deployment. But because of the whole probabilistic nature of AI and that once they see real-life data, the behavior may change, that's why you see continuous monitoring. And I would say this is the number one difference for all AI regulation that's been passed, proposed, or being considered. It's like, do you have a way to do continuous monitoring, right? And then the incident response is also very new, but it is very similar to the whistleblower in some industry, right? A way for your end user to report things that are wrong with your AI system. That is actually a part of the EU AI Act as well. You have to have a post-market monitoring system to allow people to file a complaint with your AI system. And then regular audits really is for regular applications that already have applicable regulations or are deemed high risk in the EU AI Act. So for example, HR regulation in the New York State or New York City. So if you have an HR application that is using AI to do employment decision, hiring, firing, promotion, resume screening, et cetera, then you need to have a third party to audit your AI system in order to be New York hiring law compliant. So those are the type of things that why you need to really understand the whole AI governance and AI assurance space. When you're building your AI solution or product. The good news is there's actually a lot of free resources, open source resources. You don't have to hire consulting companies or use a tool like Xennian, our tool, if you have the time and resources to do it. There are a couple of popular ones globally. The NIST AI risk management framework is well recognized, and it is from the United States. This is a United States organization, but people in different sectors, different countries have adopted this framework. And it is because this is open source and you can download the playbook. It is something that you can actually do yourself if you have the time and resources. And it's really just a matter of prioritization, right? That's how I would put it. And I have a slide next to go over that that a little bit in detail. But if you want something even simpler, I actually have recommended to many startup founders using the Microsoft Responsible AI Framework 2.0. It has 6 principles. It covers the most important things that a lot of people, a lot of your end customers and investors care about. So take a look at that. It's also open source. It's relatively easy to, you know, going from zero nothing to something. Both of these frameworks allow you to do that relatively easily. So for example, the NIST AI Risk Management Framework, it has 4 modules. Garfin, like establishing people essentially. It's all about people, process, and whether you do things, collect evidence in a spreadsheet or in a software system, really depends on if you have the budget to use the software, of course, the automation make it easier. If you don't, at least do it in a spreadsheet, right? So governance is about like having the right foundation models, foundation layer, like do you have the right policies, controls, people, and processes. Mapping is now you're starting to identify the risks that are specific to your product, service, use case, context. Measures, once you've established the policies and the controls and like the actual risks specific to your solution, that's when you can then quantify and measure this risk, right? And ultimately, once you understood what your risk exposure is, and the other word that I want to again repeat, what your, you as founder, your own risk appetites, or your investors, or your customers' risk appetites, that's how you know what you can manage, what you need to manage, right? So a lot of times, if it's determined that your AI use case is low risk, you don't need to spend all that money and go through third-party audit because it's actually not a requirement, for example. But if it's deemed high risk, and if there's regulation, then you definitely want to go through the third-party audit. So now then the question really becomes if you're medium risk, And that is sometimes subjective, sometimes really depends on your end customer, your own personal tolerance for risk, right? So that is where the expertise can come in. But at startup stage, usually what we see is like use cases that are clearly high risk, and then there's use cases that are clearly low risk. And the high risk is like all about not just the AI governance part, but also the AI assurance part. And luckily, there's also free AI assurance framework now because the industry and the experts and the think tanks all recognize that startups can be like OpenAI or Microsoft, hire a team of PhD to do AI safety and trust, right? That's essentially what the whole AI safety and trust and OpenAI and Anthropic, all these big tech companies do. They look at how do you test this model? How do you break this model? How do you put guardrails in this model? How do you put in not only code-level guardrails, but also policy guardrails so that if something does go wrong, because all these AI systems, very much like cybersecurity risk, there's a lot of unknown unknown risk. You can test as many times as you want before deployment, but ultimately once in production, there are still risks. We call that residual risk. How do you cover these unknown unknown risks? Policy guardrails is one way of doing that. This is like more mature companies do that. And that's why you're seeing Microsoft, Google, AWS, all updating their acceptable use terms of use to put in a clause about their liability. If their AI goes wrong, they're not liable, essentially. So these are practices that actually, once you understood all these guardrails, tools that you have, and frameworks that you have at your disposal, you can implement a clause in your acceptable use. To protect yourself legally, liability. Again, not again, sorry. I just need to say, this is from our lawyer, our director of AI compliance policy. We are not giving legal advice. These are purely suggestion. We highly recommend you talk to your own legal counsel. Having put that disclaimer out, back to my founder's hat. Here are tools that you can adopt. Right now, AI Verify Foundation from Singapore is a Singapore government-funded nonprofit that is trying to bridge the policy gap with the technical gap, right, what they see in the AI assurance or AI testing space. So they have a toolkit that you can use. They have some open-source tool that you can use. So explore and take a look at that. Now, some of you may ask, why Singapore? So Singapore actually came out with the first AI governance framework at Davos, the World Economic Forum, back in 2019. It's the first government that actually have done that. So they are considered thought leaders and pioneer in this field. And there are some stats that suggest Singapore actually have the highest number of data scientists per capita. Their population is very well-versed in the data science/AI domain already. ML Commons is funded by the big tech, so they are trying to establish benchmarks so then we can use to calculate some of these AI assurance testing, whether it's better than your out-of-the-box GPT-4. If your application is performing worse than the out-of-the-box GPT-4, then there's something that you definitely must do because you're performing below benchmark. This is not only, I guess, a compliance issue, but also just protection for your product to show that you are actually better than just a wrapper around GPT-4, for example. Those are things that you can show to your investors and your customer as a sales enablement tool and framework and benchmarks. That's why you use benchmarks to show that your solution is performing better than your competitors or these out-of-the-box solutions or models. Now, for those of you who don't have the time budget, or maybe you're actually not a technical founder, you 5-code your app and you need some help, And you need to, or the other part is like you're actually more mature in your sales cycle and you're now selling into Fortune 500, Fortune 1000 companies, and the procurement team asks you, show me your AI risk management practice, show me your AI governance practice, show me that you've done the proper AI assurance testing, right? All of that stuff, yes, you can use those free tools, do it, and put together evidence and document. Or you can take the easier route, which is ISO 42001. It's the world's first management system for AI standard. So for those of you who've built software or have already gotten your SOC 2 Type 2 or ISO 27001 management system for information security standards, 42001 is modeled after those, right? So it is a certifiable certificate that you can get to prove that you have proper policies, controls, process, people, process, and technology in place to done proper responsible AI to govern and test your AI system. It is really comprehensive. It is really good for you have nothing and in less than 6 weeks, we've done it in less than 6 weeks, right? From nothing to fully certified by a third-party accredited auditor. So again, back to what's the urgency for a startup to do that, right? So there are companies that are truly operating in the high-risk use cases, right? Your urgency is that if you don't sort it out from day one, you're not going to be able to sell. And worse is that you'll get sued. And even worse than that, you actually cause someone to commit suicide, like Character AI, right? You want to prevent all these things. If you can. And there are tools available, so why not? Then if you're— the other part is trust. A lot of companies are worried about buying AI tool now because of all these noise in the news. Some are noise, some are like truly scary stuff out there. How do you distinguish from noise and you're having proper policies controls? Again, it's all about people process. Technology, right? So 42001 would guide you through all of that, and you can get a real certificate that says you've done everything, right? So it's not just you telling your customer that, yeah, yeah, yeah, I've done all these things, but it's actually certified by a third party, and it becomes a true differentiation for those that have done it or those that do not. And now you may be wondering, what's the cost? So here in Canada, there's actually a lot of government funding that people can tap into and happy to share with Canadian founders. Just connect with me again on LinkedIn. That allows you to do some of these things with some subsidies or 100% covered depending on what it is. But if you're doing it yourself, you can go and buy the ISO 42001 standard, for example, for $200 on the ISO website. You can work with tools, services, like us, right? We charge from $5,000 and up for startups that are less than 20 people, right? And then the actual third-party certificate right now, what we're seeing, the lowest cost we are seeing in the market for startups are like $4,000. And then because this is not once and done thing, In order for you to maintain your certification, there are annual maintenance costs that starts around $5,000. So these are things that, as long as you understood and you feel whether it will help you with one of those things I mentioned, whether it's legal compliance, investor, customers, sales enablement, you need to understand the cost and budget it in, just like everything you do at a startup. There's 1,000 things you need to do. Do you have money to do it? Do you have time to do it? And how do you prioritize it, right? So now that you understand these things exist, then you have more information on how to prioritize responsible AI, how to prioritize AI governance, how to prioritize AI assurance, how to prioritize 42,001 in your roadmap as you're building, you know, your startup. You know, it's a marathon but running at a sprint speed, right? That's the definition I like to use when we're building startup. So what you can do, and I think this is my last slide, right, this week choose your framework, whether it's NIST, Microsoft, or 42,001. Go educate yourself, right? And then in the next 30 days, once you pick it up, right, there are some simple things that you can already do and implement using these free toolkits. And then next 90 days, if you have sales in pipeline, if you're operating in high-risk, right, domains, Those are things that you definitely want to start sooner rather than later because why build an AI model and then realize that you cannot pass legal compliance and you cannot sell, right? You wasted all this money and time. And the other part of this is it's really hard to reverse engineer response, reverse engineer responsible AI, right? Especially in these early days, if you're hiring developers to build your AI product or service, right? You may have turnover and you don't know where your training dataset came from, right? Those are like real issues that we see very often, even at big companies, but definitely a startup, right? When you have turnover, you don't know where your training dataset came from. You don't even understand how the training models trained, right? And then you cannot get compliance because you don't have those evidence and documentation set up from day one. So if you do it right from day one, you would have known where the dataset came from, what models have you tried in order to select the right one for your specific use case. All that stuff is wrapped into these AI governance and AI assurance frameworks. I'm going to end here. Go visit our website. There's a link there you can click for a 15-minute free consultation discovery call with our team. Happy to help. Back to, we started at Founder Institute and we actually got fast-tracked into the funding lab at Founder Institute. and that really helped and propelled us. We did Techstars after Founder Institute. So many, many good things come from when I met at Founder Institute, Kitchen Waterloo in the spring of 2020, and found my co-founder. So I have many, many things to be grateful for, and all because of Founder Institute. So happy to give back and help any way I can. With that, I'm gonna stop sharing and do a Q&A.

Hey, awesome, awesome presentation. A ton of really great insight, a ton of things that I think a lot of people, even as, uh, I think people who might have been founders at the earlier stage with AI startups, they might not have thought about. Uh, number 2, uh, thank you for the kind words about FI. What's funny is we've actually had a lot of our successful founders actually meet within their FI program. Another one is Udemy. I know Gagan, he met his co-founder there as well. So yeah, that's really cool to hear. I wasn't aware that you guys met at FI. But Fionn, I know we don't have a ton of time, under 30 minutes, but let's get into questions. I know you answered this briefly, but I wanna kick things off here. Let's assume you are a founder who's still kind of at the ideation phase. Maybe you're in a program like FI or something of the sort and you have an AI startup. How early is too early when it comes to thinking about responsible AI?

So I can tell you some. So we have two very disjointed set of target customer right now. We're helping the largest companies in the world. I love helping startups also. What we build for the enterprise is very configurable, customizable, and then we just configure one of those into a vanilla startup solution. That's how we're doing it today. But if I take the best practices of all these customers that we have at the tier 1 level, they do it even before they write the first line of code. Back to, you really truly need to understand what's the risk of your AI use case, right? If this is already deemed high risk, you need to budget in your longer testing cycle, right, back to the AI assurance. You need to budget in getting that third-party audit, right? That's something that you may consider figuring out how much does that cost. Does that cost $10,000 or does that cost $100,000 for your specific use case industry, right? And then like that would justify how much money you need to raise, for example, right? or how much you need to get from your piggy bank if you're self-funding. So all of that is why you need to do it from day one. And then back to some of the examples, like you said, I mentioned, right? A lot of these are easier to start from day one if you already documented, you know, today I tried GPT-4 and tomorrow I try GPT-5. Now those are the type of evidence that some of these auditors want to see, especially in regulated industry. What are the test results of all these things? You can say that you've tested it, but actually showing that here's my test results, everything's locked is what the compliance officers and the regulators want to see.

Awesome. Something that you talked about was the level of human interaction that comes with AI deems it like the more interaction, what I understand is the higher risk. We had a question here from Samuel, I'll just bring it up on stage. They were asking, is AI tutor considered a high risk?

I love this question because, for example, it's tutoring math or English. They might give you the wrong math answer and they may fail their exam, but that's not going to cause somebody to kill someone. You would think that's maybe maximum, medium risk, low risk. But studies and research and actual with some customer have shown that, If the student talking to your AI tutor is depressed because they literally just failed their entrance exam to their dream university, and they started talking to your AI tutor about committing suicide and self-harm, do you have the proper guardrail in place? Even though your AI tutor is not supposed to talk about suicide, but the fact is AI chatbot, and that's why there's a lot of focus on AI companions and these use cases now. If you're interacting with a human, you really can't control what the human asks. You need to make sure you test, and there are standards already emerged. If you look at ML Commons, test for self-harm, test that you cannot generate harmful content, test that they won't use your AI tutor to generate bomb recipes, for example. Those are minimum standards that industries are now establishing. Again, if it's as easy as just running through, you know, a checklist of things, why not, right? You're going from zero to something. Back to, is it high risk, right? There's two things. Back to what does the regulation say, what is your founder's risk appetite, and that's determining what's high risk or not high risk.

Awesome. I hope that answer was helpful. Uh, going back into mapping, uh, we have a question here from Elizabeth. She asks, would mapping include the need for guardrails and identifying what these guardrails are for?

It's similar. There's code-level guardrails and policy guardrails. If your application is external-facing, then the standard guardrails right now is make sure that you cannot use your I'm assuming it's a generative AI use case to generate child sexual abuse material, for example, or generate bomb recipes, for example. Those safety content guardrails, some of them come out of the box if you're just using GPT-4 as a wrapper and no fine-tuning. But studies and research and what we've testing have shown that, A, some of the fine-tuning and application to change these guardrails. So, that's why you need to, just like when you build any software, you need to do independent testing, not just rely on the out-of-the-box configuration from the foundation model providers. And then the other part of it is there are certain foundation models pretty much failed every benchmark when you do safety testing. I won't name those models. You can look it up. It's Stanford leaderboard for LLMs. In those cases, if you chose to use those models, then you definitely need to put in your own guardrails. Those are all code-level guardrails. Then I mentioned already, so the more sophisticated and if you really want to protect yourself, you need to put in the policy guardrails as well. Awesome. It is sometimes industry-specific also. But then a lot of general compliance requirements like Privacy regulation, make sure it doesn't leak PII, personal identifiable information, right? Those things are pretty standard across industries now.

Okay, so because a bunch of people are, you know, asking, just to be completely sure, the best practice as an early-stage founder who's building an AI application or startup is begin by documenting everything, right? That would be probably your go-to.

I love that because that is actually the concept that we try to instill in the larger company from day one, 5 years ago, and you guys got it right away. Yes, that is the best protection, not just from, you know, things are changing so fast in the startup world, but the fact is you want to protect yourself. Studies have also shown that by documenting, whether you call it in traditional software where we already do this, it's really not new, it's functional requirements, business requirements, where are your prerequisites, all your test cases in traditional software, same thing, you document everything. The additional ask, and if you're going through the high-risk, is really the data provenance part of it, like where your training data come from. I feel like that is like the biggest delta that people are trying to understand and grasp. So, as long as you like also track like where did you get your training dataset, like their licensing especially, right? All that stuff. But then the next is of course like figuring out like the laws and regulation you need to be compliant to. I would say those two are probably most important. From day one.

Okay, so we have this question here from, uh, Matthew. Under the ISO standard that you mentioned, how does an organization building an agentic AI system that makes decisions on, on taking action in cloud environments outside of the organization control? So essentially, like, if you're building an agent, right? And we have another question that's similar to this where they're asking, what if I'm building like an AI you know, while coding application, right? Do I need to be cognizant of, you know, these guardrails or responsibly using AI?

So yeah, that is a great question. So I would say there are existing regulations, compliance policy requirements that already apply. Doesn't matter if agentic AI, generative AI, predictive AI, or traditional software. Back to privacy regulations, a lot of cybersecurity compliance questions around data handling. So those things, yeah, you need to take care of, right, from day one. So specifically, AI, agentic AI system, it is relatively nuanced. The regulators are trying to figure out how to regulate it. The governance world, the policy side of things that teams that I work with are also trying to figure out a little bit. The main difference, right, for the thinking there right now is agentic AI system usually have multi-agents. Each agent have their own LLM. They could be the same LLM as the next agent, could be a different LLM. You need to test them. You need to test the end-to-end agentic AI system, but you also need to test individual agents. Then it's the orchestration between the different agents. You also need to test and govern, and then is within the agent, you have the LLM doing stuff, but you also have a set of tools that the agent can perform, right? So all of these things combined into, there's definitely more, more, more things can go wrong, right? So therefore, you need to spend more time to test and make sure. And back to, right, there's one more concept that I haven't really talked about is risk tiering, right? Is it high risk, low risk? But there's the regulatory compliance side of what's considered high risk, low risk, medium risk. There's actually another side that is based on profit and loss. If your AI goes wrong, is it going to cost your company to go bankrupt? Is it going to lose all the money? If it is, then it's considered high. The banks require high materiality, so it's like it will lose a lot of money and therefore it's considered you need to do more proper testing.

Awesome. Great answer. We have this question here which super interesting. Here, yeah, so Grace asks, AI governance is key to risk management. Do you see consolidation in global AI governance, a regulatory body, or ensuring compliance? Yeah, as somebody who's, you know, within the space, how do you see things going right now?

Yes, so a lot of standards body and nonprofit organization like AI Firefly Foundation, right? So they already cross-map their testing AI assurance framework with like the NIST controls, the EU AI Act, for example. So the popular one, I feel like there's a good support there right now. So you don't have to actually, you know, fill out 10 different compliance forms and be compliant. So yeah, definitely something that the ecosystem sees the need for, to harmonize. We call it harmonizing the different standards and regulations.

Okay, to get a little more specific, we have this question here from Dan. Dan asks, are the AI-specific government governance laws for an application that is in the medical space and must adhere to HIPAA patient privacy laws— is the expectation that the AI is built with HIPAA compliance built in? Because this is essentially two different levels of compliance that you need to be aware of, right?

Yes, bingo, you got it. Yes, existing regulations still apply. Doesn't matter if AI, not AI.

Awesome, there we go. Okay, so we have a bunch of different questions about the res— if there are any resources, or I know you did talk about risk steering as well. Like for example, when you're building out your startup, right? How do you determine high risk versus low risk? Like, you know, is there some sort of scale that you should abide by, especially when you're in the right— in the beginning stages, right, when you're putting together your idea?

So for majority of use cases, right, and again, I'm not a lawyer, consult with your legal counsel, but for majority of cases that what we're seeing right now, if you're compliant according to EU AI Act rule, because it is being treated as gold standard, a lot of countries are trying to follow their footsteps, right? Then you can actually have a very good sense of is it considered high risk, low risk, right? And there are tools out there that can do it for free. We also have a compliance checker. So again, feel free to connect with me. I can guide you in more details. But if you're, for example, operating in China, they have a different set of AI regulation. What's considered legal in China may not be legal in EU, for example, or, and vice versa, right? So those are specific use cases that you really do need to have more expertise helping you, I would say. But if you're, you know, operating in, you know, North America, EU, some of the APAC countries, we actually have a quite large footprint in APAC also, starting in, you know, Singapore, but we have customers in Hong Kong, Taiwan, Thailand. I'm happy to talk about the different philosophy, AI governance framework guideline, or go look at our AI global tracker. It'll give you some insights there as well.

Awesome. We have this question here from Elizabeth. This is a little more specific. Elizabeth is essentially building out a healthcare startup, and they are tracking a bunch of different things, right, that could be deemed as high risk depending on how they see it. So they're in pre-alpha and they're testing use cases. How do they set up the mapping so that the governance scales?

Okay, uh, So that the governance scale. So I'm assuming when you say scale, meaning that you're expecting there's a lot of iterations since you're in pre-alpha, or you may have more than one model, you may have 10, 100 in your roadmap in the future. And that's why, you know, you can right now maybe do it on a spreadsheet, but having a solution, a software system to help you automate some of these compliance requirements already mapped to these controls would be helpful. We see that as a need. That's why, you know, we build this business. But like, again, like, early day startup, right? You're always juggling with resources, budget, time. So is it the right time for you to bring in a tool? Really depends on those three things, I would say. Yes, there's solutions like ours out there that can help you scale and map, but is it the right time for you to do that? It's the question that you as a founder need to answer.

Awesome. We had this question here from someone, it's more of a comment and you can confirm or deny. They asked if using, for example, OpenAI's API, reduces your risk?

What kind of risk is one thing I would ask. It could be security risk, it could be legal compliance risk, it could be ethical risk, you know, but I would say ethical risk is the hardest to be like quote-unquote compliant out of the box because it really depends on the use case, jurisdiction that you are using the OpenAI for, right? So, for example, a customer in Malaysia, and the Malaysian AI ethics guidelines says the LLM must be loyal to the king, right? That's not a concept we have in Canada. So, did you test that your LLM is loyal to the king? Did OpenAI test that the LLM is loyal to the king, right? Security, I would say OpenAI is pretty good with security guardrails compared to other models, for example.

Okay, so we have a kind of a similar question. We're getting a lot of questions which are becoming more niche and niche, right? So we're seeing a ton of different use cases for founders across the world. So a deal here from Pakistan, asks, um, let me just bring it up on stage. In fact, are there responsible AI frameworks or validation checklists specifically, specifically for keeping LLM-driven financial insights accurate and grounded in real company data? Or as, as a whole, maybe if you could, uh, touch upon, uh, the frameworks for building out a fintech, right, an AI fintech, are they different from regular AI governance, how would you put that?

I would call this even one more step higher level, is just regulated industry in general. A lot of AI doctors, making sure that you don't recommend something that's wrong. The best practices there right now is you definitely have to have a golden dataset that's provided by human to do the most basic level of checking is not answering the popular questions wrong. That is the level number 1. But level number 2, there are open source data on Hugging Face, for example, that provides some benchmark, some toolset for you to do these testing, benchmark these testing. However, if you are training your model to benchmark against these open source exam questions, I like to say them, you're optimizing for the exam questions and you definitely will have blind spots in your LLMs for other areas that these benchmarks don't cover. But it's back to the 0 to 1. You are at the 0 to 1 stage. Doing nothing is not an option anymore. Doing something is better than a lot of startups already. Is it going to be perfect? No. But back to the original concept, What's the risk appetite of your company? That really depends on the use case, the profit and loss, right? Is it regulated, right? The regulator, is it considered high risk? So all those things combines back to how much you should test, right? And what you should test, right? All these things. So if you want to talk about specific industry, I would say yes, there are some specific industry think tank group that is working on these specific benchmarks. And problems. I know there's a couple in financial services, I know there's a couple in healthcare. So yes, these are things that experts and thought leaders are definitely trying to do the job.

Awesome. Okay, we have a really interesting question here and we're going to take a couple more questions before we wrap up. So Durga asks, how do regulatory frameworks handle models that learn continuously from new data? Are there expectations for retraining and auditing? I think a better way to put this is whenever there is a new model release, do the frameworks change with that?

So when you say regulatory frameworks, you're thinking in terms of compliance. So in financial services, there is actually a model risk management framework called SR 11-7. So this is related to the previous question. Kind of. It already requires models that have profit and loss impact in the States to go through this periodic model validation, model audits. They call it 3 lines of defense. So again, you know, some industries are more advanced. I would say the banks are most advanced in regulating AI models right now, and yes, some of those models do require periodic audits. All right, doesn't matter if it's AI or non-AI models.

Yeah, awesome. Okay, so we have a— we'll take our last two questions here. We have this question here from Edwin, and, uh, I'll rephrase this a little bit, uh, but they essentially ask, uh, in a midsize company, like a few departments working on with an AI project, what organizational structures have you observe, would you recommend for efficient AI assurance? And I think what I want to maybe talk about is, let's say you are, you're a regular business, right? And you have started using AI, right? And it's a part of your product. Like, do you need to be aware of AI governance and compliance? How do you, uh, how do you marry that into what you're already doing, essentially?

Yeah, I think the main thing, especially if you're in a mid-sized company that is trying to bring in AI versus like your AI startup that's like AI first from the beginning. What I found, and because I had my chapter 1 in traditional software, enterprise software, right, is if you're trying to budget, project plan your AI roadmap like you do with traditional software, you're bound to fail. Because traditional software, again, is deterministic, easily can tell when you can achieve 100% compliance, 100% accuracy. But probabilistic model, right, even though if you add 10 more PhDs to your team, if you don't have good data to begin with, a lot of the times, right, you're not going to get the, you know, 90% accuracy rate or 95% accuracy rate that you need to go live. But do you even need 90% accuracy rate? Do you need 95% or is 80% okay, right? So all those things, you need someone to understand these concepts and establish the right policy, right? Sometimes there's someone looking at their established industry benchmark that you use. 80% for your industry is considered acceptable, therefore you only need 80% accuracy rate before you can approve this AI, right? So all those things are why you need the AI governance and the AI assurance experts in your team. And like, we specialize in that, so happy to help and talk to your team about that. But like, back to the roles, right? At the minimum, you don't want to have your data scientist, your machine learning engineer be the only one that understand these concept. And tell you that it is, my model is 95% accuracy, the first question you should ask is, how, what's the sample size you used to test this, right? Did you test it 10 times and like 9.5 times is correct, therefore it's 95%? Back to, you know, what's the risk tiering of your AI use case? If it's a self-driving car, testing 10 times and tell me that it's 95% accuracy is not good enough, right? If it's an AI medical advice chatbot, testing 10 times is not good enough. That's why you need to combine all these concepts and really understand it's a new paradigm to test AI models and these probabilistic systems. So, I think you need someone that's independent from your development team to be the tester. I think that's how you can protect yourself, especially your founder that is non-technical.

Awesome, awesome, great, super detailed answer. Okay, so last question here, and this is because a bunch of people have asked and I'm also very curious. Fionn, what are some good resources or like maybe some, you know, blogs or reading materials that people can refer to when, if they want to stay updated on AI governance and frameworks? And also, of course, they can always add you on LinkedIn, right, if they have a question. Shoot one over to you, but like, what would you recommend other than that?

Yeah, well, definitely sign up for our newsletter on— I'll put the website name again, esenian.ai here. Sign up for the global AI regulatory tracker specifically if like you're operating in regulated industry or you just want to make sure that you follow these AI regulation developments. So I would say the tracker is our most popular resource, free resource to subscribe to. And are you allowed to send out an email afterwards for the attendees? I can give you a list and you can just email it out to people.

Uh, yeah, we, we do send a follow-up email if you have, uh, whatever resources you want us to share. We'll do that for sure.

Sounds good, I'll do that.

Awesome, perfect. There you go, folks. Uh, and in case you're wondering if you get a recording we'll be sending out a recording of the webinar in the next 48 hours, hopefully with those resources as well. Fionn, thank you so much for taking out the time to be here. Thank you for all the incredible insights. I know we could probably spend another hour going through questions, but folks, we don't have that time, so feel free to add Fionn on LinkedIn. Thank you so much for joining us today, and do stick around for the networking afterwards. Fionn, before we close things out here, any last words for all the wonderful people here?

No, I think by attending this session, you're already better than a lot of startups that are not thinking about these issues. Like I said, it's hard to reverse engineer responsible AI, so doing it from day one definitely will make your life easier down the road. Thank you everybody for your time and feel free to connect.

Awesome. I hope you all have an amazing day ahead, and if you want to be a part of another FYI event, head on over to fi.co/events. We'll see you in the next one.

Take care.