Pauline Nissen: Building a Culture of Responsible AI
In this episode of the AI Pathfinder for Private Equity podcast, Steve Budd and Pauline Nissen, Ethical AI Lead at ML6 (builder of custom AI systems) discuss the critical importance of ethical AI and AI governance in the private equity sector. They explore how to promote responsible AI practices, the implications of the upcoming EU AI Act, and the necessity for organisations to adopt a culture of ethical AI. The conversation highlights the risks of neglecting ethical considerations in AI development and offers insights into best practices and resources for navigating the evolving landscape of AI governance.
Takeaways
- Ethical AI is essential for responsible AI development
- AI governance processes help mitigate risks associated with AI
- The EU AI Act will significantly impact AI practices in Europe
- Organisations must promote a culture of ethical AI
- Investing in responsible AI can enhance user trust and reputation
- Transparency in AI practices is crucial for ethical compliance
- AI ethics involves more than just legal compliance; it requires cultural alignment
- Private equity firms should assess the ethical practices of potential investments
- AI governance should involve multiple roles within an organisation
AI Pathfinder Private Equity AI Strategy Meet-ups:
https://www.aipathfinder.co/private-equity-ai-events
Have a question or would like to suggest improvements, please contact steve@aipathfinder.co
Hello and welcome to the AI Pathfinder for Private Equity podcast. I'm on a mission to lead private equity from AI bewilderment to AI empowerment. And I'm doing that by building a dedicated network that will piece by piece create a playbook for successful AI adoption across private equity operations, origination, and value creation. I'm meeting and having very interesting conversations with people within PE and across the AI value chain. I'm very pleased to bring some of these to life through this podcast. Today we're going to talk about a topic that could make or break an AI project or even a company, ethical AI and AI governance. This is going to be hard to keep to 30 minutes, but our mission is to help you understand this complex area and give you some guidance. I know my special guest is up for the challenge. I'm very pleased to welcome Pauline Nissam, uh, ethical AI leader at LML6. Hello, Pauline. Thanks so much for joining me.
SPEAKER_01Thank you also for uh inviting me.
SPEAKER_00Fantastic. Yeah, it's really good to have you here. And uh we had a a conversation, what is it, a couple of weeks ago, and uh it's a really fascinating subject. And uh yes, we could be talking about this for some time, but uh for today we're trying to stick to that that 30-minute well look first of all um perhaps you could just give us a brief um uh view on on yourself and on ML6, and then we can get a bit more into the the the ethical AI side of things.
SPEAKER_01Yes, so I'm working at ML6 as uh the ethical AI lead. Uh so ML6 is an AI company. Uh we are building AI custom solutions for our clients. So our clients are from different industries, insurance, uh, manufacturing, retail, e-commerce, and so on. And based on what they want, based on the problem, we can build the custom AI solutions. And I'm also uh working there as an ethical AI lead. So mainly working on internal and external awareness about ethical AI. So making sure that the solutions that we are building for clients are responsible, ethical, but also, for example, legal, compliant with the AI Act which is coming. Um then externally also doing some awareness about ethics, sharing best practice via uh events, blog posts, and also this podcast, for example.
SPEAKER_00Well, uh yes, that's um that's a really good bit of context there. And and it feels like we're gonna get to the point where we're probably gonna see more of you in other companies. But I suppose the first question is why why now? What led ML6 to hire you as an ethical AI lead?
SPEAKER_01Yes, so I think it started three years ago. So we created the ethical AI unit. Uh so it's a first-team unit, so it's a unit of people working in from different units, for example, ML engineers, uh, salespeople, people from HR. Um, and and I think the idea was why do we need an ethical AI unit at MS6? Is because we realized that um AI was not just experimenting, it was really building AI solutions with an impact on society, with an impact on humans. And I think at that point in time we decided: okay, do we want to build anything with AI or do we want to make a choice in what we want to build and what we don't want to build? So that's why we created this ethical AI unit. An example of what we created back then is like our wet lines. So it's a couple of rules of red lines of projects that we don't want to build. An example could be like if a client is asking us, oh, I'm willing to improve weapons with AI. This is, for example, a red line, so something that we don't want to do because we really believe in using AI for the good for the society and really preventing harm to the society. So I think that's that's the first part. Like, why do we want to have an ethical AI unit to make sure that we build the right thing for the society? But ethics, it's also not so white and black. Um, most of the time it's grey. Ethics is also something very personal. So that's why we have in this unit um a sounding board. So, what's the goal of the sounding board is to have a kind of a discussion in a group. Let's say, for example, we have uh a new project, so a client is asking us, we we we want to build something, but it's not very yeah, it's not very sure if it's really something with an impact, with a positive impact on the society. Then we have this sounding board, so we are discussing in a group about what are the advantages, what are the the what's the negative impact, for example, of building this AI solution. And we ask ourselves, are we okay with ML6 building this solution? An example, uh it's it's very sometimes difficult, but I think it's great to have those kind of discussions and also making sure that employees are behind the idea of willing to build this in this use case. And sometimes what we have, for example, a typical example is tobacco industry. Tobacco industry, you could say it's a very negative industry, but sometimes some of the AI projects that we could build there can have a positive impact. Um, an example could be detecting defects on packaging of cigars and reducing the waste of this packaging. So that's something that's a use case with a positive impact, but maybe in a negative industry. And again, it's something very personal. So someone else will say maybe tobacco it's okay, I'm I'm fine with it with that. But yeah, defense, military sector is a no-go for me. So that's why we have those kinds of discussions. So I think it's really important to know making time and don't going too fast and really thinking what are we going to build with AI. So that's why, and and also we have an internal rule stating that if your personal ethics, your personal value is more than the ethical value of MS6. If you don't want to work on a certain project because you think, for example, that the reputation of the company is not so ethical, then you can decide like we we are very open and it's not a problem for you to not work on that project. And I think that's also one of the main reasons, the ethical part of MS6, that people are also joining ML6. I think it's something new in a lot of AI companies that we make time and we are not going too fast to make the right decisions.
SPEAKER_00Yeah, so it's not it's not just down to policies, it's down to culture, and and that's got to come uh right down from the the founders or the the leaders within the business.
SPEAKER_01And it's really something that uh it's also um important, I think, that's it's aligned with uh the management and the strategy of the company. And that's also something, yeah, we have buying from the management, so it's not like it's it's we are really working together. Uh so that's also important.
SPEAKER_00Yeah, yeah. Okay, so first and foremost, it came about because you know, the culture and what you're wanting to build as a business within the values, um, being really clear about you know, these are the projects we do, or these projects we don't, or if there's a gray area, we have a really considered um discussion about it. But I believe that that's sort of that's developed into now supporting your clients with those sorts of decisions as well. Or certainly for them to be clearer on what they can and can't do, or where there's no laws is where you know, where you should probably consider there might be laws in the future, etc.
SPEAKER_01Yeah, yeah. Um so indeed that internally, but of course we are very open and transparent also towards the clients. So sometimes they have also an AI use case, a project that they want to build, and and the project is fine from an ethics perspective. But then if we look into, for example, the data, we see that the data, a typical example is that the data is not representative. So the use case is good, but the the so to build an AI solution, you need to train your AI models, you need to give examples of previous decisions. And sometimes what they see is that the data that they have are not representative of the situation that they want to analyze. And this could be a problem if the AI system that they use will have an impact on humans, on the society. Um, because the typical example is like if you want to detect fraud and you take previous decisions and you see in your data set that many of the decisions were about women and less about men, for example, then the AI model could predict in the future that there is a pattern between the gender and the chains of fraud, which is probably not the case. But and also then you can ask yourself: is it something that we want to include in the training data? Is it a pattern that so those are the kind of ethical decisions and considerations that we discuss openly with the clients to make the right decision with them?
SPEAKER_00Yeah, we'll go into those in a little bit more detail in in a moment. I I suppose just to sort of kick things off for our listeners, you know, what what's the worst case scenario, you know, if if if we're not following, I mean, okay, we've got the scenarios that, you know, building towards uh weaponry or things like that, but generally the listeners are going to be not in that field, but they've still got to be really mindful. You know, we've already we've gone through the process of uh GDPR and being more sensitive to personal information. So what in this situation, what do you think the worst case scenario is if you're if you're not really tracking and following um a more ethical or have governance in place?
SPEAKER_01Yes, I think it's actually broader than just ethics, so indeed legal. I think the worst case scenario is like building an AI solution without what I call AI governance or risk management process. So it's a kind of a process that you have in your development phase of your AI solutions to really identify and mitigate risk. And the worst case scenario is that you're building an AI solution too fast because you want to release it to the public as soon as possible, and you don't take enough time to like identify those risks and having mitigation actions to make sure that they are not have a big impact in finding a solution. I think that's the the worst case scenario because at the end it will have can have an impact, negative impact on humans, and then it's too late. Then it has already had an impact. So that and if you could have predicted that there is a risk, I think that's something really important for the future of AI to have like those kinds of risk management processes in place, making the time before just jumping and releasing it to the public. Because we saw a lot of examples, and I think the problem is not only from ethics, I think the problem could also be like reputation damage. Like if an organization is releasing it too soon with a lot of risk. We saw a lot of examples, such as a um, I'm going to call it a DPD or air canada. Aer Canada developed a chatbot to provide uh advice to passengers about uh flight information, but at the end it it gave bad advice to the passenger. And air canada say it's not our fault, it's the responsibility of the chatbot. So that's something that you could have prevented with guard race, and that's something that you the fact that there is no one accountable for a mistake of the chatbot, that's something that you could have predicted. So, and I think indeed worst case scenario for a company is also uh yeah, and the impact on the reputation and on the user trust.
SPEAKER_00Yeah, yeah, and you mentioned um, or we've obviously mentioned ethical here and you touched on legal, but in the sort of well, the three buckets that you mentioned before to me were ethical, legal, and security as well. So there's there's there's actually building an an LLM, for example, but it's actually just the you the utilization of of an LMM within your employee uh group as well. So, you know, security is a is is is a big in this in this sense.
SPEAKER_01Yes. So um when I'm talking about risk management process, I'm really talking about ethics, legal, and security. Security is also a big topic. Um that's also something that we also look into it before we release because data breach can happen, uh prompt injection with uh Genei applications can happen. So that's something also that we we test before we release the application. And from an ethical perspective, what we look at is like fairness or re-favorizing a certain group over another. Um transparency, is it clear for the user that we are why we are making a certain decision over sites? Because sometimes you forget that not everything can be automated. So sometimes you also need to have like a human review, what we call it a human in the loop to trick the results before it's it's released to the user. We also look at society and environments. You just mentioned using LLMs. I think indeed with the with the generative AI, it's also a big topic, like making sure you can reduce the impact of LLMs on the environment. So, for example, using a smaller LLM if it's not necessary to have the latest version. So those are also the kind of things that we look at because ethics is very broad. And then legal, of course, I think GDPR, but now also the AI Act, the EU AI Act is also a very big topic.
SPEAKER_00Well, look, um well, let's let's talk about that then. I mean, you've talked about pretty much self-governance, but you've got good knowledge of the EU AI Act, and that's coming into force, is it next two years? Is that right?
SPEAKER_01Yes, August 2026.
SPEAKER_00Um Yeah, so how how is that looking to control AI's use then?
SPEAKER_01Uh I think it's going to have a very big impact on a lot of companies using or building AI solutions and and also maybe not only for the companies in Europe, but also for like uh companies in the UK or in America and the rest of the world offering products to the to the European market. So because it's also applicable to them. And what's the EU AI Act? Basically, it's it uses um a risk-based approach. So it defines categories of AI use cases. Um, and for example, the first one is the unacceptable risks. Um, so you could have an AI system, for example, to recognize emotions at a workplace or during a classroom um at school, for example. And this is something that is so unacceptable because there is some privacy. I mean, having a karma looking at your emotions, it's it's maybe a bit weird. But also there is no scientific evidence of AI being able to detect the right emotions. I think it's also it really depends on the culture as well. So minorities could be discriminated by those systems. So that's why those systems, so there is a list of systems which are unacceptable, which are prohibited. So as soon as next year in February, those AI systems will be prohibited. So you are not allowed to develop it anymore. Then you have the high risk, and and I think unacceptable risks are like obvious. I think most of the time you're not developing those kind of AI systems. And the high risk are more uh tricky because I think they can deliver a value, but they are high risk, so you there are some risks. And so, for example, it's like uh the tools using in in recruitment. Uh, there can be uh discrimination issue and so on. So that's why it's allowed under the AI Act to develop those tools, but you need to be compliant with a lot of obligations. Um and the high risk is really obligations like uh having technical documentation, conducting a risk analysis before. So it's a lot of obligations. Uh, we should not underestimate it. Then you have the transparency risk, basically generative AI project. So if you're building a chatbot, your chatbot is probably a transparency risk. What you need to do there, it's it's less extensive than high-risk AI systems, but what you need to do there is disclose transparency. So, for example, if your users are using or interacting with a chatbot, having those kind of disclaimers to say, like, hey, now you're interacting with an EI system, not risk a human. And then finally you have the no-risk. And I think sometimes you also forget that the no-risk are also they have some high some um, but it's for example, a computer vision solution detecting defects on packaging in a manufacturing um and setup. So those don't have very high risk for the society, for the humans, so that's why it's uh it's it's a there is no obligation there. And and as you mentioned, so the the AI Act will be applicable um in two years, with certain obligations, as I mentioned, the prohibited use case will be in February of next year. So the companies really have the time to prepare. Um, and it's going to be progressive as well, because not everything is clear at this moment in the law, so we need more guidelines to make sure how can we implement it. But what I really think it's interesting that the AI Act is if you look at the obligations, it's really based on ethical AI principles. One of the obligations is to increase transparency, or as I just mentioned, like informing users about the use of AI. So this is something really important from an ethical perspective, making sure that the users are informed.
SPEAKER_00It comes across as you talk it through there as fairly logical. I think that immediately there could be concerns that it could suppress innovation. And obviously from a UK perspective, we've not I'm not sure we've got any, you know, it's very loose anyway, in terms of what we're potentially gonna have. And you know, that that suppose the thoughts around that is that well, we're we're going to um innovate faster. But as you say, you know, if you're if you're a business that's going beyond these borders, you're gonna have to consider all of these factors, which are factors that should really be in your thinking. Who in an organization, so an organization that's looking to AI to develop it might be an internal tool, some it might be that they're developing a product but to use externally. Who's who's looking at this? Is there a particular role that this lands on their desk, or should it be everyone's responsibility but to become familiar?
SPEAKER_01Well, that's a great question because I think a lot of organizations they know about the AI Act, but they don't know in the organization they're also looking who should be responsible for it. I think a lot of the DPOs at this moment are being asked so because they implemented the GDPR and it feels like it's the same kind of flow. I think a lot of DPOs are asked to indeed take on that role to implement the AI Act also. But what we should not underestimate with DPO is also that you need AI skills, AI knowledge. I think for one of the obligations I just mentioned is also conducting a risk analysis. Risks, it's not only about personal data such as GDPR, it's also about, for example, uh generating harmful content. It's also about the problem of hallucination with LLM, so that you are not sure that the answer that you receive is actually correct. So those are risks I think that you really need to understand how AI is working before being able to identify and mitigate those risks. So I think a lot of DPOs are being asked, and probably in the future they will have like a DPO and maybe an ACT implementer or AI governance need. But I think it's also something that it's broader than just one person. Um because as I mentioned, like one of the obligations is also writing technical documentation. Yeah, it's not something that the DPO is going to do. So that's maybe an ML engineer or a data scientist working on an AI project, which will also be need to be aware of the implications of the AI Act. So it's uh it's broader in the company, it's just that's one person.
SPEAKER_00Yeah, and and absolutely that uh uh you know it's gotta be company-wide adoption, hasn't there? Um, and that goes with you know having some AI principles attached to the values that leadership are on board, is that everybody's uh understands. When we're talking about AI knowledge, you know, that's gonna have to really be across the organization, especially if if uh it's an organization that wants to embrace this emerging technology as well. Let's move on to I suppose some more direct advice for private equity. Now they're they're in an interesting position because obviously they're thinking about themselves as a firm and how they might apply AI, but they're also buying businesses that are going to have potentially AI within it, either as a product or something that they've got again, um and they've also got a portfolio so they could have 20, 30 plus companies um with differing um levels of of adoption. I mean that's that's a pretty big task, isn't it? To now it's not on them to uh to be day-to-day thinking about how those companies are using AI, but you know, they're liable to some degree in terms of of uh custodians of those businesses. Look, I know you're not a private equity expert, but what sort of advice would you give them um uh around this sort of, I suppose, interesting puzzle that they may have ahead of them?
SPEAKER_01I think what's interesting to to look into is like if indeed it's it's to invest in a company building an external or internal AI solution, I think it's really important to consider their approach to ethical and legal AI. Um so we just talk about the AI Act and we just mentioned okay, the AI Act, it's it's still have two years to be compliant. So maybe it's too early to say like uh if it's the company is compliant with the AI Act, it's fine. I think it's it if it's already compliant, it's a really green flag. I think it's it's a very proactive approach, so that's interesting. But I think so at this moment in time, because it's too early to say like if the company is compliant with the AI Act, then you should invest into. I think it's more looking into the mindset, the mindset related to are they willing to become compliant with the AI Act? Are they willing to build responsible AI solutions? Because I've heard some organizations stating that indeed because of like a it's kind of a blocker for innovation, AI Act, it's it's also for the whole Europe, for small and large organizations. So, I mean, if I am a small startup, nobody nobody is going to look at my AI system uh to say so. So I think that's that's really for me a big uh red flag. You should have at least a mindset to to try to look how is the AI Act going to impact my organization. Organization, do we have a plant? Do we want to hire someone to look into this? So I think that that's really important to have that mindset because you can see it from a compliance point of view. But I think it's not just about compliance, it's also about the reputation of your organization. As I just mentioned, for example, like Canada, I think because of the problems that they have had with their chatbots, it's also impacting the reputation. And maybe you don't trust the company anymore in the way they handle. And then it's not only about reputation, because this is maybe the negative approach, but the positive approach could be if you want to invest in responsible AI, you can also improve the user trust. I think it's really responsible AI and legal AI could also be an USP for a lot of companies because the articles that you see in the media about uh the risk of AI, those are the same articles that your clients or user are gonna read. And if you can reassure them that you're taking the right step, I think you can really increase this trust of your users in the product. So I would really advise to look into the mindset is uh responsible AI and legal AI important for the company before investing in it.
SPEAKER_00Yeah, yeah, that's good, good advice there. In terms of them thinking about their portfolio, I know that many of them are assessing how um I suppose their foundations for not just adopting but building out AI within their businesses, seeing which ones, which industries versicals are going to have more opportunities or threats against them. So so you know, there's a lot of work going on to try and better understand the landscape, but in terms of I'm not sure there's a lot going into thinking about um you know ethical, legal, security necessarily around AI. So, you know, if you've got um 30 companies within your portfolio, what you know, what advice would you give in terms of, you know, as a P firm, would you set up perhaps you were talking earlier about the discussions that you have as a as a as a team, you know, should this be something that's that a P firm should have a central point around learning about AI governance, ethical AI, those aspects. Just, I mean, uh this is brainstorming a little bit, but just thinking how could they support their portfolio and making the right decisions here.
SPEAKER_01As I just mentioned, it's also important to have like buying from the management from investors. So I think it's also important to promote that. So if it's not something AI governance or as I just mentioned, something uh board that we have at ML6, if it's not something that is already in place, to at least start with responsible AI culture. So promoting that it's something important that we want to invest into, that we want to be recognized as a leader, for example, in the market. I think that's the first step, building a culture of responsible AI. Um, next step are processes, having those kinds of structured processes in your organization. But what we often forget about processes, for example, you could say, okay, I'm going to have an AI policy in those organizations to make sure that we are using in the right way. But what we often forget is AI governance and using AI in the right way, it's all about people. It's all about change management. You really need first to convince them of the why before having an very long AI policy, because otherwise no one is going to read it because they don't think it's important. So I think for me that's the first step. Promoting from the management that responsible AI is important, having the right culture fit, um, and then it's going to start in the organization.
SPEAKER_00Great, great. Um, well, uh, I think that often we want to look at other examples out there of people doing it uh well. Who's getting it right in your opinion? Who can we look at in terms of best practice?
SPEAKER_01I just mentioned like reputation and user trust. I think uh very and why uh investing in responsible AI can also be seen as a USP. I think a very good example, like uh Adobe. So um I think one year ago they released Firefly, so it's a GI tool. At that moment in the GI space, it was very difficult uh with uh copyrights. So a lot of of like uh photographs, um photographers were really worried about their own image being used by other products. So and I think what's what uh Adobe did really well, it's like very being very transparent and very open to the users. Like they they disclosed what kind of pictures uh they they used, and they also gave the guarantee that it's either in the public space, either they have the the rights um to train, to use those those images uh as a training set for the Genii model. So I think that's something that you you it's a good practice to inform your users and to reassure them that there is no copyright issue, for example, because then they have all the right information to make the right decision whether to use or not the tool. And that creates trust a lot more than than other uh being like a black box and not uh disclosing uh their images, for example.
SPEAKER_00Yeah, it becomes well a USP, doesn't it? Really? And and if you're wanting to follow certain principles, then you need to be working with vendors and suppliers that that share share those those principles. And I know that um you know, in terms of creating those principles, there's some good examples out there, even from the those that um are sort of the leaders, really, like Microsoft, they've they've got some principles there that uh probably worth having a look at as as as do Google uh and others. So I think that there's some good examples out there. Um no excuse, I suppose, to create a first draft to get the ball moving. We're coming near to the end now. I suppose that um we're getting bombarded by lots of ideas to utilize AI, new tools, everyday developments in in the software. But when it comes to ethical AI and how that's gonna progress, obviously we've got the EU Act, but but others may follow in other countries. How do you stay on track with this?
SPEAKER_01Is there a source that we should be uh I think it's kind of funny because I think like in in the in the AI space, there is a very open community about AI, and it's also the case about ethical AI. So, for example, there is a very large community of experts on the AI Act, AI governance, ethical AI on LinkedIn. I mean there are a lot of experts, companies sharing their best practice very openly, like hey, we did this in our AI solutions, and we are just sharing uh the learnings, or you also have academics sharing uh a new paper, recent discovery about ethical AI. So that's something that I really like. It's like just sometimes scrolling on LinkedIn, you learn so many things of people just sharing openly their uh discoveries. Um, and and maybe if I can give two important resources uh for people interesting to look into ethical AI, the first one would be the assessment list for uh trustworthy AI. So trustworthy AI, responsible AI, it's all the same. It's a checklist that has been released by the European Commission. And it's a very interesting list because they have a lot of standard questions that you can ask yourself when you're building an AI solution. For example, uh, did you assess the impact of your AI solution on humans, on the society? So I think it's very, very interesting uh document to have a look at to really try to understand the ethical implications of an AI system. And then the next example, the second uh resource that uh I could share is also like a database, which is called incidentdatabase.ai. And it's actually a website, it's maybe less positive, but it's actually a database with a lot of examples of incidents. And I think you can learn a lot from those incidents. Uh so it's it's really like monitoring ethical AI incidents, such as the wrong use of deep fakes and so on. Yeah, those two resources are very useful to discover the world of ethical AI.
SPEAKER_00Great. Well, perhaps I'll get those links from you and put those into the uh the list, the details on the podcast page. I've come to time. I think you've done an excellent job of giving some um guidelines here for the listeners. Pauline, thank you very much for your time and uh really enjoyed the the conversation. How should people get in touch with you? It's perhaps LinkedIn. Yes, the best way. Fantastic. Well, that's it. Um as I said, we've achieved our mission. Thanks again, Pauline, and we'll see you next time.
SPEAKER_01Thank you. Bye bye.